Telegraf doesn't seem to send data to InfluxDB in dockerised images

Telegraf
,
1

Hi,

I have a dockerised telegraf (telegraf/1.21) and influxdb (influxdata-docker).

I’m trying to send data to influxdb from ttn but without any success. The config file for telegraf is:

[agent]
interval = "30s"
debug = true
quiet = false
logfile = "/var/log/telegraf/telegraf.log"
[[inputs.mqtt_consumer]]
servers = ["tcp://eu1.cloud.thethings.network:1883"]
qos = 0
connection_timeout = "30s"
topics = [ "+/devices/+/up" ]
client_id = "_my_id"
username = "my_username"
password = "my_password"
data_format = "json"
[[outputs.influxdb_v2]]
urls = ["http://172.21.0.2:8086"]
token = "my_token"
organization = "iot"
bucket = "ttn_data"
[[outputs.file]]
  files = ["stdout", "/var/log/telegraf/metrics.out"]
  use_batch_format = false
  rotation_interval = "0h"
  rotation_max_size = "0MB"
  rotation_max_archives = 5
  data_format = "influx"

It seems that the dockers have been setup properly and one sees the other:

Starting datahandler_influxdb_1 ... done
Starting datahandler_telegraf_1 ... done
Attaching to datahandler_influxdb_1, datahandler_telegraf_1
influxdb_1  | 2022-03-25T21:16:33.734688793Z    info    found existing boltdb file, skipping setup wrapper      {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt"}
influxdb_1  | ts=2022-03-25T21:16:33.895365Z lvl=info msg="Welcome to InfluxDB" log_id=0_SjrEuW000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z
influxdb_1  | ts=2022-03-25T21:16:33.899857Z lvl=info msg="Resources opened" log_id=0_SjrEuW000 service=bolt path=/var/lib/influxdb2/influxd.bolt
influxdb_1  | ts=2022-03-25T21:16:33.900228Z lvl=info msg="Resources opened" log_id=0_SjrEuW000 service=sqlite path=/var/lib/influxdb2/influxd.sqlite
influxdb_1  | ts=2022-03-25T21:16:33.912460Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0_SjrEuW000 bolt_path=/var/lib/influxdb2/influxd.bolt
influxdb_1  | ts=2022-03-25T21:16:33.912610Z lvl=info msg="Using data dir" log_id=0_SjrEuW000 service=storage-engine service=store path=/var/lib/influxdb2/engine/data
influxdb_1  | ts=2022-03-25T21:16:33.912651Z lvl=info msg="Compaction settings" log_id=0_SjrEuW000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
influxdb_1  | ts=2022-03-25T21:16:33.912670Z lvl=info msg="Open store (start)" log_id=0_SjrEuW000 service=storage-engine service=store op_name=tsdb_open op_event=start
influxdb_1  | ts=2022-03-25T21:16:33.942051Z lvl=info msg="index opened with 8 partitions" log_id=0_SjrEuW000 service=storage-engine index=tsi
influxdb_1  | ts=2022-03-25T21:16:33.944176Z lvl=info msg="Reading file" log_id=0_SjrEuW000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb2/engine/wal/fd224ef0f0444b4b/autogen/2/_00001.wal size=1576906
telegraf_1  | 2022-03-25T21:16:34Z I! Starting Telegraf 1.21.4
telegraf_1  | 2022-03-25T21:16:34Z I! Using config file: /etc/telegraf/telegraf.conf
influxdb_1  | ts=2022-03-25T21:16:34.165432Z lvl=info msg="Opened shard" log_id=0_SjrEuW000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/fd224ef0f0444b4b/autogen/2 duration=249.364ms
influxdb_1  | ts=2022-03-25T21:16:34.165582Z lvl=info msg="Open store (end)" log_id=0_SjrEuW000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=252.912ms
influxdb_1  | ts=2022-03-25T21:16:34.165659Z lvl=info msg="Starting retention policy enforcement service" log_id=0_SjrEuW000 service=retention check_interval=30m
influxdb_1  | ts=2022-03-25T21:16:34.165687Z lvl=info msg="Starting precreation service" log_id=0_SjrEuW000 service=shard-precreation check_interval=10m advance_period=30m
influxdb_1  | ts=2022-03-25T21:16:34.165749Z lvl=info msg="Starting query controller" log_id=0_SjrEuW000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
influxdb_1  | ts=2022-03-25T21:16:34.169554Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0_SjrEuW000 max_select_point=0 max_select_series=0 max_select_buckets=0
influxdb_1  | ts=2022-03-25T21:16:34.476773Z lvl=info msg=Listening log_id=0_SjrEuW000 service=tcp-listener transport=http addr=:8086 port=8086
influxdb_1  | ts=2022-03-25T21:16:34.476855Z lvl=info msg=Starting log_id=0_SjrEuW000 service=telemetry interval=8h
grafana_1   | t=2022-03-25T21:16:43+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/live/ws status=0 remote_addr=172.21.0.1 time_ms=1 size=0 referer=
influxdb_1  | ts=2022-03-25T21:16:51.247533Z lvl=info msg=Unauthorized log_id=0_SjrEuW000 error="session not found"
grafana_1   | t=2022-03-25T21:26:34+0000 lvl=warn msg="Failed to close response body" logger=plugin.manager err="context canceled"

In the /var/log/telegraf/metrics.out doesn’t seem to write anything and in telegraf.log I have:

2022-03-25T21:42:20Z D! [agent] Stopping service inputs
2022-03-25T21:42:20Z D! [inputs.mqtt_consumer] Disconnecting [tcp://eu1.cloud.thethings.network:1883]
2022-03-25T21:42:20Z D! [inputs.mqtt_consumer] Disconnected [tcp://eu1.cloud.thethings.network:1883]
2022-03-25T21:42:20Z D! [agent] Input channel closed
2022-03-25T21:42:20Z I! [agent] Hang on, flushing any cached metrics before shutdown
2022-03-25T21:42:20Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:42:20Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:42:20Z I! [agent] Stopping running outputs
2022-03-25T21:42:20Z D! [agent] Stopped Successfully
2022-03-25T21:42:28Z I! Loaded inputs: mqtt_consumer
2022-03-25T21:42:28Z I! Loaded aggregators: 
2022-03-25T21:42:28Z I! Loaded processors: 
2022-03-25T21:42:28Z I! Loaded outputs: file influxdb_v2
2022-03-25T21:42:28Z I! Tags enabled: host=169916e15b06
2022-03-25T21:42:28Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"169916e15b06", Flush Interval:10s
2022-03-25T21:42:28Z D! [agent] Initializing plugins
2022-03-25T21:42:28Z D! [agent] Connecting outputs
2022-03-25T21:42:28Z D! [agent] Attempting connection to [outputs.file]
2022-03-25T21:42:28Z D! [agent] Successfully connected to outputs.file
2022-03-25T21:42:28Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-03-25T21:42:28Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-03-25T21:42:28Z D! [agent] Starting service inputs
2022-03-25T21:42:28Z I! [inputs.mqtt_consumer] Connected [tcp://eu1.cloud.thethings.network:1883]
2022-03-25T21:42:38Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:42:38Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:42:48Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:42:48Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:42:58Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:42:58Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:43:08Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-03-25T21:43:08Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

I suppose all the connections have been done properly, however I don’t see anything in influxdb or anything comming in telegraf. TTN has data for sure, but I don’t get anything.

Some advice would be great

2

Hi,

Before I run off for the weekend I took a quick look at your post. The one thing I see is the Telegraf logs do not show that it is writing any data and there is no data in the buffers. That would indicate to me that the input is not collecting any data.

As you have debug login enabled, if data was getting written to an output I would expect to see a message like the following:

2022-03-25T22:23:08Z D! [outputs.file] Wrote batch of 1 metrics in 75.76µs

Your input does look like it successfully connected, so I think this would possibly come down to ensuring that the topics are correct.

Again just did a quick look, let me know what you think and I or one of the other community folks can take another look!

3

where are you getting your docker images from? or did you create them yourself?

I’m asking because if you created it yourself…did you set the inbound port correctly for Telegraf?? You need to expose the right ports from the container for this to work.

EXPOSE 8125/udp 8092/udp 8094

is what is in the dockerfilne for the image that InfluxData produces.

Although it looks like the input plugin is reaching out to the configured MQTT broker…

You might switch the parsing from json to json_v2 – JSON v2 input data format | Telegraf 1.22 Documentation significantly better parsing experience.

4

I’m using Docker Compose:

version: "3"

services:
  influxdb:
    build:
      context: ./influxdb
    restart: always
    ports:
      - 8086:8086
    volumes:
      - influxdb-storage:/var/lib/influxdb
    environment:
      INFLUXDB_DB: actualdb
    networks:
      vpcbr:
        ipv4_address: 172.21.0.2

  telegraf:
    build:
      context: ./telegraf
    restart: always
    ports:
      - "8125:8125/udp"
      - "8092:8092/udp"
      - 8094:8094
    depends_on:
      - influxdb
    volumes:
      - ./telegraf/config/telegraf.conf:/etc/telegraf/telegraf.conf
    networks:
      vpcbr:
        ipv4_address: 172.21.0.3

networks:
  vpcbr:
    driver: bridge
    ipam:
      config:
        - subnet: 172.21.0.0/24
          gateway: 172.21.0.1

volumes:
  influxdb-storage:

So regardless of whether they are exposed from outside or not the IP that telegraf has is 172.21.0.3 and influxdb 172.21.0.2

The images are from :
telegraf (telegraf/1.22) and influxdb (influxdata-docker).

I did nmap in the docker of telegraf and they seem the ports not to be up:

Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-26 17:18 UTC
Nmap scan report for ae4584881e23 (172.21.0.3)
Host is up (0.000060s latency).

PORT     STATE  SERVICE
8092/udp closed unknown
8094/udp closed unknown
8125/udp closed unknown

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds
5

Actually with nmap within the docker it seems that telegraf has not exposed them i.e.:

nmap localhost -sU -p 8125,8092,8094
Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-26 17:18 UTC
Nmap scan report for ae4584881e23 (172.21.0.3)
Host is up (0.000060s latency).

PORT     STATE  SERVICE
8092/udp closed unknown
8094/udp closed unknown
8125/udp closed unknown

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

However the service seems to be up:

service telegraf status
telegraf Process is running [ OK ]

Actually I totally removed the docker container and I installed it in my local machine but again I receive the same results. So it seems a configuration error, not sure if it’s in ttn. In ttn I have in uplink_data. So not sure if something else happens

6

Just in case somebody else has the same issue, it was finally due to the fact I was using ttn v3, ie. in telegraf.conf:

topics = [ "v3/+/devices/+/up" ]

instead of:

topics = [ "+/devices/+/up" ]

1 Like