Telegraf not sending data to influxdb (azure event hub => influxdb cloud 2)

Hi,

I have the following scenario: IoT devices send data to an azure iot hub, the telegraf agent collects the data and sends it to a bucket in the influxdb cloud 2.

The problem that I face is that telegraf seems to connect to azure iot hub and collect the data (because I see in the monitoring spikes of outgoing data), but not sending it to the influx db cloud.

Telegraf config:
[agent]
quiet = false
debug = true

 [[outputs.influxdb_v2]]
  ## The URLs of the InfluxDB cluster nodes.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.

  urls = ["https://westeurope-1.azure.cloud2.influxdata.com"]

  ## Token for authentication.
  token = "XXXXItsAScrect"

  ## Organization is the name of the organization you wish to write to; must exist.
  organization = "XXXX@XXXX.com"

  ## Destination bucket to write into.
  bucket = "paul"


[[inputs.eventhub_consumer]]
  ## The default behavior is to create a new Event Hub client from environment variables.
  ## This requires one of the following sets of environment variables to be set:
  ##
  ## 1) Expected Environment Variables:
#EVENTHUB_NAMESPACE = ""
#EVENTHUB_NAME = ""
#EVENTHUB_CONNECTION_STRING = ""
  ##
  ## 2) Expected Environment Variables:
  ##    - "EVENTHUB_NAMESPACE"
  ##    - "EVENTHUB_NAME"
  ##    - "EVENTHUB_KEY_NAME"
  ##    - "EVENTHUB_KEY_VALUE"

  ## Uncommenting the option below will create an Event Hub client based solely on the connection string.
  ## This can either be the associated environment variable or hard coded directly.
   connection_string = "Endpoint=sb://iothub-ns-iotpaulshu-6039161-c0b5109dc8.servicebus.windows.net/;SharedAccessKeyName=iothubowner;SharedAccessKey=XXXItsASecret;EntityPath=iotpaulshub"

  ## Set persistence directory to a valid folder to use a file persister instead of an in-memory persister
  # persistence_dir = ""

  ## Change the default consumer group
  # consumer_group = "telegraf"

  ## By default the event hub receives all messages present on the broker, alternative modes can be set below.
  ## The 3 options below only apply if no valid persister is read from memory or file (e.g. first run).
  # from_timestamp =
  # latest = true

  ## Set a custom prefetch count for the receiver(s)
  # prefetch_count = 1000

  ## Add an epoch to the receiver(s)
  # epoch = 0

  ## Change to set a custom user agent, "telegraf" is used by default
  # user_agent = "telegraf"

  ## To consume from a specific partition, set the partition_ids option.
  ## An empty array will result in receiving from all partitions.
  # partition_ids = ["0","1"]

  ## Max undelivered messages
  # max_undelivered_messages = 1000

  ## Set either option below to true to use a system property as timestamp.
  ## You have the choice between EnqueuedTime and IoTHubEnqueuedTime.
  ## It is recommended to use this setting when the data itself has no timestamp.
  # enqueued_time_as_ts = true
  # iot_hub_enqueued_time_as_ts = true

  ## Tags or fields to create from keys present in the application property bag.
  ## These could for example be set by message enrichments in Azure IoT Hub.
  # application_property_tags = []
  # application_property_fields = []

  ## Tag or field name to use for metadata
  ## By default all metadata is disabled
  # sequence_number_field = "SequenceNumber"
  # enqueued_time_field = "EnqueuedTime"
  # offset_field = "Offset"
  # partition_id_tag = "PartitionID"
  # partition_key_tag = "PartitionKey"
  # iot_hub_device_connection_id_tag = "IoTHubDeviceConnectionID"
  # iot_hub_auth_generation_id_tag = "IoTHubAuthGenerationID"
  # iot_hub_connection_auth_method_tag = "IoTHubConnectionAuthMethod"
  # iot_hub_connection_module_id_tag = "IoTHubConnectionModuleID"
  # iot_hub_enqueued_time_field = "IoTHubEnqueuedTime"

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  # data_format = "influx"
  data_format = "json"

My telegraf output:

2020-12-16T09:10:59Z I! Loaded inputs: eventhub_consumer
2020-12-16T09:10:59Z I! Loaded aggregators:
2020-12-16T09:10:59Z I! Loaded processors:
2020-12-16T09:10:59Z I! Loaded outputs: influxdb_v2
2020-12-16T09:10:59Z I! Tags enabled: host=goodvirus
2020-12-16T09:10:59Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:“goodvirus”, Flush Interval:10s
2020-12-16T09:10:59Z D! [agent] Initializing plugins
2020-12-16T09:10:59Z D! [agent] Connecting outputs
2020-12-16T09:10:59Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2020-12-16T09:10:59Z D! [agent] Successfully connected to outputs.influxdb_v2
2020-12-16T09:10:59Z D! [agent] Starting service inputs
2020-12-16T09:11:13Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:11:23Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:11:33Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:11:43Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:11:53Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:12:03Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:12:13Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:12:23Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:12:33Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:12:43Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:12:53Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:13:03Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:13:04Z D! [agent] Stopping service inputs
2020-12-16T09:13:04Z D! [agent] Input channel closed
2020-12-16T09:13:04Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020-12-16T09:13:04Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-16T09:13:04Z D! [agent] Stopped Successfully

Do you have any pointers, I´m quit desperate… Or do you need more information?

Thanks in advanved,

Paul

Hello @goodvirus,
To debug telegraf I recommend using
./telegraf -config telegraf.conf -test

or writing data to stdout

[[outputs.file]]
  ## Files to write to, "stdout" is a specially handled file.
  files = ["stdout"]

  ## Data format to output.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
  data_format = "influx"

To check if your line protocol is correct and then going from there. Can you share your lines?

Thank you.

Hello Anaisdg,

the output from ./telegraf -config telegraf.conf -test

2020-12-17T08:54:52Z I! Starting Telegraf 1.16.3
2020-12-17T08:54:52Z D! [agent] Initializing plugins
2020-12-17T08:54:52Z D! [agent] Starting service inputs
2020-12-17T08:54:56Z D! [agent] Stopping service inputs
2020-12-17T08:54:56Z D! [agent] Input channel closed
2020-12-17T08:54:56Z D! [agent] Stopped Successfully

The Output from outputs.files

telegraf --config telegraf.conf
2020-12-17T08:58:31Z I! Starting Telegraf 1.16.3
2020-12-17T08:58:31Z I! Loaded inputs: eventhub_consumer
2020-12-17T08:58:31Z I! Loaded aggregators:
2020-12-17T08:58:31Z I! Loaded processors:
2020-12-17T08:58:31Z I! Loaded outputs: file influxdb_v2
2020-12-17T08:58:31Z I! Tags enabled: host=goodvirus
2020-12-17T08:58:31Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"goodvirus", Flush Interval:10s
2020-12-17T08:58:31Z D! [agent] Initializing plugins
2020-12-17T08:58:31Z D! [agent] Connecting outputs
2020-12-17T08:58:31Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2020-12-17T08:58:31Z D! [agent] Successfully connected to outputs.influxdb_v2
2020-12-17T08:58:31Z D! [agent] Attempting connection to [outputs.file]
2020-12-17T08:58:31Z D! [agent] Successfully connected to outputs.file
2020-12-17T08:58:31Z D! [agent] Starting service inputs
2020-12-17T08:58:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:58:45Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:58:55Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:58:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:05Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:15Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:25Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:35Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
^C2020-12-17T08:59:36Z D! [agent] Stopping service inputs
2020-12-17T08:59:37Z D! [agent] Input channel closed
2020-12-17T08:59:37Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020-12-17T08:59:37Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:37Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-12-17T08:59:37Z D! [agent] Stopped Successfully

Thank you very much

Hello @goodvirus,
Can you write your data to stdout and run the test again so we can see the line protocol? Sorry if I wasn’t clear.

Hello @Anaisdg,

I sorry the output from stdout is

Dez 18 08:38:54 goodvirus telegraf[1081]: 2020-12-18T07:38:54Z W! [outputs.influxdb] Metric buffer overflow; 44 metrics have been dropped
Dez 18 08:38:54 goodvirus telegraf[1081]: 2020-12-18T07:38:54Z E! [outputs.influxdb] When writing to [localhost:8086]: Post "localhost:8086/write?db=telegraf": dial tcp 127.0.0.1:8086: connect: connection refused
Dez 18 08:38:54 goodvirus telegraf[1081]: 2020-12-18T07:38:54Z E! [agent] Error writing to outputs.influxdb: could not write any address

So it seems that the agent wants to write my localhost and ignoring my [[outputs.influxdb_v2]] specified url:
urls = ["https://westeurope-1.azure.cloud2.influxdata.com"]

I also tried running the following command with the same result:
telegraf --config telegraf.config https://westeurope-1.azure.cloud2.influxdata.com/api/v2/telegrafs/06be34c13cff6000

Thank you

Paul

Hello @goodvirus,
I’m sorry I don’t know what’s wrong. That error message

Post http://localhost:8086/write?db=telegraf: dial tcp 127.0.0.1:8086: connect: connection refused

indicates that you have a url of
http://localhost:8086/write?db=telegraf

And that you need to modify the influxdb output in your telegraf.conf to connect using the influxdb hostname:

[[outputs.influxdb]] urls = ["http://influxdb:8086"]

But your url is correct. So I don’t know why it’s throwing that error. I’ll ask the telegraf team if they have any insights.

Hello @Anaisdg,

thank you very much for your help!
Funny think is, when I take the same output configuration and a nother input (for example cpu usage) it works, so it has to do something with the event hub plugin. For example data I´m using the https://azure-samples.github.io/raspberry-pi-web-simulator/ as specified in the example for the plugin.

Best Regards,

Paul

Just an Update, now I reinstalled telegraf and reddit the configuration. Now I don’t get the error message about the localhost, but I still don#t get any data send.

I don’t think I was very clear, can you please include

[[outputs.file]]
  ## Files to write to, "stdout" is a specially handled file.
  files = ["stdout"]

  ## Data format to output.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
  data_format = "influx"

In your configuration so that we can see what the line protocol looks like before you send it to Influx? I expect to see your line protocol in stdout.

You were clear, and I did what you ask for, but there is nothing showing :frowning: (with journalctl - f or I also tried telegraf on windows, nothing in the logs)
But I can see in azure that the messages are going out

@goodvirus,
So it’s solved?

@Anaisdg: No same problem as before and I´m out of ideas…
I see outgoing Traffic, and telegraf says its connected to the event hub and influxdb cloud but it seems to collect the messages but not processing them.
Would it help if I message you the configuration with the secret keys so you could try it out yourself?

Hello @goodvirus,
I’m not sure what’s going on. I’m sorry I haven’t been able to help you. Can you please create an issue in telegraf? GitHub - influxdata/telegraf: The plugin-driven server agent for collecting & reporting metrics.

Thank you.

Hi @Anaisdg,

sorry for the late reply, I was on holidays.
I opend a issue on github, thanks for your help.

Best regards,

Paul

1 Like

Hello @goodvirus,
Hope you had a good holiday. Thanks!

I fixed to issue. The problem was that the input from the event hub was json and telegraf couldn’t prase it so it throw it away without a message. Solution is in the github issue.