Insert timestamp in _time column using telegraf/eventhub

,
1

I am trying to change the _time column with ‘machine_timestamp’ that I am sending to Telegraf via Eventhub.
Here is a sample of the message I am sending

{‘machine_timestamp’: ‘2024-07-29T14:30:00Z’,
‘measurement’: ‘TEST’,
‘sig’: ‘Temp’,
‘table_name’: ‘High Temp’,
‘value’: 24.93352988307621}

And here is the configuration of my Telegraf
[agent]
collection_jitter = “0s”
debug = true
flush_interval = “1s”
flush_jitter = “0s”
hostname = “”
interval = “1s”
logfile = “”
metric_batch_size = 1000
metric_buffer_limit = 10000
omit_hostname = true
precision = “”
quiet = false
round_interval = true
[[processors.enum]]
[[processors.enum.mapping]]
dest = “status_code”
field = “status”
[processors.enum.mapping.value_mappings]
critical = 3
healthy = 1
problem = 2
[[processors.converter]]
[processors.converter.fields]
float = [
“value”,
“min”,
“max”,
“mean”,
“threesigma”
]

[[outputs.influxdb_v2]]
  bucket = "IFMP"
  organization = "organization"
  token = "$TOKEN"
  urls = [
    "https://urll.com"
  ]

[[inputs.statsd]]
  allowed_pending_messages = 10000
  metric_separator = "_"
  percentile_limit = 1000
  percentiles = [
    50.0,
    95.0,
    99.0
  ]
  service_address = ":8125"
[[inputs.eventhub_consumer]]
  connection_string = "$EH_CONN_STRING"
  data_format = "json"
  json_time_key = "machine_timestamp"
  json_time_format = "RFC3339"
  json_string_fields = [
    "value",
    "min",
    "max",
    "mean",
    "threesigma"
  ]
  name_override = "Measurement_Name"
  tag_keys = [
    "sig",
    "table_name",
    "mountpoint",
    "interface"
  ]

When I use this config, I see from the logs that telegraf output data to influxDB
2024-07-29T22:08:17Z D! [outputs.influxdb_v2] Wrote batch of 300 metrics in 18.539421ms
but I see no data in influxDB
When I remove the json_time_key and json_time_format lines the data is written, but obviously I am not replacing the _time column value with anything.

How can I insert my own timestamps to influxDB so that they show up under the _time column?

2

Hello @alsahmou,

[[inputs.eventhub_consumer]]
  connection_string = "$EH_CONN_STRING"
  data_format = "json"
  json_time_key = "machine_timestamp"  # This should match your JSON timestamp key
  json_time_format = "2006-01-02T15:04:05Z07:00"  # This is the layout for RFC3339
  json_string_fields = [
    "value",
    "min",
    "max",
    "mean",
    "threesigma"
  ]
  name_override = "Measurement_Name"
  tag_keys = [
    "sig",
    "table_name",
    "mountpoint",
    "interface"
  ]

[[outputs.influxdb_v2]]
  bucket = "IFMP"
  organization = "organization"
  token = "$TOKEN"
  urls = [
    "https://urll.com"
  ]

You could try making the layout format like above but the format you chose should work.

You could also try using the json_v2 data format instead. But that should work. I don’t see anything wrong off the bat.
@jpowers do you? Thank you!

Ps @alsahmou what are you usign telegraf and influxdb for? I like learning about what community members are getting up to:)

3

Hi,

I think @alsahmou asked this in slack as well, so I’ll paste my response here:

When someone says it doesn’t show up in influxdb, but the logs say that it was written, this is usually a UI issue with whatever your are querying influxdb with. For example, your dates is from yesterday, does your UI display more than 24 hours of data? or if you are using grafana or something else, same question.

We ask this because when you don’t set the timestamp, then the current time is used, and if you can see that it is much more likely that recent data points are shown in the default UI time range

I think @alsahmou came back and said this was the issue, namely the UI scale was only the last 10 minutes.

Hope that helps!