Telegraf CSV file, returning only one row in prometheus, not returning as timestamp based

1

Hi Team,

I am trying to convert csv data in time series, however only last data populating in Prometheus(both telegraf and prometheus running in same server). Can someone help me, what I am missing.
My telegraf config:
[agent]
interval = “10s”
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “0s”
debug = false
quiet = false

# Read metrics from one or more CSV files

[[inputs.file]]

Files to parse each interval.

files = [“/usr/temperature.csv”]

File will be read in every interval.

data_format = “csv”
csv_header_row_count=1

Specify a time format to use for parsing time from CSV

csv_timestamp_column = “timestamp”
csv_timestamp_format = “2006-01-02T15:04:05Z07:00”

# Configuration for Prometheus client to expose metrics for scraping

[[outputs.prometheus_client]]
listen = “:9273”
temperature.csv -
timestamp,temperature,room,device
2024-06-27T12:00:00Z,22.5,room1,device1
2024-05-28T12:05:00Z,22.7,room1,device1
2024-04-22T12:10:00Z,22.8,room1,device1
2024-03-21T12:15:00Z,22.6,room1,device1
2024-02-20T12:20:00Z,22.5,room1,device1
2024-01-23T12:00:00Z,23.0,room2,device2
2024-06-29T12:05:00Z,23.1,room2,device2

below details, I have added for prometheus.yml file -
scrape_configs:

  • job_name: ‘telegraf’
    static_configs:
    • targets: [‘localhost:9273’]
2

Hello @Yuvaraj_A.N,
Can you make sure your timestamp is formatted correctly in your telegraf config?

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = true
  quiet = false

[[inputs.file]]
  files = ["/usr/temperature.csv"]
  data_format = "csv"
  csv_header_row_count = 1
  csv_timestamp_column = "timestamp"
  csv_timestamp_format = "2006-01-02T15:04:05Z07:00"

[[outputs.prometheus_client]]
  listen = ":9273"
  metric_version = 2

Also can you add debug=true in the agent portion of your config?
Are you sure that the csv is being updated?

3

Thanks for reply. Please find my updated config -

yuvaannie@instance-20240629-014749:/usr$ cat /etc/telegraf/telegraf.conf
[agent]
interval = “10s”
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “0s”
debug = false
quiet = false
logfile = “”
hostname = “”
[[inputs.file]]
files = [“/usr/temperature.csv”]
data_format = “csv”
csv_header_row_count=1
csv_delimiter = “,”
csv_tag_columns = [ “room”, “device”]
csv_timestamp_column = “timestamp”
csv_timestamp_format = “2006-01-02T15:04:05Z”
[[outputs.prometheus_client]]
listen = “:9273”
metric_version = 2
[[outputs.file]]
files = [“stdout”, “usr/file.out”]

input file -
yuvaannie@instance-20240629-014749:/usr$ cat temperature.csv

timestamp,temperature,room,device
2024-06-27T12:00:00Z,22.5,room1,device1
2024-05-28T12:05:00Z,22.7,room1,device1
2024-04-22T12:10:00Z,22.8,room1,device1
2024-03-21T12:15:00Z,22.6,room1,device1
2024-02-20T12:20:00Z,22.5,room1,device1
2024-01-23T12:00:00Z,23.0,room2,device2
2024-06-29T12:05:00Z,23.1,room2,device2

In out file - below is printing
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.6
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.5
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device2”,host=“instance-20240629-014749”,room=“room2”} 23
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:41 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device2”,host=“instance-20240629-014749”,room=“room2”} 23.1
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.5
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.7
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.8
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.6
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.5
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # HELP file_temperature Telegraf collected metric
Jul 01 15:12:51 instance-20240629-014749 telegraf[1863]: # TYPE file_temperature untyped

but Prometheus considering as duplicate and display, only one value.
How can I get all data based on timestamp

HELP file_temperature Telegraf collected metric

TYPE file_temperature untyped

file_temperature{device=“device1”,host=“instance-20240629-014749”,room=“room1”} 22.5
file_temperature{device=“device2”,host=“instance-20240629-014749”,room=“room2”} 23.1

I tried adding “temperature” as tag - No output coming.
by adding timestamp - getting below error
yuvaannie@instance-20240629-014749:/usr$ telegraf --config /etc/telegraf/telegraf.conf --test
2024-07-01T15:15:21Z I! Starting Telegraf 1.19.3
2024-07-01T15:15:21Z E! [inputs.file] Error in plugin: timestamp column: timestamp could not be found
2024-07-01T15:15:21Z E! [telegraf] Error running agent: input plugins recorded 1 errors

How to get all timestamp values in promotheus?