Telegraf - influxDB : inputs.file / inputs.tail csv format

Dear All,

I am new to this community.
I am trying to use telegraf-influxdb to collect data from an application that, in real time, delivers GPS data in three components. The application generates a CSV file of the form:

[“site_id”, “time”, “Xref”, “Yref”, “Zref”, “sat_number”, “X”, “Y”, “Z”, “E”, “N”, “U”]

and write 1 csv file for each “site_id” (measurement station) … every one second, the csv file has a new line from the application with the information of that second. My idea is to graph for each site_id, three graphs :frowning: X, Y, Z vs time) with grafana.
Example of one CSV file:

An example of each line in a CSV file:

STN1, 2020-03-28T05: 07: 59Z, 12,1757845.006700, -4994163.130100, -3545806.556600,0,1757845.012783, -4994163.118345, -3545806.553619,0.009641, -0.002598,1757845.006700
STN1, 2020-03-28T05: 08: 00Z, 12,1757845.006700, -4994163.130100, -3545806.556600,536870912,1757845.010804, -4994163.119273, -3545806.549541,0.007466,0.000905,1757845.006700

I have tried the input plugin [[inputs.file]]:

[[inputs.file]]
files = ["/home/rodrigo/GPSRT/DATA/*.csv"]
data_format = “csv”
csv_header_row_count = 0
csv_column_names = [“site_id”,“time”,“Xref”,“Yref”,“Zref”,“sat_number”,“X”,“Y”,“Z”,“E”,“N”,“U”]
csv_skip_rows = 0
csv_skip_columns = 0
csv_delimiter = “,”
csv_comment = “”
csv_trim_space = false
csv_tag_columns = [“site_id”]
csv_timestamp_column = “time”
csv_timestamp_format = “unix”

When I have tried to start the telegraph service, I have the following errors:

telegraf --debug --config telegraf.conf
2020-03-30T15:52:23Z I! Starting Telegraf 1.13.4
2020-03-30T15:52:23Z I! Loaded inputs: file
2020-03-30T15:52:23Z I! Loaded aggregators:
2020-03-30T15:52:23Z I! Loaded processors:
2020-03-30T15:52:23Z I! Loaded outputs: influxdb
2020-03-30T15:52:23Z I! Tags enabled: host=tesla.csn.uchile.cl user=rodrigo
2020-03-30T15:52:23Z I! [agent] Config: Interval:1s, Quiet:false, Hostname:“tesla.csn.uchile.cl”, Flush Interval:2s
2020-03-30T15:52:23Z D! [agent] Initializing plugins
2020-03-30T15:52:23Z D! [agent] Connecting outputs
2020-03-30T15:52:23Z D! [agent] Attempting connection to [outputs.influxdb]
2020-03-30T15:52:23Z D! [agent] Successfully connected to outputs.influxdb
2020-03-30T15:52:23Z D! [agent] Starting service inputs
2020-03-30T15:52:24Z E! [inputs.file] Error in plugin: strconv.ParseInt: parsing “2020-03-28T05:07:44Z”: invalid syntax
2020-03-30T15:52:25Z E! [inputs.file] Error in plugin: strconv.ParseInt: parsing “2020-03-28T05:07:44Z”: invalid syntax
2020-03-30T15:52:26Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-03-30T15:52:26Z E! [inputs.file] Error in plugin: strconv.ParseInt: parsing “2020-03-28T05:07:44Z”: invalid syntax
2020-03-30T15:52:27Z E! [inputs.file] Error in plugin: strconv.ParseInt: parsing “2020-03-28T05:07:44Z”: invalid syntax
2020-03-30T15:52:28Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-03-30T15:52:28Z E! [inputs.file] Error in plugin: strconv.ParseInt: parsing “2020-03-28T05:07:44Z”: invalid syntax
2020-03-30T15:52:29Z E! [inputs.file] Error in plugin: strconv.ParseInt: parsing “2020-03-28T05:07:44Z”: invalid syntax
2020-03-30T15:52:30Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics

I really don’t know if [file] plugin is the best option, maybe [tail] … and I don’t understand what I’m doing wrong with the telegraph options …

Any guide or help is very welcome

NOTE:
I can modify the way the variable “time” is written to the CSV file

Hello @rumekintun,
Welcome! Not sure if this is a typo, but looking at your CSV it looks like there are spaces in your timestamp where:
2020-03-28T05: 07: 59Z
should be
2020-03-28T05:07:59Z
could that be the problem?

Hi Ana…
nop …
STN,2020-03-28T05:08:04Z,15,…
there are no spaces…
I don’t need quotes to surround the “time” field?
or may be the site_id field ?

The options …

csv_timestamp_column = “time”
csv_timestamp_format = “unix”

are ok ?

I’m really confused about that …

thanks a lot for your answer …

I think you need to use csv_timestamp_format = “2006-01-02T15:04:05Z”

Thanks anythony… but… the same …
bu the way
Linux:
CentOS Linux release 7.7.1908 (Core)
telegraf-1.13.4-1.x86_64
influxdb-1.7.10-1.x86_64


telegraf --debug --config telegraf.conf
2020-03-30T21:01:10Z I! Starting Telegraf 1.13.4
2020-03-30T21:01:10Z I! Loaded inputs: file
2020-03-30T21:01:10Z I! Loaded aggregators:
2020-03-30T21:01:10Z I! Loaded processors:
2020-03-30T21:01:10Z I! Loaded outputs: influxdb
2020-03-30T21:01:10Z I! Tags enabled: host=tesla.csn.uchile.cl user=rodrigo
2020-03-30T21:01:10Z I! [agent] Config: Interval:1s, Quiet:false, Hostname:“xxx.xxx.xx.xx”, Flush Interval:2s
2020-03-30T21:01:10Z D! [agent] Initializing plugins
2020-03-30T21:01:10Z D! [agent] Connecting outputs
2020-03-30T21:01:10Z D! [agent] Attempting connection to [outputs.influxdb]
2020-03-30T21:01:10Z D! [agent] Successfully connected to outputs.influxdb
2020-03-30T21:01:10Z D! [agent] Starting service inputs
2020-03-30T21:01:11Z E! [inputs.file] Error in plugin: parsing time “2020-03-28T06:30:60Z”: second out of range
2020-03-30T21:01:12Z E! [inputs.file] Error in plugin: parsing time “2020-03-28T06:30:60Z”: second out of range
2020-03-30T21:01:13Z E! [inputs.file] Error in plugin: parsing time “2020-03-28T06:30:60Z”: second out of range
2020-03-30T21:01:14Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-03-30T21:01:14Z E! [inputs.file] Error in plugin: parsing time “2020-03-28T06:30:60Z”: second out of range
2020-03-30T21:01:15Z E! [inputs.file] Error in plugin: parsing time “2020-03-28T06:30:60Z”: second out of range
2020-03-30T21:01:16Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics

wow…
2020-03-28T06:30:60Z
_______________^^^^==> there is something wrong … :crazy_face:

Haha yeah, it seems like telegraf parses the timestamp now, but “60” seconds should become 1 minute. You might want to check the CSV file again to see if those times actually show up.

1 Like