Telegraf csv_timestamp_format to parse a time step

Hello all,
New to telegraf and influxdb.

Followed a very useful tutorial posted by @Anaisdg and managed to reproduce it.
Now I’d like to try telegraf on my own data.
But struggle to understand the timestamp_format bit in the configuration part.

Here is an example of the csv I tried to digest via telegraf

"TOA5","StonyLysimetersDrain","CR1000","5308","CR1000.Std.25","CPU:StonyLysimeter_Drainage.CR1","22253","StonyLysimetersDrain"
"TIMESTAMP","RECORD","BattV_Avg","PTemp_C_Avg","Drain_Tot(1)","Drain_Tot(2)","Drain_Tot(3)","Drain_Tot(4)","Drain_Tot(5)","Drain_Tot(6)","Drain_Tot(7)","Drain_Tot(8)","Drain_Tot(9)","Drain_Tot(10)","Drain_Tot(11)","Drain_Tot(12)","Drain_Tot(13)","Drain_Tot(14)","Drain_Tot(15)","Drain_Tot(16)","Drain_Tot(17)","Drain_Tot(18)","Drain_Tot(19)","Drain_Tot(20)","Drain_Tot(21)","Drain_Tot(22)","Drain_Tot(23)","Drain_Tot(24)","AirTC_Avg","RH_Max","SlrkW_Avg","SlrMJ_Tot","Rain_mm_Tot","WS_ms_Avg"
"TS","RN","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","",""
"","","Avg","Avg","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Tot","Avg","Max","Avg","Tot","Tot","Avg"
"2015-06-19 16:15:00",,,,3,5,4,3,3,4,4,3,4,4,3,0,3,4,3,3,NAN,NAN,NAN,NAN,NAN,NAN,NAN,NAN,-31.71,15.87,NAN,NAN,0,0
"2015-06-19 16:30:00",,,,4,4,4,3,2,4,5,3,3,4,3,0,3,5,3,4,3,3,3,4,3,4,4,3,-31.91,15.46,NAN,NAN,0,0
"2015-06-19 16:45:00",,,,3,4,4,3,3,4,5,3,3,3,3,0,3,4,2,4,4,4,3,4,4,3,4,2,-32.25,15.16,NAN,NAN,0,0

Here is the input plugin config:

data_format = "csv"

csv_header_row_count = 1
 csv_skip_rows = 1
csv_skip_columns = 0
csv_comment = "#"

Guess I could use this config to subset column?

csv_measurement_column = ["Drain_Tot(1)","Drain_Tot(2)","Drain_Tot(3)","Drain_Tot(4)","Drain_Tot(5)","Drain_Tot(6)","Drain_Tot(7)","Drain_Tot(8)","Drain_Tot(9)","Drain_Tot(10)","Drain_Tot(11)","Drain_Tot(12)","Drain_Tot(13)","Drain_Tot(14)","Drain_Tot(15)","Drain_Tot(16)","Drain_Tot(17)","Drain_Tot(18)","Drain_Tot(19)","Drain_Tot(20)","Drain_Tot(21)","Drain_Tot(22)","Drain_Tot(23)","Drain_Tot(24)","AirTC_Avg","RH_Max","SlrkW_Avg","SlrMJ_Tot","Rain_mm_Tot","WS_ms_Avg"]


csv_timestamp_column = "TIMESTAMP"
csv_timestamp_format = "2015-06-19 16:15:00"

The log file after I run telegraf --config file.conf

2020-11-07T10:19:05Z D! [agent] Initializing plugins
2020-11-07T10:19:05Z D! [agent] Connecting outputs
2020-11-07T10:19:05Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-07T10:19:05Z D! [agent] Successfully connected to outputs.influxdb
2020-11-07T10:19:05Z D! [agent] Starting service inputs
2020-11-07T10:19:10Z E! [inputs.file] Error in plugin: parsing time "TS" as "2015-06-19 16:15:00": cannot parse "TS" as "2"

Thank you in advance for any feedback.

Solution found after a bit more playing.

The documentation here states that the time format must be specified as Go reference time.
So changed the config csv_timestamp_format = "2006-01-02 15:04:05"

The error Error in plugin: parsing time "TS" as "2015-06-19 16:15:00": cannot parse "TS" as "2" was caused by a newline.

3 Likes

Thanks. I spent hours trying to get this right.