Telegraf csv unsupported type problem

[andbos]

Hi,

I’m trying to get CSV data into InfluxDB using Telegraf and inputs.tail. For several applications it works but for one I get “unsupported type”. Debug doesn’t say much more but it might possibly have to do with the timestamp format. Timestamp can be for instance 20200601070000 so in the telegraf.conf I configured

csv_timestamp_format = “20060102150405”

Is this a valid format?

Example from log file:

2020-06-01T07:30:00Z E! [inputs.tail] Malformed log line in “/var/log/metrics.csv”: [“20200601071500,app_01,900,f,6,0,2,0,4,4,0,2,0,0,2,2,2,170,1010,0,2,0,0,1,0,0,0,1,1,0,0,15,18,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.011111111,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0”]: unsupported type

From the configuration file:

csv_column_types = [“int”,“string”,“int”,“string”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”,“float”]
csv_timestamp_column = “timestamp”
csv_timestamp_format = “20060102150405”

Best regards,
Andreas

[Anaisdg]

Hello @andbos,
Welcome!
Unfortunately that is not a supported timestamp option.
The csv_timestamp_column option specifies the key containing the time value and csv_timestamp_format must be set to unix , unix_ms , unix_us , unix_ns , or a format string in using the Go “reference time” , "2006-01-02T15:04:05Z07:00", which is defined to be the specific time : Mon Jan 2 15:04:05 MST 2006 .

[andbos]

Hi,

Thank you. I had hoped that csv_timestamp_format = “20060102150405” would be that specific time but yes, timezone is missing.
Would it maybe be possible to modify the timestamp before writing to InfluxDB? I have seen example using inputs.logparser.grok but how to combine that with inputs.tail?

Best regards,
Andreas

[daniel]

The csv_timestamp_format you have should work, I think you just need to set the timestamp column to parse as a string in csv_column_types:

csv_column_types = ["string", ...]

[andbos]

Thanks, Daniel. Works like a charm now