CSV file data into InfluxDB via Telegraf

Telegraf
,
1

Hi,
I’m having a problem with getting csv data into InfluxDB via the Telegraf files.csv plugin.

This is the config:

## CSV file test
# https://docs.influxdata.com/telegraf/v1.9/data_formats/input/csv/
[[inputs.file]]
files = ["/ubs/blue/autotest/var/tmp/clamp1.csv"]
data_format = "csv"
csv_header_row_count = 1
csv_delimiter = ","
csv_tag_columns = ["REGION"]
csv_measurement_column = "METRIC"
csv_timestamp_column = "DATETIMESTAMP"
# timestamp example "20190225T18:26:53.700430"
csv_timestamp_format = "20060102T15:04:05.000000"

This is a few lines out of the the file:
DATETIMESTAMP,REGION,METRIC,LATENCYNS 20190304T16:03:55.266548,Clamp Simulator,latency,309474238 20190304T16:03:56.266697,Clamp Simulator,latency,162048311

And this is the result from select * from latency
1551717149000000000 20190304T16:32:28.978691 555844021 latency Clamp Simulator test_host 1551717150000000000 20190304T16:32:29.979692 880542363 latency Clamp Simulator test_host

It looks like its stored all the fields in the csv as one column so I’m unable to display and plot the value (latencyns).#

Any clues about what I have done wrong?

Kind regards

Richard Lucas

2

hi , does your csv file has newlines as follows ?

DATETIMESTAMP,REGION,METRIC,LATENCYNS
20190304T16:03:55.266548,Clamp Simulator,latency,309474238
20190304T16:03:56.266697,Clamp Simulator,latency,162048311

also the date timestamp should be different , else you will have only one record at the end if all timestamps and tags are the same

please check also the required date format ( example : 2019-03-04T16:05:56Z )

The  `csv_timestamp_column`  option specifies the column name containing the time value and  `csv_timestamp_format`  must be set to a Go “reference time” which is defined to be the specific time:  `Mon Jan 2 15:04:05 MST 2006` .

Consult the Go [time][time parse] package for details and additional examples on how to set the time format.
1 Like
3

Hi Marc,

Yes, it does contain new lines - I’d mangled the MD… It should look like this.

DATETIMESTAMP,REGION,METRIC,LATENCYNS
20190304T16:03:55.266548,Clamp Simulator,latency,309474238
20190304T16:03:56.266697,Clamp Simulator,latency,162048311

The time-stamp varies by 1 second per record in this test file.

If you have any ideas, they would be appreciated.

Kind regards,

Richard

4

You can try changing the date format in the config and the csv file as follows :
2019-03-04T16:05:56Z

5

I’ll try that as a test, but we will have multiple measurements per second, so will need to know how to handle that.

6

The timestamp format is correct, it needs to be this particular date in 2006. This is the “reference time” in Go.

I tried your example and it looks like it is working to me, each column is parsed out separately:

$ telegraf --input-filter file --test
> latency,REGION=Clamp\ Simulator,host=loaner DATETIMESTAMP="20190304T16:03:55.266548",LATENCYNS=309474238i,METRIC="latency" 1551715435000000000
> latency,REGION=Clamp\ Simulator,host=loaner DATETIMESTAMP="20190304T16:03:56.266697",LATENCYNS=162048311i,METRIC="latency" 1551715436000000000

> select * from latency
name: latency
time                DATETIMESTAMP            LATENCYNS METRIC  REGION          host
----                -------------            --------- ------  ------          ----
1551715435000000000 20190304T16:03:55.266548 309474238 latency Clamp Simulator loaner
1551715436000000000 20190304T16:03:56.266697 162048311 latency Clamp Simulator loaner
7

Hi Richard , is your problem solved ?

what version of telegraf are you using ( i tested with 1.9.2 and it works )

This link describes your problem and it seems that if you comment the line
csv_delimiter = “,” , it will work …

Or you can upgrade to the latest version … ?

csv_delimiter issue in telegraf 1.8.0

8

Hi Marc

The environment is RHEL7, running Telegraf 1.9.4. I’m going to be trying it it again as soon as the morning calls have finished. It encouraging that it works for others.

I’ll let you know how I get on.

Thanks
Richard

9

Hi Marc, Daniel,

Many thanks for the clues you gave me. Turns out that it was working, but I just didn’t understand the result of the influx query. After comparing that with Daniel’s examples, I could see the problem was myself, not Telegraf or Influx.

I owe you beer points for your time spend helping me out

Many thanks,

Richard

1 Like
10

Hello,

I want to use an FTP server as input. My goal is therefore that Telegraph collects all CSV data from my FTP server and then transmits it to InfluxdB. How can I do that ?

Please help me.

Best Regards

Camel