Reading CSV Files into Influxdb that are stored on a FTP Server

Globgogabgalab · January 5, 2023, 1:48pm

Hi all,

I am trying to read CSV files on a FTP server (protected with username and password) into Influxdb. These files are produced constantly. I have 1 file for each day plus a the csv file for today, which is filled with data up to now.

Please see below the structure of the csv files:

(s)	Date	Time	Lue01.An.ABL_T.Istwert	Lue01.An.ABL_rF.Istwert	Lue01.An.ABL_aF.Istwert	Lue01.An.ABL_Tp.Istwert
1672876809	05.01.2023	00:00:09	29.8	52.8	14.8	19.5
1672876819	05.01.2023	00:00:19	29.7	52.8	14.8	19.4
1672876829	05.01.2023	00:00:29	29.7	52.9	14.8	19.4
…	…	…	…	…	…	…
1672928879	05.01.2023	14:27:59	29.7	48	13.4	17.7
1672928889	05.01.2023	14:28:09	29.7	47.9	13.3	17.7

The URL of the files looks like this:

http://192.168.X.X/M2_FLASH/WEBPAGES/DATA/LUE01_ABL_D230105.CSV

Is there a way to capture this data in indluxdb?

Many thanks in advance!

grant1 · January 5, 2023, 3:55pm

Hi @Globgogabgalab

Very easy to do with Node-RED. Have you explored that product at all?

Globgogabgalab · January 6, 2023, 8:21am

Hi @grant1 ,

Thanks for the tip! I am only using the node-red application in my home assistant setup to automate my zigbee lights. Is there a tutorial out there on how to feed that data into influxdb?

Many thanks and best regards

grant1 · January 6, 2023, 10:16am

Not that I am aware of, but I will send you via private message a few links and the Node-RED forum folks can probably help you along if you get stuck.

Jay_Clifford · January 6, 2023, 11:19am

Hi @Globgogabgalab,
You can also use Telegraf for this if you want a more performant solution. I agree though Node-Red is the easiest! I am a massive fan!

github.com

influxdata/telegraf/blob/release-1.24/plugins/inputs/directory_monitor/README.md

# Directory Monitor Input Plugin

This plugin monitors a single directory (traversing sub-directories),
and takes in each file placed in the directory.  The plugin will gather all
files in the directory at the configured interval, and parse the ones that
haven't been picked up yet.

This plugin is intended to read files that are moved or copied to the monitored
directory, and thus files should also not be used by another process or else
they may fail to be gathered. Please be advised that this plugin pulls files
directly after they've been in the directory for the length of the configurable
`directory_duration_threshold`, and thus files should not be written 'live' to
the monitored directory. If you absolutely must write files directly, they must
be guaranteed to finish writing before the `directory_duration_threshold`.

## Global configuration options <!-- @/docs/includes/plugin_config.md -->

In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.

This file has been truncated. show original

Directory Monitor allows you to monitor a specific directory and reads any csv’s you wish to parse there before placing them in a separate directory

Globgogabgalab · January 6, 2023, 4:32pm

Hi @Jay_Clifford ,

Thank you very much for your help! I have checked out the documentation but I am still unsure how to enter the URL/IP and the username/password of the FTP server next to the path.

Can you help me out here?

Jay_Clifford · January 9, 2023, 11:56am

Ah @Globgogabgalab, sorry that was my misunderstanding.

So there is currently no telegraf plugin which will allow you to read files directly from an FTP server. So i agree with @grant1 I woudl use node-red for the easiest approach and then use the InfluxDB plugin for node red to write that data to influxDB. Otherwise you would need to use the execd plugin of Telegraf and write a custom script to copy over the file and parse in telegraf. Note if telegraf can be run directly on your FTP server then you can use the directory plugin.