I am trying to read CSV files on a FTP server (protected with username and password) into Influxdb. These files are produced constantly. I have 1 file for each day plus a the csv file for today, which is filled with data up to now.
Thanks for the tip! I am only using the node-red application in my home assistant setup to automate my zigbee lights. Is there a tutorial out there on how to feed that data into influxdb?
Hi @Globgogabgalab,
You can also use Telegraf for this if you want a more performant solution. I agree though Node-Red is the easiest! I am a massive fan!
Directory Monitor allows you to monitor a specific directory and reads any csv’s you wish to parse there before placing them in a separate directory
Thank you very much for your help! I have checked out the documentation but I am still unsure how to enter the URL/IP and the username/password of the FTP server next to the path.
So there is currently no telegraf plugin which will allow you to read files directly from an FTP server. So i agree with @grant1 I woudl use node-red for the easiest approach and then use the InfluxDB plugin for node red to write that data to influxDB. Otherwise you would need to use the execd plugin of Telegraf and write a custom script to copy over the file and parse in telegraf. Note if telegraf can be run directly on your FTP server then you can use the directory plugin.
But one problem i have.
The inputs in the text files are in the fromat “milliseconds”
in witch place in the telegraf.config file
i can say the format is precision = “ms” from the input??
[[inputs.directory_monitor]]
## The directory to monitor and read files from.
directory = "/www/Huber"
#
## The directory to move finished files to.
finished_directory = "/www/Huber/writte"
#
## The directory to move files to upon file error.
## If not provided, erroring files will stay in the monitored directory.
# error_directory = ""
#
## The amount of time a file is allowed to sit in the directory before it is picked up.
## This time can generally be low but if you choose to have a very large file written to the directory and it's potentially slow,
## set this higher so that the plugin will wait until the file is fully copied to the directory.
# directory_duration_threshold = "50ms"
#
## A list of the only file names to monitor, if necessary. Supports regex. If left blank, all files are ingested.
# files_to_monitor = ["^.*\.csv"]
#
## A list of files to ignore, if necessary. Supports regex.
# files_to_ignore = [".DS_Store"]
#
## Maximum lines of the file to process that have not yet be written by the
## output. For best throughput set to the size of the output's metric_buffer_limit.
## Warning: setting this number higher than the output's metric_buffer_limit can cause dropped metrics.
# max_buffered_metrics = 10000
#
## The maximum amount of file paths to queue up for processing at once, before waiting until files are processed to find more files.
## Lowering this value will result in *slightly* less memory use, with a potential sacrifice in speed efficiency, if absolutely necessary.
# file_queue_size = 100000
#
## Name a tag containing the name of the file the data was parsed from. Leave empty
## to disable. Cautious when file name variation is high, this can increase the cardinality
## significantly. Read more about cardinality here:
## https://docs.influxdata.com/influxdb/cloud/reference/glossary/#series-cardinality
# file_tag = ""
#
## Specify if the file can be read completely at once or if it needs to be read line by line (default).
## Possible values: "line-by-line", "at-once"
parse_method = "line-by-line"
#
## The dataformat to be read from the files.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"
