Hi,
I will be regularly receiving 2 csv files, each of which has the same number of rows. The rows between the files are related 1:1 using a key field. File1 has the time field in it that I need and file2 has the measurement field that I want.
I have successfully managed to use telegraf to process file2 to extract the field I need since I need to use the regex processor.
What I would like to know though is what is the best practice for merging the two csv files so that I have a single file to process for import. Is this something you’d expect to be done prior to importing?
Typically it wouldn’t be a big problem to write a script to do that, but my sample files are 1 million lines each, so I was wondering if telegraf can do that or if there are any other bulk import tools that would allow me to do that before I go ahead and create something. Trying to import the first file using telegraf, I received a lot of drops which concerned me quite a lot too.
As a side note, given the number of rows, a 10 second interval isn’t long enough, but setting that to 30 minutes or something means waiting 30 minutes before it starts processing for the first time – is there anything to help with that? Or can I just ignore the “it didn’t complete in the interval” message.
Final side question – does Telegraf have the option to remove the CSV files once they have been successfully imported?
Many thanks for any assistance.
Kind regards,
D.
