Structured csv input with variable contents

Telegraf
1

Hi,
There is a task to index csv structured files where the structure depends on one or several fields. For example in one file you may have:

field1,field2,keyfield1,field3,…,fieldN
field1,field2,keyfield2,anotherfield3,anotherfield4,…,anotherfieldM

Each record set and length can be completely different except first common fields and keyfields, which we know. We also know the headers set for each keyfield.
Is there a way to help telegraf automatically process the files depending on the keyfield value and output to influxdb?

Thank you!
Alex

2

Does each file have a header line with the names of the columns?

3

Hmm. That would be too easy. Files does not have headers row)

4

So the idea is that based on a column the remaining columns would have certain names?

If I’m understanding correctly then it can’t be done. The only thing you could do here is give an explicit list of header names to the plugin, with good names for the common prefix columns and bad names like: field1, field2, field3 to the remaining columns. If you could arrange for the different types to be in differently named files you might have some other options.

5

Yes, that was my first try - to distribute different sets by different files. The hope was to avoid this additional processing.
Thank you for confirming the idea)