I have already completed a project of real-time monitoring by using Python for data ingestion of the CSV file. Since we have completed this but we need know to move ahead with the more complex project and I wish someone has solved this before already.
We are thinking to create a tool in which anyone can dump multiple CSV files, and respective data is stored within Influx with corresponding tags and values at once place i.e. one DB only of each CSV. Different CSV files can be easily segregated using a tag which can state the sources of CSV.
However, each CSV files will have different types of columns and we will not be aware at first place what are the tags and fields of these columns.
So I am stuck here on how to best detect/identify tags and columns automatically for different CSV files.
I have come up with a solution like appending tag_ and field_ in each CSV column and detect that but that seems like inappropriate and not long time solution. Or different functions for each type of CSV that’s unreliable as well.
Hope anyone can help me out here and that would be wonderful