Easiest way (for a beginner) to import csv data with time

Telegraf
1

hello all,

I have been trying to find easy ways to import data from a CSV file to influxdb, as I am a novice to all of this.

I have looked through older posts, also on other sources on the web and there doesn’t appear to be a very easy way to import data to influxdb for a novice like myself.
I currently have influx installed on my windows 10 laptop, but also on an ubuntu VPS, that captures IoT data from devices (temp, humidity, etc). The latter one built by someone else, so I can access the VPS through SSH, but I mostly use Grafana to visualise and nothing more.

It would be extremely useful to me to be able to add additional data to influxdb, as I have a lot of data from other non-internet connected sensors in the form of csv and using influxDB to store them all and visualising the via grafana would be so much more effective than my current practices.
The data I have is typically like this:
Time, Temp, RH, (followed by other metrics sometime).
A lot of this is historical/older data.

Using InfluxDB for a while has proven extremely valuable for the type of work that I do so far, so ideally I would like to import all other data I have into InfluxDB and manage and visualise them from there.

With my current skill level it seems almost impossible for me to follow some tutorials/instructions I’ve seen. I do get stuck at some point, or if I do manage to complete something the import hasn’t worked.
It would be good for me to understand if InfluxDB is, or is not the right tool for me, considering my skill level.

Does anyone know of a very simple way to import CSV data from a file into an InfluxDB database?
Perhaps there has been some developments recently that olders posts don’t cover.

thanks

1 Like
2

IMHO the easiest way is to read the csv files with Telegraf.
Of course you have to change the telegraf.conf accordingly.
With a inputs.file plugin and the CSV input data format most csv files should be readable.
But this also strongly depends on how the csv files and the contained data are formatted.
A little tricky is sometimes the timestamp.

Here is an example of what such a configuration might look like:

[[inputs.file]]
  files = ["mydata.csv"]
  data_format = "csv"
  csv_header_row_count = 1  # first line is header
  csv_timestamp_column = "timestamp"
  csv_timestamp_format = "2006-01-02T15:04:05Z07:00"
  name_override = "mydata"

Then you can run Telegraf with the --once option and you have the corresponding csv files in InfluxDB:

telegraf.exe --config telegraf.conf --once

Are your csv files all structured the same or very different?
It would be best to post a few lines of your various csv files here, then we can better judge whether and how difficult the csv import could be. Please post the csv snippets here in markdown format:

```csv
put 2-3 lines of a csv file here as an example
```
3

Thank you for your detailed response and for taking the time to answer in this detail.

I am afraid I am in a minority in this forum, as I am indeed a complete novice and you will confirm that from my reply below.

Unfortunately, the instructions appear to be the same as the ones I am reading in various tutorials and articles with which I get stuck all the time. I think my skill level is not suited for using InfluxDB properly as I do not do any coding, not am I fluent it it and I can only really use instructions step-by-step as I read them. If the instructions don’t work, or assume a range of pre-requisite steps that are not described step by step, or require additional background knowledge to debug problems, then I get completely stuck. I think a certain level of competence in coding, as well as familiarity with databases is needed in order to use InfluxDB, which I don’t have.

I struggle to interpret instructions such as:
‘Of course you have to change the telegraf.conf accordingly. With a inputs.file plugin and the CSV input data format most csv files should be readable’.
I have found the githb link for the inputs.file plugin (telegraf/plugins/inputs/file at master · influxdata/telegraf · GitHub) , but I don’t know what to do with it. Where do I install? Where do I put those github files on my windows 10 laptop?

It would help a lot if there was a more straightforward process for complete novices like myself who would greatly benefit by utilising InfluxDB to visualise all their data for analysis, but are not skilled in coding.
For example, something along the lines of: ‘format all your CSV data files in this way and then use this command to input your data into influxDB’. For myself and many others, formatting our csv data (say using EXCEL) is not a problem, we can change the format to whatever is needed to input to influxdb.

Most of my CSV data have 1 line at the top which describes what the data below is.
(I do apologise, I don’t think I have manage to make the markdown format to work)
For example:
date,temperature,humidity,co2
2021-03-10 09:30,25,50,575
2021-03-10 10:00,25,52,600

Other csv files could have multiple rows at the top to describe the data below (for example device id numbers, project, location, etc. etc.). In the above example there are 3 metrics recorded, but other files might have only 1 metric (temperature), whereas others might have 6-10 metrics. I hope this makes sense.

In any case, for me and many others in my field editing the CSV files to make them all more suited for importing to InfluxDB is not a problem. We can change the ‘date’ format to a different one, we can make all of the csv files with only 1 header row, or none. Whatever would make the process easier to import to InfluxDB.

4

Phew, so if your knowledge level is that low, it’s going to be really difficult. I wonder if this is the right tool for you? InfluxDB and Grafana are not Microsoft Excel. Even if you have all the csv files in InfluxDB, you will have problems writing the queries in InfluxDB or Grafana.

You should have a certain willingness to read in and deal with it. You don’t necessarily have to be able to program, but you should be able to adapt e.g. the config file for Telegraf. And if you want to visualize the data in InfluxDB or Grafana, you might have to deal with the flux query language.

There is no upload button in the InfluxDB GUI to upload csv files. The only thing you can upload in the GUI so far are files in influx line protocol format.
To get your csv files into InfluxDB, I would use the Telegraf agent. Please read up on Telegraf or watch some Youtube tutorials about it.

5

Thanks again for your reply and for taking the time.

I currently use Grafana connected to InfluxDB on a VPS and I can plot all graphs I need on Grafana, so that seems easy enough through the interface and it currently all works for what I need.
I also have Grafana, InfluxDB and Telegraf installed locally on my Window 10 laptop and I can connect from Grafana to InfluxDB locally and create the charts that I need. Problems for people like me start when I start wanting to import CSV files, etc. where the concepts of using a plugin for Telegraf, etc. come into play.

It seems that for what I need InfluxDB is probably not the right tool for me, but I will give it another try with the documentation as suggested.

thanks