Should I divide my data into several measurements?

Store
1

Hi,

I just inserted one day of data into InfluxDB to test if it worked, and it doesn’t.
This one day file contains 1.678.586 data points that I stored in one measurement. The thing is that in the end I plan to store years of data in this database, and I don’t want to delete anything with retention policies (I don’t want to lose any precision, even for old data).
Is a good practice to divide my data into several measurements (like one per day fer exemple) ?

2

Can you provide an example / snapshot of your data, please?

3

Sorry but no, it’s confidential. I can’t even tell you where it comes from. All I can say is that it comes from an earth observation satellite. The original CSV file weights 207mB.
The CSV file has 17 columns. One of them is the timestamp, 5 of them are numeric values that I stored as fields, the rest is stored as tags. My storage dropped by 544 mB after the data insertion.

4

No problem.

Can you confirm which version of InfluxDB you are using, and if you’re using TSM or TSI?

Can you also clarify what you mean by “it doesn’t” with regards to it “working”? I’d like to understand what you expect and how it fails.

If you can also provide the output of show series cardinality, that would help too.

Thanks :+1:

5

I am using InfluxDB 1.7.4. I have absolutely no idea if I am using TSI or TSM. How can I check that ?

By “it doesn’t work” I mean that it either takes several minutes to perform a query from the influx CLI or crashes when I try to plot the data with grafana.

The result of show series cardinality is 403253

6

Do you have index-version set to tsi1 in influxdb.conf, or the environment variable INFLUXDB_DATA_INDEX_VERSION set to tsi1?

7

Ok found it, the conf file says engine = "tsm1"

8

You should try to run the same dataset with tsi1 and you will hopefully see better performance. Your cardinality wasn’t particularly high, so tsm1 should actually be fine (until millions of series).

I’ll have someone else take a look at this thread, but it’ll be during PST office hours; and hopefully they can provide some further insight.

9

Ok thank you.
Do I have to create a new database or is changing the conf file sufficicent ?

10

You’d need a new database after the config changes, or you’d need to migrate your existing data:

You can build tsi indices for existing databases: