I’m currently in the middle of doing some testing of some other databases (we’ve been using Cassandra for time series storage for 3 years) and I’m trying to load up at least a subset of the ~15 billion points we have in order to do some testing of InfluxDB. I realize it should take a bit to load that much data, but at the rate I’m going it’s going to take several weeks. I’ve got a Python script running on an EC2 machine in the same region as my InfluxCloud-based database which is just pulling the data from CSV files and then using the influx python client library to write those values to Influx. I’ve got it running in batches of 8000 points and each write to influx is taking about 4-8 seconds. I’ve read that performance is best in batches of 5000-8000 points and each batch of writes (around 750,000 points) I’m pulling from a CSV file are for the same measurement + tag, in chronological order. From what I’ve read, that should give me really good performance. Any other tips or am I doing something wrong?
Are you writing to a single node (the OSS version of InfluxDB) or do you have a cluster (InfluxEnterprise)? Generally, in the region of 750,000 writes to a single node is going to be pretty taxing, according to the documentation and hardware sizing guidelines. Also, do you have TSI enabled? That can help with series cardinality performance. Here’s the link for the guidelines on how to enable TSI. Keep us posted with any further questions and good luck!
I’ve got experience with loading about 1 billion points. Each point had varying amount of fields (somewhere between 1 and 100 fields) and one tag. I tweaked my batch size so that one batch takes about one second to import. I don’t remember the exact batch size but it was somewhere between 100,000 and 300,000. In my case, I wrote my import code in Go, I imported from csv and I used line protocol. I did not time how long it took but I’d say somewhere between a few hours and 1-2 days.
Make sure you don’t do any read while you are writing. It slows down the writes.
At first, I started by running the import script from my development pc, reading the csv file from a network drive and sending it to a remote InfluxDB server. I was on a 10 Mbit connection shared between reading the csv and sending to InfluxDB and it was very slow. I could log on the InfluxDB server so I moved the script there and executed it from there. I could then read the csv through a 1 Gbit connection and the script executed on the InfluxDB server so there was no network latency to slow things down.
Thanks everyone for the responses. Turns out it was just the fact that I was using Python and it was having a hard time reading my CSV files, transforming some timestamps, and sending it off to Influx. We switched to using Go to do the heavy lifting and we’ve seen a significant speedup in performance. However, now we’re starting to see 503’s after a period of time sending data to Influx, which is now blocking getting the rest of the data in there…we just get random responses like the one below. To provide some clarity, we are using InfluxCloud using the trial (Standard 1 I think is what the tier is called).
ERROR: <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>
