sloach
November 11, 2018, 1:01am
1
Hi all. I upgraded from 1.6.4 to 1.7 two days ago, and I’m having major issues with memory usage and I/O load with the new release.
I noticed in the release notes that the on-disk format of TSI indexes changed. Does that mean it’s not possible to go back to 1.6.4?
I will enter issues but right now I have to figure out how to get things working…
gunnar
November 11, 2018, 1:38am
2
@sloach Can you please open an issue on the InfluxDB issue describing the high CPU usage? Including a CPU profile in the issue description will help us narrow down the problem faster.
Generally, InfluxDB can be rolled back to a previous release unless there is a breaking change noted in the changelog .
I also want to add, there was briefly a bug in the nightly releases of InfluxDB between the 1.6.4 and 1.7.0 that would cause corrupt TSM files to be generated in some scenarios. If you ran a nightly release, please mention that in the issue.
sloach
November 11, 2018, 2:01am
3
Thanks, I opened an issue.
opened 02:00AM - 11 Nov 18 UTC
closed 10:28AM - 15 Nov 18 UTC
area/performance
kind/bug
area/memory
__System info:__
InfluxDB 1.7, upgraded from 1.6.4. Running on the standard Do… cker image.
__Steps to reproduce:__
I upgraded a large InfluxDB server to InfluxDB 1.7. Nothing else changed. We are running two InfluxDB servers of a similar size, the other one was left at 1.6.4.
This ran fine for about a day, then it started running into our memory limit and continually OOMing.
We upped the memory and restarted. It ran fine for about 4 hours then started using very high disk i/o (read) which caused our stats writing processes to back off.
Please see the timeline on the heap metrics below:
- you can see relatively stable heap usage before we upgraded at 6am on Nov 9
- at around 4pm there is a step-up in heap usage
- around 11:30pm there is another step-up and it starts hitting the limit (OOM, causes a restart)
- at 12:45pm on the 10th we restart with more RAM
- around 4 hours later it starts using high i/o and you can see spikes in heap usage
![image](https://user-images.githubusercontent.com/11753520/48308130-10064180-e52b-11e8-9772-c911b07856cf.png)
1.7.1 was just published. We believe that should address this issue.
Please give that a shot.
In terms of downgrade, you can…the only thing that you will need to do is to rebuild the TSI index if you are using that.
I was facing the same challenge when upgraded influxdb to 1.8.9 on raspbian buster OS so I have to downgrade back to 1.8.5
please try the instructions I mentioned