Pi zero out of memory problem

Hi.

I am using influxdb 1.8 on a pi zero w - it had been working well but then started crashing at 4AM each day. This coincided with me creating a second database.

Syslog showed errors at this time related to compaction.

I have reduced the shard size and retention policy on the default policy, and it now stays up but I still get errors between 4AM and 7AM and nothing is written to the databases during this time.

The error I get just before things resume is
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=influxd,pid=456,uid=999

From reading on here my guess is that I need to convert from tsm to tsi and maybe increase my swapfile size.

I am not sure if compaction has been done, or if it will just keep retrying daily and fail.
I would be grateful to know if I am heading in the right direction or if there are any other measures I can take.

Thanks

Are you using the TSI index?

I believe I am using TSM as I get this in the syslog ---- lvl=info msg="TSM compaction (start)

@grumpazoid -

To serve queries, InfluxDB maintains a series index - this can either be “in-memory” or file based via “TSI” files, which are then memory mapped as needed to access different parts of the index. The in-memory option generally uses more memory than the file based version. You can configure this via a config setting: https://docs.influxdata.com/influxdb/v1.8/administration/config/#index-version-inmem

I recommend using TSI for RPi. Increasing swap limit can help too if you have the disk available. Pis can be challenging (but also fun!) as they are resource constrained.

Let us know how you solved it!

1 Like

@philjb

Thanks for the advice. Last nights “outage” at 4AM was only for 25 minutes but I have followed the instructions and changed to TSI this morning. I deleted the .idx files as per instructions.
I can still see my data in grafana so will see how it behaves tonight and report back.

1 Like

All was working OK after the conversion to TSI but at the moment the pi is not serving web pages.
First thing this morning it was taking a long time to serve up Grafana and I could see on the CPU graph there had been a few “outages”.

At the moment it is not responding. Getting a response from a ping mostly fails.

Ok the answer for me to was to take the excellent advice from Andreas Spiess

I restored a backup image to the pi that I had taken before I initiated tsi - so now back to using default tsm.

Setting store-enabled false was all that was required. This has reduced cpu and memory use massively and everything is working well again.

Good solution! Thanks for sharing it. This is new setting to me: https://docs.influxdata.com/influxdb/v1.8/administration/config/#store-enabled-true