I have installed influx 2.4 on a red hat 8.5 last May 2023. Everythin was running fine until last night, where influxd got killed by oom. Indeed the memory consumption of influx increases from 2GB (last night at 00:00 UTC time to 16GB at 3:45 UTC time). I didn’t see anything suspicious in the log file.
The only thing i spot is our influx was logging duration acitivity every hour but last night during this memory increase nothing.
if someone has any idea of what could have happened that would be appreciated.
thanks
Sep 5 23:35:03 io-ls-influxdb influxd-systemd-start.sh[365610]: ts=2023-09-05T23:35:03.644859Z lvl=info msg=“Snapshot for path written” log_id=0dBY4iLl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1076 duration=170.464ms
Sep 5 23:45:59 io-ls-influxdb influxd-systemd-start.sh[365610]: ts=2023-09-05T23:45:59.659165Z lvl=info msg=“Snapshot for path written” log_id=0dBY4iLl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1076 duration=184.763ms
Sep 5 23:56:50 io-ls-influxdb influxd-systemd-start.sh[365610]: ts=2023-09-05T23:56:50.624808Z lvl=info msg=“Snapshot for path written” log_id=0dBY4iLl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1076 duration=149.904ms
Sep 6 00:09:59 io-ls-influxdb influxd-systemd-start.sh[365610]: ts=2023-09-06T00:09:59.594748Z lvl=info msg=“Snapshot for path written” log_id=0dBY4iLl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1076 duration=120.439ms
Sep 6 00:10:51 io-ls-influxdb influxd-systemd-start.sh[365610]: ts=2023-09-06T00:10:51.754617Z lvl=info msg=“Snapshot for path written” log_id=0dBY4iLl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1078 duration=70.193ms
Sep 6 03:10:56 io-ls-influxdb influxd-systemd-start.sh[365610]: ts=2023-09-06T03:10:56.740051Z lvl=info msg=“Snapshot for path written” log_id=0dBY4iLl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1078 duration=20.110ms
Sep 6 03:57:49 io-ls-influxdb influxd-systemd-start.sh[365610]: ts=2023-09-06T03:57:49.754213Z lvl=info msg=“Snapshot for path written” log_id=0dBY4iLl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1078 duration=34.245ms
Sep 6 04:08:39 io-ls-influxdb influxd-systemd-start.sh[846816]: ts=2023-09-06T04:08:39.915982Z lvl=info msg=“Opened file” log_id=0k6AgiNG000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1010/000000117-000000002.tsm id=0 duration=66.316ms
Sep 6 04:08:39 io-ls-influxdb influxd-systemd-start.sh[846816]: ts=2023-09-06T04:08:39.916094Z lvl=info msg=“Opened shard” log_id=0k6AgiNG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1010 duration=175.186ms
Sep 6 04:08:39 io-ls-influxdb influxd-systemd-start.sh[846816]: ts=2023-09-06T04:08:39.966493Z lvl=info msg=“Opened file” log_id=0k6AgiNG000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/998/000000094-000000002.tsm id=0 duration=75.124ms
Sep 6 04:08:39 io-ls-influxdb influxd-systemd-start.sh[846816]: ts=2023-09-06T04:08:39.966596Z lvl=info msg=“Opened shard” log_id=0k6AgiNG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/998 duration=206.758ms
Sep 6 04:08:40 io-ls-influxdb influxd-systemd-start.sh[846816]: ts=2023-09-06T04:08:40.053198Z lvl=info msg=“Opened file” log_id=0k6AgiNG000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1000/000000100-000000002.tsm id=0 duration=50.372ms
Sep 6 04:08:40 io-ls-influxdb influxd-systemd-start.sh[846816]: ts=2023-09-06T04:08:40.053298Z lvl=info msg=“Opened shard” log_id=0k6AgiNG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/6b4ae9f01b33c180/autogen/1000 duration=137.191ms
1 Like
