Influxdb 2 container problems

InfluxDB 2
, , ,
1

Hello
I have been having issues getting my Influxdb container to start. I had issues getting it to start without crashing. In the end I decided to start a fresh install using the old token and restoring my buckets from backup. I managed to only restore 4 out of 13 buckets and it seemed to hang while restoring the 5th bucket so I cancelled it. It never seems to finish starting up listing the following in the logs:

=start
ts=2023-11-01T08:40:41.068278Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=2 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=191.238ms
ts=2023-11-01T08:40:40.985374Z lvl=info msg="TSI log compaction (start)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=start
ts=2023-11-01T08:40:40.987601Z lvl=info msg="index opened with 8 partitions" log_id=0lEWVmQG000 service=storage-engine index=tsi
ts=2023-11-01T08:40:41.068355Z lvl=info msg="TSI log compaction (start)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=6 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=start
ts=2023-11-01T08:40:41.068365Z lvl=info msg="TSI log compaction (start)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=6 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=start
ts=2023-11-01T08:40:41.071632Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=7 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=96ms bytes=610 kb_per_sec=6
ts=2023-11-01T08:40:41.073052Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=1 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=98ms bytes=713 kb_per_sec=7
ts=2023-11-01T08:40:41.073299Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=1 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=98.439ms
ts=2023-11-01T08:40:41.073373Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=204.565ms
ts=2023-11-01T08:40:41.072911Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=3 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=97ms bytes=712 kb_per_sec=7
ts=2023-11-01T08:40:41.073625Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=7 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=98.127ms
ts=2023-11-01T08:40:41.073473Z lvl=info msg="loading changes (start)" log_id=0lEWVmQG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2023-11-01T08:40:41.074447Z lvl=info msg="loading changes (end)" log_id=0lEWVmQG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.979ms
ts=2023-11-01T08:40:41.077114Z lvl=info msg="TSI log compaction (start)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=start
ts=2023-11-01T08:40:41.077821Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=1 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=195ms bytes=713 kb_per_sec=3
ts=2023-11-01T08:40:41.079182Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=3 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=210ms bytes=712 kb_per_sec=3
ts=2023-11-01T08:40:41.079599Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=103ms bytes=615 kb_per_sec=5
ts=2023-11-01T08:40:41.079844Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=3 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=104.535ms
ts=2023-11-01T08:40:41.080392Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=1 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=197.646ms
ts=2023-11-01T08:40:41.082211Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=105.836ms
ts=2023-11-01T08:40:41.082537Z lvl=info msg="TSI log compaction (start)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=3 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=start
ts=2023-11-01T08:40:41.084222Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=3 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=215.587ms
ts=2023-11-01T08:40:41.084663Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=7 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=106ms bytes=610 kb_per_sec=5
ts=2023-11-01T08:40:41.085065Z lvl=info msg="TSI log compaction (end)" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=7 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=106.987ms
ts=2023-11-01T08:40:41.083798Z lvl=info msg="Log file compacted" log_id=0lEWVmQG000 service=storage-engine index=tsi tsi1_partition=2 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=105ms bytes=709 kb_per_sec=6

It never ends. It just keeps generating more and more of the above logs. in the past it used to fly through these during startup and would get to the part where it’s listening on port 8086 within 10-15 seconds.

I don’t know if this is a bug in version 2.7+ and I’m not sure what to do anymore.

Any ideas? what do the logs mean and why is the container taking so long to start? Even stopping the container takes more than 10mins. I tried removing the container and recreating it to no avail.

2

Did you recently upgrade or were you already on version 2.7? Did you make any configuration changes? Are you using the same image?
If none of that has changed, I’m guessing the only difference between when it could start fine and now is some added data. Trying to start on some backed up historical data, from before this started happening, would be a good experiment if you happen to keep backups.

3

Hi all,
Sorry, I have not been able to come here to post an update until now.

I could not get it to work no matter what I did including trying on an x86 powerful laptop so I resorted to restoring from backup. Unfortunately my backups continued to run for the dodgy data so I had to restore from one that was a month old. I lost a month’s worth but everything is back now.