I noticed that the files under _series folder kept growing even I have a retention policy of 1 hour. Though series cardinality remains at 1mil~2mil level. Is it expected? Or what is the proper way to clean up those files under _series folder.
also seeing this issue while running influx v2.7 running in EKS on a statefulset.
Had to run influxd inspect build-tsi and restart pod but that is rather irritating and not exactly an easy thing to automate in our environment.
Would appreciate hearing if anyone else has any suggestions to resolve
1 Like
Are you seeing Series partition compaction in your logs? Do you see the error series partition compaction failed with a message explaining something went wrong?
Hi David.
Only just spotted this here.
I do not see anything that looks incorrect unfortunately.
I have just had to deleted data manually to clear room. This isn’t a huge drama for us as we only need 1 hour data retention.
Since deleting, I starting catching logs (we run this in k8s):
cat out | grep "Series partition compaction"
ts=2025-08-26T11:06:16.331196Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=2 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/02 op_event=start
ts=2025-08-26T11:06:16.527594Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=2 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/02 op_event=end op_elapsed=196.406ms
ts=2025-08-26T11:06:17.374505Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=6 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/06 op_event=start
ts=2025-08-26T11:06:17.529996Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=6 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/06 op_event=end op_elapsed=155.496ms
ts=2025-08-26T11:07:01.382942Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=1 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/01 op_event=start
ts=2025-08-26T11:07:01.397757Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=4 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/04 op_event=start
ts=2025-08-26T11:07:01.746330Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=4 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/04 op_event=end op_elapsed=348.579ms
ts=2025-08-26T11:07:01.776145Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=1 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/01 op_event=end op_elapsed=393.215ms
ts=2025-08-26T11:07:04.363792Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=5 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/05 op_event=start
ts=2025-08-26T11:07:04.555564Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=5 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/05 op_event=end op_elapsed=191.773ms
ts=2025-08-26T11:07:05.395385Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=7 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/07 op_event=start
ts=2025-08-26T11:07:05.585556Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=7 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/07 op_event=end op_elapsed=190.177ms
ts=2025-08-26T11:07:06.382622Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=0 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/00 op_event=start
ts=2025-08-26T11:07:06.614765Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=0 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/00 op_event=end op_elapsed=232.150ms
ts=2025-08-26T11:07:07.363173Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=3 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/03 op_event=start
ts=2025-08-26T11:07:07.562647Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=3 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/03 op_event=end op_elapsed=199.481ms
ts=2025-08-26T12:47:10.016307Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=2 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/02 op_event=start
ts=2025-08-26T12:47:10.439245Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=2 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/02 op_event=end op_elapsed=422.947ms
ts=2025-08-26T12:47:12.711515Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=7 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/07 op_event=start
ts=2025-08-26T12:47:13.284633Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=7 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/07 op_event=end op_elapsed=573.125ms
ts=2025-08-26T12:47:16.197626Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=4 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/04 op_event=start
ts=2025-08-26T12:47:16.950771Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=4 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/04 op_event=end op_elapsed=753.154ms
ts=2025-08-26T12:47:17.747280Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=6 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/06 op_event=start
ts=2025-08-26T12:47:17.991393Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=6 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/06 op_event=end op_elapsed=244.117ms
ts=2025-08-26T12:48:01.771091Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=5 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/05 op_event=start
ts=2025-08-26T12:48:01.991045Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=5 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/05 op_event=end op_elapsed=219.960ms
ts=2025-08-26T12:48:05.745884Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=1 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/01 op_event=start
ts=2025-08-26T12:48:05.766263Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=0 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/00 op_event=start
ts=2025-08-26T12:48:06.272497Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=1 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/01 op_event=end op_elapsed=526.621ms
ts=2025-08-26T12:48:06.277734Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=0 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/00 op_event=end op_elapsed=511.478ms
ts=2025-08-26T12:48:06.741200Z lvl=info msg="Series partition compaction (start)" log_id=0yaR_DH0000 service=storage-engine partition=3 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/03 op_event=start
ts=2025-08-26T12:48:06.985638Z lvl=info msg="Series partition compaction (end)" log_id=0yaR_DH0000 service=storage-engine partition=3 op_name=series_partition_compaction path=/var/lib/influxdb2/engine/data/081be4748dfc8153/_series/03 op_event=end op_elapsed=244.438ms
