The shard group for a specific bucket is 24 hours, and no retention policy is used. When deleting data from 10 days ago, deleting less than 100MB takes more than 10 minutes.
The reason we cannot use automatic deletion is that we log deletion events to syslog. Is there a way to log syslog messages when using automatic deletion? I just can’t understand it—sometimes deleting less than 100MB of data takes more than 4 hours. There are no other conditions for deletion, only the time range.
The slow deletion you’re experiencing is not typical behavior, but there have been reports of deletion issues with InfluxDB 2.x. There was a bug that could cause the retention service to crash, which was fixed in InfluxDB 2.7.7. If you’re using an earlier version, upgrading might help:
Retention policy not working at all
InfluxDB deletes data at the shard group level, not individual points. The retention enforcement service only deletes a shard group when the entire time range covered by the shard group is beyond the bucket retention period. In your case, with a 24-hour shard group duration, this means that data might persist longer than expected.
When does data actually get deleted?
InfluxDB does have a retention enforcement service that runs periodically, but I know if it can be configured to log to syslog…if it doesn’t meet your logging requirements, maybe consider using the InfluxDB API to perform deletions programmatically? This way, you could implement your own logging when deletions occur. Here’s information on using the delete API:
Delete data | InfluxDB OSS v2 Documentation
Given the unusual deletion times you’re experiencing, it might be worth investigating other factors such as system resources, concurrent operations, or potential bugs in your specific InfluxDB version. Or submit an issue here.
Thanks!
