I have an InfluxDB bucket for storing some of my system log data.
Initially, the retention period was set to 28 days, but later, due to some business requirements, I changed it to 48 hours.
However, I’ve noticed that, after making this change, the previous data has not been cleared, and I am still able to query data from 7 days ago .
Could you please explain why this is happening?
@lzl82891314 There’s a lot of variables that play into when data actually gets deleted and it gets a little more complicated when you change the retention period of a bucket. Essentially it boils down to InfluxDB v2 expiring shard groups and not specific points. A shard group will only get expired when all the points in the group have expired. I’m guessing the old data points are stored in a shard group that was created before you changed the retention period and data in the shard group hasn’t fully expired.
For the nitty-gritty details, here’s some docs:
Thank you for your reply.
I also want to ask, if a new retention period is configured for a bucket, will the old shards change accordingly?
For instance, if the original expiration was 28 days, and old shards might have had an expiration of 1 day, when I change the period to 48 hours, the new shards should by default expire in 1 hour.
So, will the expiration time of the old shards that still contain data also change to 1 hour?
@lzl82891314 I’m waiting to hear back from InfluxDB engineers, but I think that’s what should happen, but I don’t if that’s what actually does happen.
@lzl82891314 Just heard back. The shard expiration is set when the shard is created based on the current retention period of the bucket. If you change the retention period of a bucket, it does not change the expiration date on shards that have already been created. So the reason your data isn’t being deleted is because the shards the data is being stored in was created using the old retention period. So that data will stick around until the the 28 day retention period the shard was created with ends.
With InfluxDB OSS v2, the only way to really get around this and force the shorter retention period is to query all the data out of the the current bucket and write it to a new bucket with the desired retention period. This will create new shards for that bucket using the new retention period. Once the data is moved over, you can delete the old bucket.
Other than that, you just have to wait for the shard to expire.
EDIT: You could also manually delete the data base on a time range using the delete API–Delete data | InfluxDB OSS v2 Documentation.
I got it, thank you once again for your reply.
