Question about retentions

Hi all

I’ve got an old database with a lot of data. She used the autogen with infinite retention.

I’ve created a new retention on it, with 6 months retention, used by default.
Now, I saw all my old data are not visible. According the documentation, it seems normal and I can get my old data modifying my requests.

I can forget these old datas, so OK, not a problem.

But, how can I be sure my old data from this previous retention autogen is removed?
On the influxdb.conf, the retention is enabled with interval 30m, but after one days no space freed on my server.

Does someone got informations?

Thanks

Sometimes this happens when the shard group duration is set differently than the retention policy.

What’s your current shard group duration?

@katy On a related note, if I set my retention policy as inf with default shard group duration as 7days, I see that there is an expiry time for every shard created. Does this mean my data will get deleted if the shard expires? If so, how can I avoid this as I’d like to store data indefinitely
I’m new to influx and any help would be appreciated
Thank you

The 7-day default splits the retention policy into 7-day windows. The expiry time is the time when the data is actually deleted.

The database will only drop whole shards at a time. If you use the influx cli to run SHOW SHARDS you should see a listing of your database with their respective shards. Each of those shards has an expiry_time . Each check-interval the database will drop any of the shards who’s expiry times are older than time.Now() .

@katy I understand that. But what if I don’t want my shards to be deleted. I want to keep data infinitely. I’ve set retention policy as inf yet my shards have an expiry after 7 days. Will the shard get deleted at the end of the week? If so, how can I avoid that?
Thank you

No, the default retention policy will keep your data indefinitely.

1 Like

@katy, in case of the default retention policy “autogen”, on running SHOW SHARDS, I see that this,
start_time |end_time |expiry_time
2019-02-11T00:00:00Z |2019-02-18T00:00:00Z |2019-02-18T00:00:00Z

The shard is still there even though the expiry time has passed.
Also, if the policy is autogen, what is the significance of the same end_time and expiry_time??

Hi Katy sorry to disturb :slight_smile: ,

I am confused about the INF in the default retention policy , has the behaviour changed
between releases ?

From your answer : “No, the default retention policy will keep your data indefinitely.”

From this post : https://www.influxdata.com/blog/influxdb-shards-retention-policies/

That post says :
“If your retention policy is on autogen (or infinite), the data is not actually stored infinitely—this just means the retention policy matches the shard group duration, so the retention policy is effectively disabled.”

From the answer of sm97 : “The shard is still there even though the expiry time has passed.”
I would conclude that you are right and that the post is wrong …

best regards ,

@MarcV The vocabulary around retention policies and shards is confusing. I think what the blog post is saying is that although it doesn’t explicitly say infinite, the effect is the same. So the default retention policy will keep your data indefinitely, but if you change the shard duration or RP duration, it can cause some different behavior.