What is the retention policy and how exactly it work?

1

what is the retention policy and how exactly it work?

as you know we can set retention policy “check-interval” on config file of influx.
what happen if i set this parameter and after that meet interval, which part of database remove?
what happen to the old data? i mean if i have database of 1 year ago retention will remove it ?

Thanks,

2

@Indeed_1 The database will only drop whole shards at a time. If you use the influx cli to run SHOW SHARDS you should see a listing of your database with their respective shards. Each of those shards has an expiry_time. Each check-interval the database will drop any of the shards who’s expiry times are older than time.Now().

Does that help?

1 Like
3

@jackzampolin thank you Jack,
i check my config file , in shard section write this

[shard-precreation]
enabled = true
check-interval = "10m"
advance-period = “30m”

what is mean? each 10 minutes my shard will be remove? but when i enter “SHOW SHARDS” seems all shards remain! would you please clarify this happen?

Thanks,

4

@Indeed_1 That section is for shard-precreation. It will create empty shard files ahead of time. The section that deals with retentions (and the defaults) looks as follows:

[retention]
  enabled = true
  check-interval = "30m0s"

This means the database will check every 30 minutes to see if the shard.expiry_time < now(). Also it will only drop shards if there is a duration > 0d. By the default retention policy that gets created with each database has infinite retention. You can check this with SHOW RETENTION POLICIES ON "mydb".

1 Like
5

@jackzampolin Thank you dear Jack,

I’ve create new RP but now I have two RP :

name duration shardGroupDuration replicaN default

autogen 0s 168h0m0s 1 true
myRP 1h0m0s 1h0m0s 1 false

which will be run? any priority ?
should i remove autogen till myRP work correctly?

Thanks,

6

another issue after RP run , when i “select count” of tables seems old data still exist ! is that index or something like that?

7

@Indeed_1 I suggest you check out this documentation on retention policies. That should help clear up some of your questions.

8

When I apply a new default retention policy to an existing DB I seem to lose access to any data already collected. SHOW SHARDS shows that a new shard with the new retention policy was created, but the old shard still exists. However, queries seem to be answered from the new shard which is empty. Is this the expected behaviour?

Thanks.

9

@Bill_Gunter You can query the old shard you just need to specify the retention policy. When querying use the fully qualified measurement name: "database"."retentionPolicy"."measurement". When querying without the fully qualified measurement name the db will return data from the DEFAULT retention policy, your empty new one. This is intended behavior.

10

Wow, that makes complete sense. I didn’t make the connection between the retentionpolicy in a query and adding a new retentionpolicy. Thanks!

1 Like
11

So how does retention policies map to measurements? If I have 5 RPs and 5 measurements which is applied to which?

12

I hope the answer over here helps.

13

hi. is there a way to check when the next retention policy check will run? My config says 30m, but is there a command to check ? Thanks

14

The closest thing you can do is run SHOW SHARDS and then check the expiry_time for them