Relationship between Retention Policy and Shard Group Duration

InfluxDB 1
,
1

I am now a beginner with InfluxDB.
What i am confused is Relationship between Retention Policy and Shard Group Duration after reading this article (Simplifying InfluxDB: Shards and Retention Policies | InfluxData)

What i know is that default retention policy(autogen) infinitely stores data and never remove it.
But in the article above, it says time-series data is stored on shard group and If nothing is specified, the shard group duration defaults to 7 days. And InfluxDB drops an entire shard group if duration expired
For me, this description conflicts with default retention policy which store data infinitely.

What I’m curious about is what happens if retention policy and shard group duration is default. Is data permanently stored? or is shard group dropped? What happens?

2

Hello @Mangkyu,
Welcome!
May I ask why are you using InfluxDB 1.x?
I would recommend using 2.x so you don’t have to worry about creating retention policies ( you only have to set a retention period). Shard group just defines when data is moved from memory to disk. Shard group duration is different from the retention policy. The shard and the data in it is moved to disk after 7 days.

3

Hello @Anaisdg
I’m currently using InfluxDB 1.8, so maybe I should consider using 2.x.

Thank you for your kind explanation so I could understand what it means
Thank you very much!

1 Like
4

Hello ,

Can you anwer me a question about the deletion of data and the bucket’s retention period?

How the shard duration effects the deletion of data and shards?

I read this Influxdata documentation:

  1. A shard contains encoded and compressed time series data for a given time range defined by the shard group duration. All points in a series within the specified shard group duration are stored in the same shard.

  2. In InfluxDB OSS, a shard group typically contains only a single shard.

  3. Shard deletion:
    The InfluxDB retention enforcement service routinely checks for shard groups older than their bucket’s retention period. Once the start time of a shard group is beyond the bucket’s retention period, InfluxDB deletes the shard group and associated shards and TSM files.

Example:

All data is stored in a bucket:

I use InfluxDB OSS Version 2.1.

retention period=1d

shard duration=1h

Data is stored every 10s.

My understanding:

I have stored data over 1 day in this bucket. So i have 24 shards and every shard is stored in a shard group with the duration of 1h.

After the first day + 10 second the retention enforcement service realize that the first stored shard group have at least one “old” metric (recorded 00:00:10) but it deletes the whole shard group which contains metrics recorded from 00:00:10 to 01:00:10 .

After this I do not have the same amount of data (24h) available, because I only have 23 shard group existing that is 23h.

Is this understanding correct or i am wrong?

Thanks for help!

Greetings from Germany,

Dave

5

@dave
What a beautiful understanding. That’s perfect explanation.