# Retention policy deletion check let the memory explode

**URL:** <https://community.influxdata.com/t/retention-policy-deletion-check-let-the-memory-explode/6558>\
**Category:** Systems\
**Tags:** influxdb\
**Created:** [September 12, 2018, 8:52am UTC](https://community.influxdata.com/t/retention-policy-deletion-check-let-the-memory-explode/6558 "2018-09-12T08:52:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![zilongzhang](https://avatars.discourse-cdn.com/v4/letter/z/cc9497/32.png) [@zilongzhang](https://community.influxdata.com/u/zilongzhang)\
**Post date:** [September 12, 2018, 8:52am UTC](https://community.influxdata.com/t/retention-policy-deletion-check-let-the-memory-explode/6558/1 "2018-09-12T08:52:24Z")

</div>

I just start using influxdb(1.7) to collect cgroup measurement,  
and i find retention policy deletion check has a very expensive cost on memory and cpu,  
the influx log like below:  
ts=2018-09-12T00:08:01.600466Z lvl=info msg=“Retention policy deletion check (start)” log\_id=0AU6aqCG000 service=retention trace\_id=0AUmYLG0000 op\_name=retention\_delete\_check op\_event=start  
…  
s=2018-09-12T00:32:43.064925Z lvl=info msg=“Retention policy deletion check (end)” log\_id=0AU6aqCG000 service=retention trace\_id=0AUmYLG0000 op\_name=retention\_delete\_check op\_event=end op\_elapsed=1481464.468ms  
…  
ts=2018-09-12T01:08:01.600467Z lvl=info msg=“Retention policy deletion check (start)” log\_id=0AU6aqCG000 service=retention trace\_id=0AUpz3l0000 op\_name=retention\_delete\_check op\_event=start  
…  
ts=2018-09-12T01:50:50.442853Z lvl=info msg=“Retention policy deletion check (end)” log\_id=0AU6aqCG000 service=retention trace\_id=0AUpz3l0000 op\_name=retention\_delete\_check op\_event=end op\_elapsed=2568842.397ms  
ts=2018-09-12T01:50:50.442905Z lvl=info msg=“Retention policy deletion check (start)” log\_id=0AU6aqCG000 service=retention trace\_id=0AUsQrIW000 op\_name=retention\_delete\_check op\_event=start  
ts=2018-09-12T01:50:50.443269Z lvl=info msg=“Retention policy deletion check (end)” log\_id=0AU6aqCG000 service=retention trace\_id=0AUsQrIW000 op\_name=retention\_delete\_check op\_event=end op\_elapsed=0.370ms

when i watch the heap size on the chrongraf,it looks like this:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/influxdata/original/2X/1/134bb3ef711a5824eca708603d48b4b5c05c96fb.png)  
cpu also has the same performance:

here is some more informations:  
series has reached 4 million ,

shard groups for telegraf:

205 telegraf one\_hour 2018-09-11T20:00:00Z 2018-09-11T21:00:00Z 2018-09-12T03:00:00Z  
207 telegraf one\_hour 2018-09-11T21:00:00Z 2018-09-11T22:00:00Z 2018-09-12T04:00:00Z  
209 telegraf one\_hour 2018-09-11T22:00:00Z 2018-09-11T23:00:00Z 2018-09-12T05:00:00Z  
211 telegraf one\_hour 2018-09-11T23:00:00Z 2018-09-12T00:00:00Z 2018-09-12T06:00:00Z  
213 telegraf one\_hour 2018-09-12T00:00:00Z 2018-09-12T01:00:00Z 2018-09-12T07:00:00Z  
215 telegraf one\_hour 2018-09-12T01:00:00Z 2018-09-12T02:00:00Z 2018-09-12T08:00:00Z  
217 telegraf one\_hour 2018-09-12T02:00:00Z 2018-09-12T03:00:00Z 2018-09-12T09:00:00Z  
219 telegraf one\_hour 2018-09-12T03:00:00Z 2018-09-12T04:00:00Z 2018-09-12T10:00:00Z

the retention policies:  
one\_hour 6h0m0s 1h0m0s 1 true

tsm file:  
-rw-r–r-- 1 root root 390814115 9月 12 10:36 000000008-000000002.tsm  
-rw-r–r-- 1 root root 132929882 9月 12 10:41 000000009-000000001.tsm  
-rw-r–r-- 1 root root 132502656 9月 12 10:46 000000010-000000001.tsm  
-rw-r–r-- 1 root root 127318427 9月 12 10:52 000000011-000000001.tsm  
-rw-r–r-- 1 root root 391 9月 12 10:00 fields.idx

influxdb config:  
[meta]  
dir = “/data/tdwadmin/tdwenv/influxdb/influxdb/var/lib/influxdb/meta”  
retention-autocreate = true  
[data]  
dir = “/data/tdwadmin/tdwenv/influxdb/influxdb/var/lib/influxdb/data”  
wal-dir = “/data/tdwadmin/tdwenv/influxdb/influxdb/var/lib/influxdb/wal”  
wal-fsync-delay = “0”  
index-version = “inmem”  
trace-logging-enabled = true  
query-log-enabled = false  
cache-max-memory-size = “20g”  
cache-snapshot-memory-size = “128m”  
max-index-log-file-size = “16m”  
max-series-per-database = 0  
max-values-per-tag = 0  
compact-full-write-cold-duration = “1h”  
cache-snapshot-write-cold-duration = “10m”  
max-concurrent-compactions = 4

[coordinator]  
write-timeout = “10s”

[retention]  
enable = true  
check-intrerval = 30

[http]  
bind-address = “:8086”  
realm = “InfluxDB”  
log-enabled = false  
access-log-path = “influx.http.log”  
max-body-size = 0

and, yeah, the server has 128G RAM,48 Cores with a SSD of 256G.

details showing at github issue: [Retention policy deletion check let the memory explode · Issue #10277 · influxdata/influxdb · GitHub](https://github.com/influxdata/influxdb/issues/10277)
