Influx 1.7 compact configuration

Hi,

We have an Influx Setup with a Retention Policy of 40 days and a Shard Duration of 1 day. The database, contains about
7813 different measurements.

This are the diagnostics:

> show diagnostics
name: build
Branch Build Time Commit                                   Version
------ ---------- ------                                   -------
1.7.11            f11ad4780c8a61108108a18b141c1d067d920a80 1.7.11

name: config
bind-address   reporting-disabled
------------   ------------------
127.0.0.1:8088 false

name: config-coordinator
log-queries-after max-concurrent-queries max-select-buckets max-select-point max-select-series query-timeout write-timeout
----------------- ---------------------- ------------------ ---------------- ----------------- ------------- -------------
0s                0                      0                  0                0                 0s            10s

name: config-cqs
enabled query-stats-enabled run-interval
------- ------------------- ------------
true    false               1s

name: config-data
cache-max-memory-size cache-snapshot-memory-size cache-snapshot-write-cold-duration compact-full-write-cold-duration dir                    max-concurrent-compactions max-index-log-file-size max-series-per-database max-values-per-tag series-id-set-cache-size wal-dir               wal-fsync-delay
--------------------- -------------------------- ---------------------------------- -------------------------------- ---                    -------------------------- ----------------------- ----------------------- ------------------ ------------------------ -------               ---------------
1073741824            26214400                   10m0s                              4h0m0s                           /var/lib/influxdb/data 0                          1048576                 1000000                 100000             100                      /var/lib/influxdb/wal 0s

name: config-httpd
access-log-path bind-address enabled https-enabled max-connection-limit max-row-limit
--------------- ------------ ------- ------------- -------------------- -------------
                :8086        true    false         0                    0

name: config-meta
dir
---
/var/lib/influxdb/meta

name: config-monitor
store-database store-enabled store-interval
-------------- ------------- --------------
_internal      true          10s

name: config-precreator
advance-period check-interval enabled
-------------- -------------- -------
30m0s          10m0s          true

name: config-retention
check-interval enabled
-------------- -------
30m0s          true

name: config-subscriber
enabled http-timeout write-buffer-size write-concurrency
------- ------------ ----------------- -----------------
true    30s          1000              40

name: network
hostname
--------
equinox-measure-db-2

name: runtime
GOARCH GOMAXPROCS GOOS  version
------ ---------- ----  -------
amd64  2          linux go1.13.8

name: system
PID currentTime                    started                        uptime
--- -----------                    -------                        ------
1   2022-07-15T08:00:19.743038094Z 2022-07-14T23:36:56.515900801Z 8h23m23.227137293s




The performance is outstanding, but certain times the InfluxDB starts compacting files and demanding an outrageous ammount of memory, that ends up colapsing the pod (we have 3-StatefulSet configuration on Kubernetes)
and thus reseting and reseting the pod.

ts=2022-07-14T23:36:21.539857Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgly0gl000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:21.723065Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000353-000000003.tsm
ts=2022-07-14T23:36:21.723402Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000417-000000003.tsm
ts=2022-07-14T23:36:21.723690Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000433-000000002.tsm
ts=2022-07-14T23:36:21.723993Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000449-000000002.tsm
ts=2022-07-14T23:36:21.724281Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000465-000000002.tsm
ts=2022-07-14T23:36:35.041983Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:35.648758Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000015-000000002.tsm
ts=2022-07-14T23:36:35.648765Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000031-000000002.tsm
ts=2022-07-14T23:36:35.648770Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000033-000000001.tsm
ts=2022-07-14T23:36:35.648775Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000035-000000001.tsm
ts=2022-07-14T23:36:36.279794Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:36.647693Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000015-000000002.tsm
ts=2022-07-14T23:36:36.647699Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000031-000000002.tsm
ts=2022-07-14T23:36:36.647704Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000033-000000001.tsm
ts=2022-07-14T23:36:36.647711Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000035-000000001.tsm
ts=2022-07-14T23:36:37.161966Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:37.648826Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/392/000000063-000000003.tsm
ts=2022-07-14T23:36:37.648848Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/392/000000065-000000001.tsm
ts=2022-07-14T23:36:37.648866Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/392/000000067-000000001.tsm
ts=2022-07-14T23:36:47.859812Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:48.649430Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000063-000000003.tsm
ts=2022-07-14T23:36:48.649492Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000127-000000003.tsm
ts=2022-07-14T23:36:48.649585Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000129-000000001.tsm
ts=2022-07-14T23:36:48.649624Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000131-000000001.tsm
ts=2022-07-14T23:36:48.649691Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000133-000000001.tsm
ts=2022-07-14T23:36:48.649735Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000135-000000001.tsm
ts=2022-07-14T23:36:48.649867Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000137-000000001.tsm
ts=2022-07-14T23:36:48.649908Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000139-000000001.tsm

TSM files are about 50MB size. In this case the very last compaction seems to have failed. When re-entering the POD, for this 390 entry… the TSM file is just 30MB.

Reports say it’s healthy, though.

bash-5.1# influx_inspect verify -dir /var/lib/influxdb/
/var/lib/influxdb/data/_internal/monitor/387/000000029-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/389/000000137-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/391/000000061-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/395/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/398/000000102-000000004.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000001-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000002-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000003-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000004-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000005-000000001.tsm: healthy
/var/lib/influxdb/data/measures/autogen/322/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/324/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/326/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/328/000000014-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/330/000000014-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/332/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/334/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/336/000000016-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/338/000000018-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/340/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/342/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/344/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/346/000000018-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/348/000000659-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/349/000000655-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/351/000043069-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/354/000062571-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/356/000063568-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/358/000063911-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/360/000064809-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/362/000064330-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/364/000063625-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/366/000063229-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/368/000063382-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/370/000044940-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/372/000034298-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/374/000011502-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/376/000000020-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/378/000000627-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/380/000000035-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/382/000000467-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/384/000000035-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/386/000000297-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/388/000000035-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/390/000000141-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/392/000000069-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/393/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/394/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/396/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/397/000000836-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/400/000000008-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/400/000000009-000000001.tsm: healthy
Broken Blocks: 0 / 1170443, in 15.453021259s
bash-5.1# influx_inspect report /var/lib/influxdb/data/
DB        RP      Shard   File                    Series  New (est) Min Time             Max Time             Load Time
measures  autogen 322     000000015-000000002.tsm 13443   13395     2022-06-05T00:00:00Z 2022-06-05T23:59:59Z 1.081799ms
measures  autogen 324     000000015-000000002.tsm 13487   37        2022-06-06T00:00:00Z 2022-06-06T23:59:59Z 1.12432ms
measures  autogen 326     000000015-000000002.tsm 14120   644       2022-06-07T00:00:00Z 2022-06-07T23:59:59Z 1.17641ms
measures  autogen 328     000000014-000000002.tsm 14070   364       2022-06-08T00:00:00Z 2022-06-08T23:59:59Z 921.942µs
measures  autogen 330     000000014-000000002.tsm 13854   62        2022-06-09T00:00:00Z 2022-06-09T23:59:59Z 81.316974ms
measures  autogen 332     000000015-000000002.tsm 14239   445       2022-06-10T00:00:00Z 2022-06-10T23:59:59Z 1.050787ms
measures  autogen 334     000000015-000000002.tsm 14109   36        2022-06-11T00:00:00Z 2022-06-11T23:59:59Z 1.024758ms
measures  autogen 336     000000016-000000002.tsm 14151   0         2022-06-12T00:00:00Z 2022-06-12T23:59:59Z 841.824µs
measures  autogen 338     000000018-000000003.tsm 14723   250       2022-06-13T00:00:00Z 2022-06-13T23:59:59Z 961.697µs
measures  autogen 340     000000017-000000002.tsm 15132   310       2022-06-14T00:00:00Z 2022-06-14T23:59:59Z 895.204µs
measures  autogen 342     000000017-000000002.tsm 15275   513       2022-06-15T00:00:00Z 2022-06-15T23:59:59Z 887.176µs
measures  autogen 344     000000017-000000002.tsm 15562   407       2022-06-16T00:00:00Z 2022-06-16T23:59:59Z 1.072807ms
measures  autogen 346     000000018-000000003.tsm 16014   399       2022-06-17T00:00:00Z 2022-06-17T23:59:59Z 1.10627ms
measures  autogen 348     000000659-000000002.tsm 15594   40        2022-06-18T00:00:00Z 2022-06-18T23:59:59Z 1.219078ms
measures  autogen 349     000000655-000000002.tsm 15300   108       2022-06-19T00:00:01Z 2022-06-19T23:59:20Z 1.551301ms
measures  autogen 351     000043069-000000002.tsm 16365   598       2022-06-20T00:04:43Z 2022-06-20T23:59:59Z 6.038947ms
measures  autogen 354     000062571-000000002.tsm 16749   518       2022-06-21T00:00:00Z 2022-06-21T23:59:59Z 1.102775ms
measures  autogen 356     000063568-000000003.tsm 16728   352       2022-06-22T00:00:00Z 2022-06-22T23:59:59Z 1.089733ms
measures  autogen 358     000063911-000000002.tsm 16945   271       2022-06-23T00:00:00Z 2022-06-23T23:59:59Z 1.071481ms
measures  autogen 360     000064809-000000003.tsm 17025   118       2022-06-24T00:00:00Z 2022-06-24T23:59:58Z 1.176687ms
measures  autogen 362     000064330-000000002.tsm 16899   41        2022-06-25T00:00:00Z 2022-06-25T23:59:59Z 1.296276ms
measures  autogen 364     000063625-000000003.tsm 16974   107       2022-06-26T00:00:00Z 2022-06-26T23:59:59Z 1.048704ms
measures  autogen 366     000063229-000000002.tsm 17219   230       2022-06-27T00:00:00Z 2022-06-27T23:59:59Z 1.070574ms
measures  autogen 368     000063382-000000003.tsm 17518   237       2022-06-28T00:00:00Z 2022-06-28T23:59:59Z 1.088376ms
measures  autogen 370     000044940-000000002.tsm 17766   319       2022-06-29T00:00:00Z 2022-06-29T20:43:26Z 1.331376ms
measures  autogen 372     000034298-000000002.tsm 17565   247       2022-06-30T00:14:15Z 2022-06-30T20:46:50Z 1.035767ms
measures  autogen 374     000011502-000000002.tsm 18598   622       2022-07-01T03:03:28Z 2022-07-01T23:59:59Z 1.327519ms
measures  autogen 376     000000020-000000002.tsm 18211   0         2022-07-02T00:00:00Z 2022-07-02T23:59:59Z 1.06314ms
measures  autogen 378     000000627-000000002.tsm 18254   136       2022-07-03T00:00:00Z 2022-07-03T23:59:59Z 1.211287ms
measures  autogen 380     000000035-000000002.tsm 18447   130       2022-07-04T00:00:00Z 2022-07-04T23:59:59Z 1.215077ms
measures  autogen 382     000000467-000000002.tsm 18365   336       2022-07-05T00:00:00Z 2022-07-05T23:59:59Z 1.003446ms
measures  autogen 384     000000035-000000002.tsm 18352   107       2022-07-06T00:00:00Z 2022-07-06T23:59:59Z 1.106262ms
measures  autogen 386     000000297-000000002.tsm 18800   434       2022-07-07T00:00:00Z 2022-07-07T23:59:59Z 63.806903ms
_internal monitor 387     000000029-000000002.tsm 2869    2863      2022-07-08T00:00:00Z 2022-07-08T23:59:50Z 890.386µs
measures  autogen 388     000000035-000000002.tsm 19009   287       2022-07-08T00:00:00Z 2022-07-08T23:59:59Z 14.324961ms
_internal monitor 389     000000137-000000002.tsm 2869    108       2022-07-09T00:00:00Z 2022-07-09T23:59:50Z 683.52µs
measures  autogen 390     000000141-000000002.tsm 18879   0         2022-07-09T00:00:00Z 2022-07-09T23:59:59Z 1.022444ms
_internal monitor 391     000000061-000000002.tsm 2869    109       2022-07-10T00:00:00Z 2022-07-10T21:41:30Z 764.208µs
measures  autogen 392     000000069-000000002.tsm 18766   0         2022-07-10T00:00:00Z 2022-07-10T23:57:37Z 1.100684ms
measures  autogen 393     000000037-000000002.tsm 18987   260       2022-07-11T00:03:18Z 2022-07-11T23:58:15Z 998.865µs
measures  autogen 394     000000037-000000002.tsm 19315   554       2022-07-12T00:03:52Z 2022-07-12T23:55:30Z 1.27939ms
_internal monitor 395     000000037-000000002.tsm 2923    161       2022-07-12T12:45:50Z 2022-07-12T12:46:00Z 474.398µs
measures  autogen 396     000000037-000000002.tsm 19518   691       2022-07-13T00:01:18Z 2022-07-13T23:44:53Z 3.105263ms
measures  autogen 397     000000836-000000002.tsm 21044   620       2022-07-14T00:04:34Z 2022-07-14T23:59:59Z 1.066556ms
_internal monitor 398     000000102-000000004.tsm 3085    163       2022-07-14T02:50:30Z 2022-07-14T23:59:50Z 672.903µs
_internal monitor 399     000000002-000000001.tsm 2599    108       2022-07-15T01:42:10Z 2022-07-15T03:24:40Z 455.202µs
_internal monitor 399     000000005-000000001.tsm 2599    0         2022-07-15T06:50:10Z 2022-07-15T08:32:40Z 495.398µs
_internal monitor 399     000000004-000000001.tsm 2599    0         2022-07-15T05:07:30Z 2022-07-15T06:50:00Z 525.34µs
_internal monitor 399     000000003-000000001.tsm 2599    0         2022-07-15T03:24:50Z 2022-07-15T05:07:20Z 530.806µs
_internal monitor 399     000000001-000000001.tsm 2707    0         2022-07-15T00:00:00Z 2022-07-15T01:42:00Z 467.984µs
measures  autogen 400     000000008-000000002.tsm 19825   63        2022-07-15T00:00:00Z 2022-07-15T08:36:50Z 1.163938ms
measures  autogen 400     000000009-000000001.tsm 19940   98        2022-07-15T08:35:44Z 2022-07-15T09:59:31Z 1.157804ms

Summary:
  Files: 52
  Time Range: 2022-06-05T00:00:00Z - 2022-07-15T09:59:31Z
  Duration: 969h59m31s 

Statistics
  Series:
     - measures (est): 24386 (87%)
     - _internal (est): 3512 (12%)
  Total (est): 27859
Completed in 757.76343ms

The Kubernetes configuration gives about 1500Mi Memory per StatefulSet, and about 300m for CPU.
I believe the problem resides in the tsmi configuration, but I have no idea what are appropiate values for it. Any idea?