Hi,
We have an Influx Setup with a Retention Policy of 40 days and a Shard Duration of 1 day. The database, contains about
7813 different measurements.
This are the diagnostics:
> show diagnostics
name: build
Branch Build Time Commit Version
------ ---------- ------ -------
1.7.11 f11ad4780c8a61108108a18b141c1d067d920a80 1.7.11
name: config
bind-address reporting-disabled
------------ ------------------
127.0.0.1:8088 false
name: config-coordinator
log-queries-after max-concurrent-queries max-select-buckets max-select-point max-select-series query-timeout write-timeout
----------------- ---------------------- ------------------ ---------------- ----------------- ------------- -------------
0s 0 0 0 0 0s 10s
name: config-cqs
enabled query-stats-enabled run-interval
------- ------------------- ------------
true false 1s
name: config-data
cache-max-memory-size cache-snapshot-memory-size cache-snapshot-write-cold-duration compact-full-write-cold-duration dir max-concurrent-compactions max-index-log-file-size max-series-per-database max-values-per-tag series-id-set-cache-size wal-dir wal-fsync-delay
--------------------- -------------------------- ---------------------------------- -------------------------------- --- -------------------------- ----------------------- ----------------------- ------------------ ------------------------ ------- ---------------
1073741824 26214400 10m0s 4h0m0s /var/lib/influxdb/data 0 1048576 1000000 100000 100 /var/lib/influxdb/wal 0s
name: config-httpd
access-log-path bind-address enabled https-enabled max-connection-limit max-row-limit
--------------- ------------ ------- ------------- -------------------- -------------
:8086 true false 0 0
name: config-meta
dir
---
/var/lib/influxdb/meta
name: config-monitor
store-database store-enabled store-interval
-------------- ------------- --------------
_internal true 10s
name: config-precreator
advance-period check-interval enabled
-------------- -------------- -------
30m0s 10m0s true
name: config-retention
check-interval enabled
-------------- -------
30m0s true
name: config-subscriber
enabled http-timeout write-buffer-size write-concurrency
------- ------------ ----------------- -----------------
true 30s 1000 40
name: network
hostname
--------
equinox-measure-db-2
name: runtime
GOARCH GOMAXPROCS GOOS version
------ ---------- ---- -------
amd64 2 linux go1.13.8
name: system
PID currentTime started uptime
--- ----------- ------- ------
1 2022-07-15T08:00:19.743038094Z 2022-07-14T23:36:56.515900801Z 8h23m23.227137293s
The performance is outstanding, but certain times the InfluxDB starts compacting files and demanding an outrageous ammount of memory, that ends up colapsing the pod (we have 3-StatefulSet configuration on Kubernetes)
and thus reseting and reseting the pod.
ts=2022-07-14T23:36:21.539857Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgly0gl000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:21.723065Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000353-000000003.tsm
ts=2022-07-14T23:36:21.723402Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000417-000000003.tsm
ts=2022-07-14T23:36:21.723690Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000433-000000002.tsm
ts=2022-07-14T23:36:21.723993Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000449-000000002.tsm
ts=2022-07-14T23:36:21.724281Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/lib/influxdb/data/measures/autogen/382/000000465-000000002.tsm
ts=2022-07-14T23:36:35.041983Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglykrW000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:35.648758Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000015-000000002.tsm
ts=2022-07-14T23:36:35.648765Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000031-000000002.tsm
ts=2022-07-14T23:36:35.648770Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000033-000000001.tsm
ts=2022-07-14T23:36:35.648775Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/394/000000035-000000001.tsm
ts=2022-07-14T23:36:36.279794Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzbG0000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:36.647693Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000015-000000002.tsm
ts=2022-07-14T23:36:36.647699Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000031-000000002.tsm
ts=2022-07-14T23:36:36.647704Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000033-000000001.tsm
ts=2022-07-14T23:36:36.647711Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/393/000000035-000000001.tsm
ts=2022-07-14T23:36:37.161966Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzf9l000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:37.648826Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/392/000000063-000000003.tsm
ts=2022-07-14T23:36:37.648848Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/392/000000065-000000001.tsm
ts=2022-07-14T23:36:37.648866Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/392/000000067-000000001.tsm
ts=2022-07-14T23:36:47.859812Z lvl=info msg="Finished compacting files" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bglzj40000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-07-14T23:36:48.649430Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000063-000000003.tsm
ts=2022-07-14T23:36:48.649492Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000127-000000003.tsm
ts=2022-07-14T23:36:48.649585Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000129-000000001.tsm
ts=2022-07-14T23:36:48.649624Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000131-000000001.tsm
ts=2022-07-14T23:36:48.649691Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000133-000000001.tsm
ts=2022-07-14T23:36:48.649735Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000135-000000001.tsm
ts=2022-07-14T23:36:48.649867Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000137-000000001.tsm
ts=2022-07-14T23:36:48.649908Z lvl=info msg="Compacting file" log_id=0bgYa0zl000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0bgl~P2G000 op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/lib/influxdb/data/measures/autogen/390/000000139-000000001.tsm
TSM files are about 50MB size. In this case the very last compaction seems to have failed. When re-entering the POD, for this 390 entry… the TSM file is just 30MB.
Reports say it’s healthy, though.
bash-5.1# influx_inspect verify -dir /var/lib/influxdb/
/var/lib/influxdb/data/_internal/monitor/387/000000029-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/389/000000137-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/391/000000061-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/395/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/398/000000102-000000004.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000001-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000002-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000003-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000004-000000001.tsm: healthy
/var/lib/influxdb/data/_internal/monitor/399/000000005-000000001.tsm: healthy
/var/lib/influxdb/data/measures/autogen/322/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/324/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/326/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/328/000000014-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/330/000000014-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/332/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/334/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/336/000000016-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/338/000000018-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/340/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/342/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/344/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/346/000000018-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/348/000000659-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/349/000000655-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/351/000043069-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/354/000062571-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/356/000063568-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/358/000063911-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/360/000064809-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/362/000064330-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/364/000063625-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/366/000063229-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/368/000063382-000000003.tsm: healthy
/var/lib/influxdb/data/measures/autogen/370/000044940-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/372/000034298-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/374/000011502-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/376/000000020-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/378/000000627-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/380/000000035-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/382/000000467-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/384/000000035-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/386/000000297-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/388/000000035-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/390/000000141-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/392/000000069-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/393/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/394/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/396/000000037-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/397/000000836-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/400/000000008-000000002.tsm: healthy
/var/lib/influxdb/data/measures/autogen/400/000000009-000000001.tsm: healthy
Broken Blocks: 0 / 1170443, in 15.453021259s
bash-5.1# influx_inspect report /var/lib/influxdb/data/
DB RP Shard File Series New (est) Min Time Max Time Load Time
measures autogen 322 000000015-000000002.tsm 13443 13395 2022-06-05T00:00:00Z 2022-06-05T23:59:59Z 1.081799ms
measures autogen 324 000000015-000000002.tsm 13487 37 2022-06-06T00:00:00Z 2022-06-06T23:59:59Z 1.12432ms
measures autogen 326 000000015-000000002.tsm 14120 644 2022-06-07T00:00:00Z 2022-06-07T23:59:59Z 1.17641ms
measures autogen 328 000000014-000000002.tsm 14070 364 2022-06-08T00:00:00Z 2022-06-08T23:59:59Z 921.942µs
measures autogen 330 000000014-000000002.tsm 13854 62 2022-06-09T00:00:00Z 2022-06-09T23:59:59Z 81.316974ms
measures autogen 332 000000015-000000002.tsm 14239 445 2022-06-10T00:00:00Z 2022-06-10T23:59:59Z 1.050787ms
measures autogen 334 000000015-000000002.tsm 14109 36 2022-06-11T00:00:00Z 2022-06-11T23:59:59Z 1.024758ms
measures autogen 336 000000016-000000002.tsm 14151 0 2022-06-12T00:00:00Z 2022-06-12T23:59:59Z 841.824µs
measures autogen 338 000000018-000000003.tsm 14723 250 2022-06-13T00:00:00Z 2022-06-13T23:59:59Z 961.697µs
measures autogen 340 000000017-000000002.tsm 15132 310 2022-06-14T00:00:00Z 2022-06-14T23:59:59Z 895.204µs
measures autogen 342 000000017-000000002.tsm 15275 513 2022-06-15T00:00:00Z 2022-06-15T23:59:59Z 887.176µs
measures autogen 344 000000017-000000002.tsm 15562 407 2022-06-16T00:00:00Z 2022-06-16T23:59:59Z 1.072807ms
measures autogen 346 000000018-000000003.tsm 16014 399 2022-06-17T00:00:00Z 2022-06-17T23:59:59Z 1.10627ms
measures autogen 348 000000659-000000002.tsm 15594 40 2022-06-18T00:00:00Z 2022-06-18T23:59:59Z 1.219078ms
measures autogen 349 000000655-000000002.tsm 15300 108 2022-06-19T00:00:01Z 2022-06-19T23:59:20Z 1.551301ms
measures autogen 351 000043069-000000002.tsm 16365 598 2022-06-20T00:04:43Z 2022-06-20T23:59:59Z 6.038947ms
measures autogen 354 000062571-000000002.tsm 16749 518 2022-06-21T00:00:00Z 2022-06-21T23:59:59Z 1.102775ms
measures autogen 356 000063568-000000003.tsm 16728 352 2022-06-22T00:00:00Z 2022-06-22T23:59:59Z 1.089733ms
measures autogen 358 000063911-000000002.tsm 16945 271 2022-06-23T00:00:00Z 2022-06-23T23:59:59Z 1.071481ms
measures autogen 360 000064809-000000003.tsm 17025 118 2022-06-24T00:00:00Z 2022-06-24T23:59:58Z 1.176687ms
measures autogen 362 000064330-000000002.tsm 16899 41 2022-06-25T00:00:00Z 2022-06-25T23:59:59Z 1.296276ms
measures autogen 364 000063625-000000003.tsm 16974 107 2022-06-26T00:00:00Z 2022-06-26T23:59:59Z 1.048704ms
measures autogen 366 000063229-000000002.tsm 17219 230 2022-06-27T00:00:00Z 2022-06-27T23:59:59Z 1.070574ms
measures autogen 368 000063382-000000003.tsm 17518 237 2022-06-28T00:00:00Z 2022-06-28T23:59:59Z 1.088376ms
measures autogen 370 000044940-000000002.tsm 17766 319 2022-06-29T00:00:00Z 2022-06-29T20:43:26Z 1.331376ms
measures autogen 372 000034298-000000002.tsm 17565 247 2022-06-30T00:14:15Z 2022-06-30T20:46:50Z 1.035767ms
measures autogen 374 000011502-000000002.tsm 18598 622 2022-07-01T03:03:28Z 2022-07-01T23:59:59Z 1.327519ms
measures autogen 376 000000020-000000002.tsm 18211 0 2022-07-02T00:00:00Z 2022-07-02T23:59:59Z 1.06314ms
measures autogen 378 000000627-000000002.tsm 18254 136 2022-07-03T00:00:00Z 2022-07-03T23:59:59Z 1.211287ms
measures autogen 380 000000035-000000002.tsm 18447 130 2022-07-04T00:00:00Z 2022-07-04T23:59:59Z 1.215077ms
measures autogen 382 000000467-000000002.tsm 18365 336 2022-07-05T00:00:00Z 2022-07-05T23:59:59Z 1.003446ms
measures autogen 384 000000035-000000002.tsm 18352 107 2022-07-06T00:00:00Z 2022-07-06T23:59:59Z 1.106262ms
measures autogen 386 000000297-000000002.tsm 18800 434 2022-07-07T00:00:00Z 2022-07-07T23:59:59Z 63.806903ms
_internal monitor 387 000000029-000000002.tsm 2869 2863 2022-07-08T00:00:00Z 2022-07-08T23:59:50Z 890.386µs
measures autogen 388 000000035-000000002.tsm 19009 287 2022-07-08T00:00:00Z 2022-07-08T23:59:59Z 14.324961ms
_internal monitor 389 000000137-000000002.tsm 2869 108 2022-07-09T00:00:00Z 2022-07-09T23:59:50Z 683.52µs
measures autogen 390 000000141-000000002.tsm 18879 0 2022-07-09T00:00:00Z 2022-07-09T23:59:59Z 1.022444ms
_internal monitor 391 000000061-000000002.tsm 2869 109 2022-07-10T00:00:00Z 2022-07-10T21:41:30Z 764.208µs
measures autogen 392 000000069-000000002.tsm 18766 0 2022-07-10T00:00:00Z 2022-07-10T23:57:37Z 1.100684ms
measures autogen 393 000000037-000000002.tsm 18987 260 2022-07-11T00:03:18Z 2022-07-11T23:58:15Z 998.865µs
measures autogen 394 000000037-000000002.tsm 19315 554 2022-07-12T00:03:52Z 2022-07-12T23:55:30Z 1.27939ms
_internal monitor 395 000000037-000000002.tsm 2923 161 2022-07-12T12:45:50Z 2022-07-12T12:46:00Z 474.398µs
measures autogen 396 000000037-000000002.tsm 19518 691 2022-07-13T00:01:18Z 2022-07-13T23:44:53Z 3.105263ms
measures autogen 397 000000836-000000002.tsm 21044 620 2022-07-14T00:04:34Z 2022-07-14T23:59:59Z 1.066556ms
_internal monitor 398 000000102-000000004.tsm 3085 163 2022-07-14T02:50:30Z 2022-07-14T23:59:50Z 672.903µs
_internal monitor 399 000000002-000000001.tsm 2599 108 2022-07-15T01:42:10Z 2022-07-15T03:24:40Z 455.202µs
_internal monitor 399 000000005-000000001.tsm 2599 0 2022-07-15T06:50:10Z 2022-07-15T08:32:40Z 495.398µs
_internal monitor 399 000000004-000000001.tsm 2599 0 2022-07-15T05:07:30Z 2022-07-15T06:50:00Z 525.34µs
_internal monitor 399 000000003-000000001.tsm 2599 0 2022-07-15T03:24:50Z 2022-07-15T05:07:20Z 530.806µs
_internal monitor 399 000000001-000000001.tsm 2707 0 2022-07-15T00:00:00Z 2022-07-15T01:42:00Z 467.984µs
measures autogen 400 000000008-000000002.tsm 19825 63 2022-07-15T00:00:00Z 2022-07-15T08:36:50Z 1.163938ms
measures autogen 400 000000009-000000001.tsm 19940 98 2022-07-15T08:35:44Z 2022-07-15T09:59:31Z 1.157804ms
Summary:
Files: 52
Time Range: 2022-06-05T00:00:00Z - 2022-07-15T09:59:31Z
Duration: 969h59m31s
Statistics
Series:
- measures (est): 24386 (87%)
- _internal (est): 3512 (12%)
Total (est): 27859
Completed in 757.76343ms
The Kubernetes configuration gives about 1500Mi Memory per StatefulSet, and about 300m for CPU.
I believe the problem resides in the tsmi configuration, but I have no idea what are appropiate values for it. Any idea?