Infuxdb V2 startup fails - suspected data corruption

Dee_Gee · March 12, 2022, 10:09am

I have a problem which is a more informed version of a previous support request:

Objective:

monitor engine data on many many vehicles
monitor locally with raspberry pi
downsample locally and sent further down sampled data to Azure server

Setup

processor board is a Raspberry Pi compute module 4 lite
most disk io is on PCIe M.2 ssd
It seems failure though is on the SD card which atop shows has 101% activity
base data collected on a millisecond basis
locally data downsampled to 1s with a task, this seems OK
down sampling to 30s data is output to Azure … this struggles when collecting engine data
RPI when collecting engine data has CPU approaching 100% (all 4 processors)
I have installed PCIe NVMe 2244 SSD to achieve disk throughput for influx - great write rate

Symptoms:

run 30sec downsampling every 15 min and it struggles … sometimes not completing in 15 min
if task incomplete, after shutdown BOOT startup fails

Suspicion & Observations:

FIRSTLY … I think the ultimate cause is because the downsampling task to 30 sec fails to complete within 30 sec. Then another one starts
I think this is because the data saving is consuming close to 100% of the 4 CPUs
adding a downsampling just creates too much load
it seems when reboot after failed task - influx startup fails
I suspect data corruption
no errors on startup speed until task is killed
corrupted startup consumes ~100% of 4 CPUs and 85% Memory
once influx stopped CPU and Memory negligible
after startup cmd killed CPU and Memory reduces to negligible

Actions to asses (oops!):

changed startup failure time to 10 hour
changed debug output from 1sec to 1min for

$  sudo systemctl show influxdb | grep ^Timeout
TimeoutStartUSec=10h
TimeoutStopUSec=1min 30s
TimeoutAbortUSec=1min 30s
TimeoutStartFailureMode=terminate
TimeoutStopFailureMode=terminate
TimeoutCleanUSec=infinity

changed debug reporting time from 1sec to 60sec in /usr/lib/influxdb/scripts/influxd-systemd-start.sh

Configuration:
RPi InfluxDB version=2.1.1 commit=657e1839de

uname -a
Linux e45f013eb289 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

64bit …

$ uname -m
aarch64

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian

Diagnostics:

no error level messages

$ journalctl -b 0 -u influxdb
-- Journal begins at Fri 2022-01-28 03:15:01 UTC, ends at Sat 2022-03-12 07:34:10 UTC. --
Mar 12 02:01:55 e45f013eb289 systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.044063Z lvl=info msg="Welcome to InfluxDB" log_id=0_AycXjG000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.074748Z lvl=info msg="Resources opened" log_id=0_AycXjG000 service=bolt path=/nvmedata/usb/influxdb/influxd.bolt
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.075374Z lvl=info msg="Resources opened" log_id=0_AycXjG000 service=sqlite path=/nvmedata/usb/influxdb/influxd.sqlite
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.177318Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0_AycXjG000 bolt_path=/nvmedata/usb/influxdb/influxd.bolt
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.185990Z lvl=info msg="Using data dir" log_id=0_AycXjG000 service=storage-engine service=store path=/nvmedata/usb/influxdb/engine/data
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.186216Z lvl=info msg="Compaction settings" log_id=0_AycXjG000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.186271Z lvl=info msg="Open store (start)" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open op_event=start
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts... result= 000
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.272276Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.325940Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/682b9c75d02514d4/autogen/37/000000005-000000001.tsm id=1 duration=10.651ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.330004Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/682b9c75d02514d4/autogen/37/000000003-000000002.tsm id=0 duration=9.207ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.336453Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.345564Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/6c7bd8e7a33be275/autogen/10/000000004-000000002.tsm id=0 duration=3.023ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.352628Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.356381Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/6c7bd8e7a33be275/autogen/31/000000004-000000002.tsm id=0 duration=0.779ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.380168Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.388642Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/682b9c75d02514d4/autogen/37 duration=175.476ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.388708Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/6c7bd8e7a33be275/autogen/10 duration=131.247ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.388881Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/6c7bd8e7a33be275/autogen/31 duration=132.613ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.409462Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/39/000000001-000000001.tsm id=0 duration=21.806ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.411104Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/39 duration=99.881ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.525000Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.530540Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/33/000000004-000000002.tsm id=0 duration=1.491ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.531998Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/33 duration=142.104ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.536963Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.553600Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/35/000000001-000000001.tsm id=0 duration=4.968ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.554570Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/35 duration=164.371ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.573835Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.637790Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/36/000000001-000000001.tsm id=0 duration=28.027ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.638623Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/36 duration=248.314ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.636864Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.623221Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.650296Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/aa5305b98283ad62/autogen/5/000000001-000000001.tsm id=0 duration=2.503ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.651262Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/aa5305b98283ad62/autogen/5 duration=118.366ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.687736Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/38/000000001-000000001.tsm id=0 duration=35.181ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.688609Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/83be37adc16da39d/autogen/38 duration=277.272ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.720819Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.742331Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/aa5305b98283ad62/autogen/28/000000006-000000001.tsm id=1 duration=7.960ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.751041Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/aa5305b98283ad62/autogen/28/000000004-000000002.tsm id=0 duration=7.274ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.752729Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/aa5305b98283ad62/autogen/28 duration=197.937ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.756531Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.770868Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/d503b8cf27253f4f/autogen/32/000000001-000000001.tsm id=0 duration=5.674ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.771561Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/d503b8cf27253f4f/autogen/32 duration=119.899ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.900149Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.903402Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.906664Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.907698Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/aa5305b98283ad62/autogen/3/000000003-000000002.tsm id=0 duration=4.790ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.908214Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/aa5305b98283ad62/autogen/3 duration=269.284ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.909385Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/d503b8cf27253f4f/autogen/6/000000003-000000002.tsm id=0 duration=2.743ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.916260Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/d503b8cf27253f4f/autogen/4/000000001-000000001.tsm id=0 duration=3.677ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.918938Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/d503b8cf27253f4f/autogen/4 duration=230.105ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.919030Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/d503b8cf27253f4f/autogen/6 duration=165.305ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.923080Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.926846Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/d7fe24428a72b90b/autogen/29/000000005-000000001.tsm id=1 duration=0.727ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.927919Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/d7fe24428a72b90b/autogen/29/000000003-000000002.tsm id=0 duration=1.808ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.928939Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/d7fe24428a72b90b/autogen/29 duration=157.228ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.943837Z lvl=info msg="index opened with 8 partitions" log_id=0_AycXjG000 service=storage-engine index=tsi
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.946673Z lvl=info msg="Opened file" log_id=0_AycXjG000 service=storage-engine engine=tsm1 service=filestore path=/nvmedata/usb/influxdb/engine/data/d7fe24428a72b90b/autogen/7/000000010-000000002.tsm id=0 duration=0.629ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.947365Z lvl=info msg="Opened shard" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/nvmedata/usb/influxdb/engine/data/d7fe24428a72b90b/autogen/7 duration=38.718ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.951031Z lvl=info msg="Open store (end)" log_id=0_AycXjG000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=764.765ms
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.951283Z lvl=info msg="Starting retention policy enforcement service" log_id=0_AycXjG000 service=retention check_interval=30m
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.951628Z lvl=info msg="Starting precreation service" log_id=0_AycXjG000 service=shard-precreation check_interval=10m advance_period=30m
Mar 12 02:02:02 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:02:02.955578Z lvl=info msg="Starting query controller" log_id=0_AycXjG000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Mar 12 02:03:19 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 2 attempts... result= 000
Mar 12 02:04:22 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 3 attempts... result= 000
Mar 12 02:05:24 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 4 attempts... result= 000
Mar 12 02:06:27 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 5 attempts... result= 000
Mar 12 02:07:31 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 6 attempts... result= 000
Mar 12 02:08:32 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 7 attempts... result= 000
Mar 12 02:09:37 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 8 attempts... result= 000
Mar 12 02:10:42 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 9 attempts... result= 000
Mar 12 02:11:47 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 10 attempts... result= 000
Mar 12 02:12:48 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 11 attempts... result= 000
Mar 12 02:13:50 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 12 attempts... result= 000
Mar 12 02:14:57 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 13 attempts... result= 000
Mar 12 02:16:06 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 14 attempts... result= 000
Mar 12 02:17:09 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 15 attempts... result= 000
Mar 12 02:18:13 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 16 attempts... result= 000
Mar 12 02:19:17 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 17 attempts... result= 000
Mar 12 02:20:21 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 18 attempts... result= 000
Mar 12 02:21:28 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 19 attempts... result= 000
Mar 12 02:22:41 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 20 attempts... result= 000
Mar 12 02:23:48 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 21 attempts... result= 000
Mar 12 02:24:51 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 22 attempts... result= 000
Mar 12 02:26:00 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 23 attempts... result= 000
Mar 12 02:27:03 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 24 attempts... result= 000
Mar 12 02:28:09 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 25 attempts... result= 000
Mar 12 02:29:17 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 26 attempts... result= 000
Mar 12 02:30:21 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 27 attempts... result= 000
Mar 12 02:31:28 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 28 attempts... result= 000
Mar 12 02:32:38 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 29 attempts... result= 000
Mar 12 02:32:42 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:32:29.670748Z lvl=info msg="Retention policy deletion check (start)" log_id=0_AycXjG000 service=retention op_name=retention_delete_check op_event=start
Mar 12 02:33:54 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 30 attempts... result= 000
Mar 12 02:34:22 e45f013eb289 influxd-systemd-start.sh[918]: ts=2022-03-12T02:34:18.366630Z lvl=info msg="Deleted shard group" log_id=0_AycXjG000 service=retention op_name=retention_delete_check db_instance=83be37adc16da39d db_shard_group=33 db_rp=autogen
Mar 12 02:35:30 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 31 attempts... result= 000
Mar 12 02:36:59 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 32 attempts... result= 000
Mar 12 02:37:59 e45f013eb289 influxd-systemd-start.sh[914]: /usr/lib/influxdb/scripts/influxd-systemd-start.sh: line 28:   918 Killed                  /usr/bin/influxd
Mar 12 02:37:59 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 33 attempts... result= 000
Mar 12 02:38:59 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 34 attempts... result= 000
Mar 12 02:39:59 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 35 attempts... result= 000
Mar 12 02:40:59 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 36 attempts... result= 000
Mar 12 02:41:59 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 37 attempts... result= 000
Mar 12 02:42:59 e45f013eb289 influxd-systemd-start.sh[914]: InfluxDB API at http://localhost:8086/ready unavailable after 38 attempts... result= 000

Dee_Gee · March 22, 2022, 6:56am

I have made changes to the config.toml to allow only 2 queries and each query less than 50% RAM
Performance seems better

query-queue-size = 2
query-memory-bytes = 750000000

But still, 10sec aggregate worked but 1sec aggregate failed when run together

Dee_Gee · March 22, 2022, 8:05am

The following errors are the best debug I have.
How can I get more verbose debug?

Mar 22 08:03:38 e45f013eb289 influxd-systemd-start.sh[1167]: ts=2022-03-22T08:03:38.864900Z lvl=info msg="Error exhausting result iterator" log_id=0_O55jA0000 service=task-executor error="context canceled" name=to9
Mar 22 08:03:39 e45f013eb289 influxd-systemd-start.sh[1167]: ts=2022-03-22T08:03:39.147910Z lvl=error msg="Failed to finish run" log_id=0_O55jA0000 service=task-executor taskID=091815e817270000 runID=0918290da1e70000 error="engine: context canceled"

Load 1 CAN bus…

Load with 2 x CAN buses

mhall119 · March 23, 2022, 7:43pm

Can you share the full Flux code for your two tasks?

Dee_Gee · March 24, 2022, 2:26am

I was trying to trigger downsampling 5m then 15 min then 30h and running manually
I downsample ms data to 1 sec locally on Rpi (this fails) next step would be to downsample and send to server at 30s- this typically is successful nwhen run manually

Task Sample

import "influxdata/influxdb/tasks"
option task = {
    name: "xxclient_1s",
    every: 30h,
}

dataset = from(bucket: "xxclient")
    |> range(start: tasks.lastSuccess(orTime: -2d))
    |> filter(fn: (r) => r["_measurement"] == "Electronic Engine Controller 1" or r["_measurement"] == "COG & SOG, Rapid Update" or r["_measurement"] == "Position, Rapid Update" or r["_measurement"] == "Water Depth" or r["_measurement"] == "Temperature" or r["_measurement"] == "Fuel Consumption (Liquid) 1" or r["_measurement"] == "Fuel Economy (Liquid)" or r["_measurement"] == "Intake/Exhaust Conditions 1" or r["_measurement"] == "Engine Temperature 1" or r["_measurement"] == "Engine Hours, Revolutions" or r["_measurement"] == "Electronic Engine Controller 2" or r["_measurement"] == "Tachograph")
    |> filter(fn: (r) => r["_field"] != "Engine Starter Mode")

dataset
    |> aggregateWindow(every: 1s, fn: mean)
    |> to(
        org: "myOrg",
        bucket: "xxclient_1s",
    )

Topic		Replies	Views
RPi influxdb-v2 Losing access - connection refused InfluxDB 2 raspberry	4	1865	March 20, 2022
Performance & Stability InfluxDB 2	0	583	January 2, 2021
Is influxdb stable on raspberry PI influxdb	3	563	August 8, 2022
Influxdb 2 container problems InfluxDB 2 influxdb , backup , performance , raspberry	2	336	January 4, 2024
InfluxDB is super slow to start on Raspberry Pi InfluxDB 2 getting-started	1	2865	February 26, 2020

Infuxdb V2 startup fails - suspected data corruption

Related topics