Hi,
My goal is to have a fleet of RaspberryPi based devices with Influxdb-v2 collecting lots of data. I am downsampling to 1s on the RPi with a task and another task downsampling to 30s and sending to a server. This generally works.
I am writing A LOT of data but the %CPU seems OK.
My problem is that I get a fault on the RPi (details later) that causes influx to become inaccessible.
curl -k -s http://localhost:8086/ready
NO RESPONSE
$ influx config list
Active Name URL Org
* default http://localhost:8086 myOrg
$ influx org list
Error: failed to list orgs: Get "http://localhost:8086/api/v2/orgs": dial tcp [::1]:8086: connect: connection refused
$ influx bucket list
Error: failed to list buckets: Get "http://localhost:8086/api/v2/buckets?limit=20&org=myOrg": dial tcp [::1]:8086: connect: connection refused
Journal …
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.224208Z lvl=info msg="Welcome to InfluxDB" log_id=0Y3iCnwl000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.433689Z lvl=info msg="Resources opened" log_id=0Y3iCnwl000 service=bolt path=/var/lib/influxdb/influxd.bolt
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.434483Z lvl=info msg="Resources opened" log_id=0Y3iCnwl000 service=sqlite path=/var/lib/influxdb/influxd.sqlite
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.606161Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0Y3iCnwl000 bolt_path=/var/lib/influxdb/influxd.bolt
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.611287Z lvl=info msg="Using data dir" log_id=0Y3iCnwl000 service=storage-engine service=store path=/var/lib/influxdb/engine/data
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.611423Z lvl=info msg="Compaction settings" log_id=0Y3iCnwl000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.611489Z lvl=info msg="Open store (start)" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open op_event=start
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.812596Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.830697Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.849139Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.870805Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/063d19dabfb8c20e/autogen/14/_00001.wal size=38719
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.915996Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/8/000000002-000000002.tsm id=0 duration=44.911ms
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.916395Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/063d19dabfb8c20e/autogen/8/_00003.wal size=358358
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.933122Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/2/000000001-000000001.tsm id=0 duration=26.948ms
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.981255Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.007815Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/4e299239e660ce70/autogen/13/_00001.wal size=72280
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162483Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/2 duration=483.603ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162494Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/8 duration=483.589ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162439Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/4e299239e660ce70/autogen/13 duration=279.372ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162678Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/14 duration=483.432ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.282568Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.285877Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.285918Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.289960Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.356496Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/1/000000001-000000001.tsm id=0 duration=0.331ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.356873Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/1 duration=194.140ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.389293Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/4/000000001-000000001.tsm id=0 duration=32.267ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.389655Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/4 duration=225.996ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.392782Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/5/000000001-000000001.tsm id=0 duration=35.032ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.393072Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/5 duration=230.364ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.416379Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/11/000000001-000000001.tsm id=0 duration=52.019ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.416649Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/11 duration=252.899ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 2 attempts...
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.512858Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.513322Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.532881Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.544303Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.556020Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/8153c1c4c598769d/autogen/12/_00001.wal size=307997
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.583908Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/6/000000005-000000002.tsm id=0 duration=31.742ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.584461Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/6 duration=227.516ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.587518Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/7/000000001-000000001.tsm id=0 duration=30.614ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.587952Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/7 duration=198.219ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.648677Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/e02b0ab0bcbdc7b8/autogen/3/000000005-000000002.tsm id=0 duration=56.476ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.649090Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/e02b0ab0bcbdc7b8/autogen/3/_00001.wal size=10620578
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.662050Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/12 duration=268.920ms
Nov 27 00:40:12 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 3 attempts...
Nov 27 00:40:12 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:12.684460Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/e02b0ab0bcbdc7b8/autogen/3/_00002.wal size=5679039
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.262287Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/e02b0ab0bcbdc7b8/autogen/3 duration=1845.594ms
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.265842Z lvl=info msg="Open store (end)" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=2654.356ms
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.266476Z lvl=info msg="Starting retention policy enforcement service" log_id=0Y3iCnwl000 service=retention check_interval=30m
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.266568Z lvl=info msg="Starting precreation service" log_id=0Y3iCnwl000 service=shard-precreation check_interval=10m advance_period=30m
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.267678Z lvl=info msg="Starting query controller" log_id=0Y3iCnwl000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 4 attempts...
Nov 27 00:40:14 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 5 attempts...
Nov 27 00:40:15 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 6 attempts...
Nov 27 00:40:16 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 7 attempts...
Nov 27 00:40:17 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 8 attempts...
Linux e45f013eb289 5.10.63-v8+ #1459 SMP PREEMPT Wed Oct 6 16:42:49 BST 2021 aarch64 GNU/Linux
InfluxDB 2.1.1 (git: 657e1839de) build_date: 2021-11-09T03:03:48Z
Any suggestions… I have looked at LOTS of attempted solutions!!!