RPi influxdb-v2 Losing access - connection refused

Dee_Gee · November 27, 2021, 2:28am

Hi,

My goal is to have a fleet of RaspberryPi based devices with Influxdb-v2 collecting lots of data. I am downsampling to 1s on the RPi with a task and another task downsampling to 30s and sending to a server. This generally works.

I am writing A LOT of data but the %CPU seems OK.

My problem is that I get a fault on the RPi (details later) that causes influx to become inaccessible.

curl -k -s http://localhost:8086/ready 
NO RESPONSE

$ influx config list
Active  Name    URL                     Org
*       default http://localhost:8086   myOrg

$ influx org list
Error: failed to list orgs: Get "http://localhost:8086/api/v2/orgs": dial tcp [::1]:8086: connect: connection refused

$ influx bucket list
Error: failed to list buckets: Get "http://localhost:8086/api/v2/buckets?limit=20&org=myOrg": dial tcp [::1]:8086: connect: connection refused

Journal …

Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.224208Z lvl=info msg="Welcome to InfluxDB" log_id=0Y3iCnwl000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.433689Z lvl=info msg="Resources opened" log_id=0Y3iCnwl000 service=bolt path=/var/lib/influxdb/influxd.bolt
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.434483Z lvl=info msg="Resources opened" log_id=0Y3iCnwl000 service=sqlite path=/var/lib/influxdb/influxd.sqlite
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.606161Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0Y3iCnwl000 bolt_path=/var/lib/influxdb/influxd.bolt
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.611287Z lvl=info msg="Using data dir" log_id=0Y3iCnwl000 service=storage-engine service=store path=/var/lib/influxdb/engine/data
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.611423Z lvl=info msg="Compaction settings" log_id=0Y3iCnwl000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.611489Z lvl=info msg="Open store (start)" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open op_event=start
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.812596Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.830697Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.849139Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.870805Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/063d19dabfb8c20e/autogen/14/_00001.wal size=38719
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.915996Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/8/000000002-000000002.tsm id=0 duration=44.911ms
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.916395Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/063d19dabfb8c20e/autogen/8/_00003.wal size=358358
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.933122Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/2/000000001-000000001.tsm id=0 duration=26.948ms
Nov 27 00:40:10 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:10.981255Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.007815Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/4e299239e660ce70/autogen/13/_00001.wal size=72280
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162483Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/2 duration=483.603ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162494Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/8 duration=483.589ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162439Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/4e299239e660ce70/autogen/13 duration=279.372ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.162678Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/063d19dabfb8c20e/autogen/14 duration=483.432ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.282568Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.285877Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.285918Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.289960Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.356496Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/1/000000001-000000001.tsm id=0 duration=0.331ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.356873Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/1 duration=194.140ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.389293Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/4/000000001-000000001.tsm id=0 duration=32.267ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.389655Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/4 duration=225.996ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.392782Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/5/000000001-000000001.tsm id=0 duration=35.032ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.393072Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/5 duration=230.364ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.416379Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/11/000000001-000000001.tsm id=0 duration=52.019ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.416649Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/11 duration=252.899ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 2 attempts...
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.512858Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.513322Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.532881Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.544303Z lvl=info msg="index opened with 8 partitions" log_id=0Y3iCnwl000 service=storage-engine index=tsi
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.556020Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/8153c1c4c598769d/autogen/12/_00001.wal size=307997
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.583908Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/6/000000005-000000002.tsm id=0 duration=31.742ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.584461Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/6 duration=227.516ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.587518Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/7/000000001-000000001.tsm id=0 duration=30.614ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.587952Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/7 duration=198.219ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.648677Z lvl=info msg="Opened file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb/engine/data/e02b0ab0bcbdc7b8/autogen/3/000000005-000000002.tsm id=0 duration=56.476ms
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.649090Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/e02b0ab0bcbdc7b8/autogen/3/_00001.wal size=10620578
Nov 27 00:40:11 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:11.662050Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/8153c1c4c598769d/autogen/12 duration=268.920ms
Nov 27 00:40:12 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 3 attempts...
Nov 27 00:40:12 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:12.684460Z lvl=info msg="Reading file" log_id=0Y3iCnwl000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb/engine/wal/e02b0ab0bcbdc7b8/autogen/3/_00002.wal size=5679039
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.262287Z lvl=info msg="Opened shard" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb/engine/data/e02b0ab0bcbdc7b8/autogen/3 duration=1845.594ms
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.265842Z lvl=info msg="Open store (end)" log_id=0Y3iCnwl000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=2654.356ms
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.266476Z lvl=info msg="Starting retention policy enforcement service" log_id=0Y3iCnwl000 service=retention check_interval=30m
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.266568Z lvl=info msg="Starting precreation service" log_id=0Y3iCnwl000 service=shard-precreation check_interval=10m advance_period=30m
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28902]: ts=2021-11-27T00:40:13.267678Z lvl=info msg="Starting query controller" log_id=0Y3iCnwl000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Nov 27 00:40:13 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 4 attempts...
Nov 27 00:40:14 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 5 attempts...
Nov 27 00:40:15 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 6 attempts...
Nov 27 00:40:16 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 7 attempts...
Nov 27 00:40:17 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 8 attempts...

Linux e45f013eb289 5.10.63-v8+ #1459 SMP PREEMPT Wed Oct 6 16:42:49 BST 2021 aarch64 GNU/Linux
InfluxDB 2.1.1 (git: 657e1839de) build_date: 2021-11-09T03:03:48Z

Any suggestions… I have looked at LOTS of attempted solutions!!!

Anaisdg · December 15, 2021, 6:44pm

Hello @Dee_Gee,
I’m not sure.
This looks related though

github.com/influxdata/influxdb

startup script influxd-systemd-start.sh stuck in while loop if http auth set

opened 09:30AM - 29 Jul 21 UTC

closed 03:07PM - 19 Oct 21 UTC

IgorSimic

__Actual behavior:__ systemd service stuck in /usr/lib/influxdb/scripts/influxd-systemd-start.sh in while loop if http authentification is set I belive that problem is introduced in commit c8de72ddbc5fdf20f821ca473f1fdf92820f9ac3 __Environment info:__ * System info: Linux 5.10.53-sunxi64 aarch64 * InfluxDB version: InfluxDB v1.8.7 (git: 1.8 v1.8.7) * Other relevant environment details: dpkg -l | grep influx ii influxdb 1.8.7-1 arm64 Distributed time-series database. cat /etc/issue Ubuntu 18.04.5 LTS

I’m going to try and loop in an expert.

Jay_Clifford · December 16, 2021, 9:40am

Hi @Dee_Gee,
Just some follow up questions:

How much data is the PI ingesting on average. Could you also provide a sample of this data?
How often are you triggering both downsample tasks? What is your time window for the downsampling?
Does any of the downsampled data get stored back in InfluxDB on the PI? Or is just offloaded to the server?

I am wondering if it’s less of a CPU issue and more of a disk issue.

wbaker · December 28, 2021, 7:41pm

There error logs you posted are related to server startup:

Nov 27 00:40:17 e45f013eb289 influxd-systemd-start.sh[28900]: InfluxDB API at http://localhost:8086/ready unavailable after 8 attempts...

etc. mean that the systemd service is in the process of starting up, and is waiting for the HTTP API to be ready. The HTTP API isn’t ready until all of the other services are ready, and the startup script will wait indefinitely for that to happen.

Is your issue related to the server not start starting successfully? Does it eventually start? Or - is it crashing during normal operation, and then restarting? If it’s crashing and then needing to restart, the logs that happen before the startup/just prior to the crash would be most useful, it it’s starting/crashing in a loop as a result of queries/writes.

Dee_Gee · March 20, 2022, 4:29am

I will get some data size estimates. How do you want samples?
I was trying to trigger downsampling 5m then 15 min then 30h and running manually
I downsample ms data to 1 sec locally ion Rpi (this fails) next step would be to downsample and send to server at 30s

Task Sample

import "influxdata/influxdb/tasks"
option task = {
    name: "xxclient_1s",
    every: 30h,
}

dataset = from(bucket: "xxclient")
    |> range(start: tasks.lastSuccess(orTime: -2d))
    |> filter(fn: (r) => r["_measurement"] == "Electronic Engine Controller 1" or r["_measurement"] == "COG & SOG, Rapid Update" or r["_measurement"] == "Position, Rapid Update" or r["_measurement"] == "Water Depth" or r["_measurement"] == "Temperature" or r["_measurement"] == "Fuel Consumption (Liquid) 1" or r["_measurement"] == "Fuel Economy (Liquid)" or r["_measurement"] == "Intake/Exhaust Conditions 1" or r["_measurement"] == "Engine Temperature 1" or r["_measurement"] == "Engine Hours, Revolutions" or r["_measurement"] == "Electronic Engine Controller 2" or r["_measurement"] == "Tachograph")
    |> filter(fn: (r) => r["_field"] != "Engine Starter Mode")

dataset
    |> aggregateWindow(every: 1s, fn: mean)
    |> to(
        org: "myOrg",
        bucket: "tiqclient_1s",
    )

Topic		Replies	Views
Infuxdb V2 startup fails - suspected data corruption InfluxDB 2	4	1091	March 24, 2022
My influxdb is not able to start anymore :-/ InfluxDB 1 influxdb	4	8105	September 19, 2021
Run: open server: listen: listen tcp 127.0.0.1:8088: bind: address already in use Store influxdb , outputs	28	27782	February 21, 2023
dial tcp [::1]:8086: connect: connection refused ... sometimes	5	7333	October 11, 2021
InfluxDB 1.7.4 fails after 9 months without issues	13	8666	May 9, 2022

RPi influxdb-v2 Losing access - connection refused

Related topics