InfluxDB 2.3 on Debian 9.x has suddenly given up

Message:

Aug 19 11:33:53 xili influxd-systemd-start.sh[610]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts…

The 2 things i’ve done:

  • changed the name/value pair in /etc/influxdb/config.toml file: ‘http-bind-address = “:8086”’ to "http-bind-address = “MY_IP:8086"”, ""http-bind-address = “localhost:8086"”, http-bind-address = “127.0.0.1:8086” and http-bind-address = “0.0.0.0:8086” → BUT THEN SET BACK to default value ‘http-bind-address = “:8086”’ !!!

  • Set some firewall rules:
    iptables -A INPUT -p tcp -s xxx/24 --dport 8086 -j ACCEPT
    iptables -A INPUT -p tcp -s yyy/32 --dport 8086 -j ACCEPT
    iptables -A INPUT -p tcp --dport 8086 -j DROP

  • Installed “iptables-persistent”

…since then, influxdb dose not work anymore. The only one file i’ve touched is /etc/influxdb/config.toml - this is 100% true, no way!!!

Thank you for your feedbacks.

Best regards, Hans

Please give more detail regarding “…since then, influxdb dose not work
anymore.”

Do you mean it does not start, is not available even on the machine itself, or
simply cannot be accessed from elsewhere on the network?

Please tell us how you are testing “does it work?” and give us details on what
the results of that testing are (eg: extracts from the log file when starting
it, if it fails to start correctly, or network packet captures if it works on
the local machine but is not reachable from elsewhere, etc).

Thanks,

Antony.

Always message like that with the service status command:

Aug 19 11:33:53 xili influxd-systemd-start.sh[610]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts…

Seems to be FULL of BUGS, version 2.3 on Debian 9.x: Influxdb daemon still runs after purge ALL influxdb things

…does not work anymore even after a complete cleanup procedure then reinstall via “apt install influxdb2 influxdb2-cli”:

Linux xili 4.19.0-0.bpo.9-amd64 #1 SMP Debian 4.19.118-2+deb10u1~bpo9+1 (2020-06-09) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Aug 19 12:17:16 2022 from 51.154.150.136
root@xili:~# service influxdb status
● influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: activating (start) since Fri 2022-08-19 12:44:42 CEST; 8s ago
     Docs: https://docs.influxdata.com/influxdb/
Cntrl PID: 904 (influxd-systemd)
    Tasks: 9 (limit: 9830)
   CGroup: /system.slice/influxdb.service
           ├─904 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh
           ├─905 /usr/bin/influxd
           └─930 curl -k -s -o /dev/null http://localhost:8086/ready -w %{http_code}

Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.741529Z lvl=info msg="Compaction settings" log_id=0cPR7dOW000 service=storage-engine service=store max_concurrent_compactions=1 throughp
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.741907Z lvl=info msg="Open store (start)" log_id=0cPR7dOW000 service=storage-engine service=store op_name=tsdb_open op_event=start
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.742415Z lvl=info msg="Open store (end)" log_id=0cPR7dOW000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapse
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.742803Z lvl=info msg="Starting retention policy enforcement service" log_id=0cPR7dOW000 service=retention check_interval=30m
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.743178Z lvl=info msg="Starting precreation service" log_id=0cPR7dOW000 service=shard-precreation check_interval=10m advance_period=30m
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.744463Z lvl=info msg="Starting query controller" log_id=0cPR7dOW000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_qu
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.750269Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0cPR7dOW000 max_select_point=0 max_sel
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.761854Z lvl=info msg=Listening log_id=0cPR7dOW000 service=tcp-listener transport=http addr=:8086 port=8086
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: ts=2022-08-19T10:44:42.763364Z lvl=info msg=Starting log_id=0cPR7dOW000 service=telemetry interval=8h
Aug 19 12:44:42 xili influxd-systemd-start.sh[904]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
lines 1-21/21 (END)


“even after a complete cleanup procedure”

As described (shell script) here:

After ~1 minute, that “unavailable” erros occurs - then it repeats in and (endless?) loop:

root@xili:~# service influxdb status
● influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: activating (start) since Fri 2022-08-19 12:46:12 CEST; 1min 2s ago
     Docs: https://docs.influxdata.com/influxdb/
Cntrl PID: 954 (influxd-systemd)
    Tasks: 9 (limit: 9830)
   CGroup: /system.slice/influxdb.service
           ├─954 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh
           ├─955 /usr/bin/influxd
           └─987 curl -k -s -o /dev/null http://localhost:8086/ready -w %{http_code}

Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: ts=2022-08-19T10:46:13.237038Z lvl=info msg="Open store (end)" log_id=0cPRD9uW000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapse
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: ts=2022-08-19T10:46:13.239288Z lvl=info msg="Starting retention policy enforcement service" log_id=0cPRD9uW000 service=retention check_interval=30m
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: ts=2022-08-19T10:46:13.239720Z lvl=info msg="Starting precreation service" log_id=0cPRD9uW000 service=shard-precreation check_interval=10m advance_period=30m
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: ts=2022-08-19T10:46:13.242078Z lvl=info msg="Starting query controller" log_id=0cPRD9uW000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_qu
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: ts=2022-08-19T10:46:13.247855Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0cPRD9uW000 max_select_point=0 max_sel
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: ts=2022-08-19T10:46:13.262843Z lvl=info msg=Listening log_id=0cPRD9uW000 service=tcp-listener transport=http addr=:8086 port=8086
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: ts=2022-08-19T10:46:13.263289Z lvl=info msg=Starting log_id=0cPRD9uW000 service=telemetry interval=8h
Aug 19 12:46:13 xili influxd-systemd-start.sh[954]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Aug 19 12:46:44 xili influxd-systemd-start.sh[954]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
lines 1-21/21 (END)

More “unavilable” entries after a few minutes:

root@xili:~# service influxdb status
● influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: activating (start) since Fri 2022-08-19 12:55:15 CEST; 1min 26s ago
     Docs: https://docs.influxdata.com/influxdb/
Cntrl PID: 1242 (influxd-systemd)
    Tasks: 9 (limit: 9830)
   CGroup: /system.slice/influxdb.service
           ├─1242 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh
           ├─1243 /usr/bin/influxd
           └─1293 curl -k -s -o /dev/null http://localhost:8086/ready -w %{http_code}

Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: ts=2022-08-19T10:55:16.242070Z lvl=info msg="Starting retention policy enforcement service" log_id=0cPRjI~G000 service=retention check_interval=30m
Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: ts=2022-08-19T10:55:16.242456Z lvl=info msg="Starting precreation service" log_id=0cPRjI~G000 service=shard-precreation check_interval=10m advance_period=30m
Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: ts=2022-08-19T10:55:16.244753Z lvl=info msg="Starting query controller" log_id=0cPRjI~G000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_q
Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: ts=2022-08-19T10:55:16.251536Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0cPRjI~G000 max_select_point=0 max_se
Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: ts=2022-08-19T10:55:16.263719Z lvl=info msg=Starting log_id=0cPRjI~G000 service=telemetry interval=8h
Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: ts=2022-08-19T10:55:16.265375Z lvl=info msg=Listening log_id=0cPRjI~G000 service=tcp-listener transport=http addr=:8086 port=8086
Aug 19 12:55:16 xili influxd-systemd-start.sh[1242]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Aug 19 12:55:47 xili influxd-systemd-start.sh[1242]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Aug 19 12:56:20 xili influxd-systemd-start.sh[1242]: InfluxDB API at http://localhost:8086/ready unavailable after 2 attempts...

Maybe it has sometinh to do with the iptables rules ive set…?

iptables -A INPUT -p tcp -s xxx/24 --dport 8086 -j ACCEPT
iptables -A INPUT -p tcp -s yyy/32 --dport 8086 -j ACCEPT
iptables -A INPUT -p tcp --dport 8086 -j DROP

root@xili:~# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s xxx/24 -p tcp -m tcp --dport 8086 -j ACCEPT
-A INPUT -s yyy/32 -p tcp -m tcp --dport 8086 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8086 -j DROP
root@xili:~#

SELF-CITE: "seems to be full of bugs … "

One more to go!! :wink:
→ One problem (also a bug?) more: 403 Forbidden: User is inactive when create user - #5 by ugvsffzawv

FW rules were the problem:

iptables -A INPUT -p tcp -s xxx/24 --dport 8086 -j ACCEPT
iptables -A INPUT -p tcp -s yyy/32 --dport 8086 -j ACCEPT
iptables -A INPUT -p tcp --dport 8086 -j DROP

apt install iptables-persistant
init 6

After configuring fw rules and make them persistant and then restart via “init 6”:

influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: activating (start) since Fri 2022-08-19 13:26:16 CEST; 17s ago
     Docs: https://docs.influxdata.com/influxdb/
Cntrl PID: 608 (influxd-systemd)
    Tasks: 9 (limit: 9830)
   CGroup: /system.slice/influxdb.service
           ├─608 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh
           ├─644 /usr/bin/influxd
           └─672 curl -k -s -o /dev/null http://localhost:8086/ready -w %{http_code}

Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.623371Z lvl=info msg="Compaction settings" log_id=0cPTVuwG000 service=storage-engine service=store max_concur
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.623382Z lvl=info msg="Open store (start)" log_id=0cPTVuwG000 service=storage-engine service=store op_name=tsd
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.624506Z lvl=info msg="Open store (end)" log_id=0cPTVuwG000 service=storage-engine service=store op_name=tsdb_
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.625523Z lvl=info msg="Starting retention policy enforcement service" log_id=0cPTVuwG000 service=retention che
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.625541Z lvl=info msg="Starting precreation service" log_id=0cPTVuwG000 service=shard-precreation check_interv
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.630003Z lvl=info msg="Starting query controller" log_id=0cPTVuwG000 service=storage-reads concurrency_quota=1
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.634264Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0cPTVuwG000
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.652960Z lvl=info msg=Listening log_id=0cPTVuwG000 service=tcp-listener transport=http addr=:8086 port=8086
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: ts=2022-08-19T11:26:17.653513Z lvl=info msg=Starting log_id=0cPTVuwG000 service=telemetry interval=8h
Aug 19 13:26:17 xili influxd-systemd-start.sh[608]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running
Aug 19 13:28:22 xili influxd-systemd-start.sh[875]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
lines 1-21/21 (END)


But iptables -A INPUT -p tcp -s 127.0.0.1/32 --dport 8086 -j ACCEPT does also not help… i’m confused. Really.


But iptables -A INPUT -p tcp -s 127.0.0.1/32 --dport 8086 -j ACCEPT does also not help… i’m confused. Really.

OK… we have the luck to have

1.) public ip (directy to public 24-CIDR subnet with static IP → “leased line”)

2.) a private IP 192.168.71.20 (backup connection [also ingoing ports like the leased lin]) via NAT

So now i used "http-bind-address = “192.168.71.20:8086"” in the config.toml file.

But now, “telnet localhost 8086” and “telnet 127.0.0.1 8086” does not work anymore.

Questions:

  • How to bind TCP daemon to multiple, BUT NOT (!!) all (NOT 0.0.0.0), adpters via config.toml file?
    I need that ones: 127.0.0.1, localhost, 192.168.71.20

Or for another “strategy” - whitelisting to the public leased line ip - is it possible to whitelist IPs via InfluxDB config? (As same as i have done via iptables - but iptables will break the wohle c**p as described above…)

Or is the only one way to reach my to to use a NAT router betweem the public IP (leased line) and the Hyper-V-Bridge!?

CONCRETE: A hardware solution to bypass InfluxDB2 problems…? (I need to ask myself: Am i in the wrong film or is it just a bad dream. And no, i didn’t take some LSD… LOL…)

OK - so now i applied that iptables config:

iptables -A INPUT -i eth0 -p tcp --destination-port 8086 -j DROP

…which blocks all port 8086 access to the public, leased line ip. But it does not block that the adapter on that nat router (without puiblic internet forward to port 8086)…

So now i can connect via WLAN to the 192.168.71.0/24 subnet and using Windows InfluxDB frontends for the whole crap…

Maybe somebody knows how to replace the adapter eth0 with ip that is on eth0 (nic layer vs. ip address layer) on that iptables command :question: :question: :question:

iptables -A INPUT -i eth0 -p tcp --destination-port 8086 -j DROP

…i woud prefer to define: block all connections from all incoming source ip addresse on destination port 8086 on destination ip x.x.x.

IS THAT POSSIBLE?

No anwser? Hello…!?

??? → (Body is too short (minimum is 20 characters)…)

I think this is happening because of the print-config command in the influxd-systemd-start.sh script.

It was deprecated in 2.2 and removed in 2.3. Running the commands manually in that file succeed in the process starting like influxd --bolt-path=/var/lib/influxdb/influxd.bolt --engine-path=/var/lib/influxdb/engine.

But once the script gets to BIND_ADDRESS=$(influxd print-config --key-name http-bind-address)

Thing’s start failing.

Thank you very much for your feedback! :slight_smile: