Hello! Im having an issue writing into influxdb 1.2.4
Scenario> Arduino sending via HTTP POST to Influx. It works OK for an hour aprox and then stops writing.
If i try a select on the influx it works ok, post to other databases works too.
On logs i cant see any error, i just cant connect.
I try sending the data slower (from 200 ms to 400 ms) and the result was that the problem was not solved, but it takes almost the double time to show up.
Ive tested changing the default retention policy, shard duration, cache-max-memory-size of TSM, cache-snapshot-memory-size and enabled the shard precreation, but no luck
Influx is running on raspbian (Rpi3), and im sending only 4 values every 2 seconds. The Load Av, Disk usage, etc its fine.
Ideas?
@AgustinM This sounds most likely to be a network issue. What driver are you using to send data to the database? Are there retries in your script? Can you share more information about both of those topics?
Hi!, im using Ethernet.h and y retry the connection permanently
I found new info, it seams that the error is when influxdb compacts cache.
LOG:
Jun 14 12:30:59 raspberrypi influxd[711]: [httpd] 172.16.100.147 - - [14/Jun/2017:12:30:59 -0300] “POST /write?db=arduino HTTP/1.1” 204 0 “-” “Arduino/1.0” 7
Jun 14 12:31:00 raspberrypi influxd[711]: [httpd] 172.16.100.147 - - [14/Jun/2017:12:31:00 -0300] “POST /write?db=arduino HTTP/1.1” 204 0 “-” “Arduino/1.0” 7
Jun 14 12:31:01 raspberrypi influxd[711]: [I] 2017-06-14T15:31:01Z SELECT mean(temperatura) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND ti
Jun 14 12:31:01 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:31:01 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22temperatura%22
Jun 14 12:31:08 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:31:08 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 7cf20
Jun 14 12:31:18 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:31:18 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 82e7e
Jun 14 12:31:28 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:31:28 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 88ddc
Jun 14 12:31:30 raspberrypi influxd[711]: [I] 2017-06-14T15:31:30Z SELECT mean(humedad) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND time >
Jun 14 12:31:30 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:31:30 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22humedad%22%29+
Jun 14 12:31:38 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:31:38 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 8ed3b
Jun 14 12:31:48 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:31:48 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 94c99
Jun 14 12:31:58 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:31:58 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 9abf7
Jun 14 12:31:58 raspberrypi influxd[711]: [I] 2017-06-14T15:31:58Z Compacting cache for /var/lib/influxdb/data/telegraf/autogen/46 engine=tsm1
Jun 14 12:31:58 raspberrypi influxd[711]: [I] 2017-06-14T15:31:58Z Snapshot for path /var/lib/influxdb/data/telegraf/autogen/46 deduplicated in 1.8344ms engi
Jun 14 12:31:58 raspberrypi influxd[711]: [I] 2017-06-14T15:31:58Z Removing /var/lib/influxdb/wal/telegraf/autogen/46/_00190.wal engine=tsm1 service=wal
Jun 14 12:31:58 raspberrypi influxd[711]: [I] 2017-06-14T15:31:58Z Snapshot for path /var/lib/influxdb/data/telegraf/autogen/46 written in 40.680493ms engine
Jun 14 12:31:59 raspberrypi influxd[711]: [I] 2017-06-14T15:31:59Z beginning level 1 compaction of group 0, 2 TSM files engine=tsm1
Jun 14 12:31:59 raspberrypi influxd[711]: [I] 2017-06-14T15:31:59Z compacting level 1 group (0) /var/lib/influxdb/data/telegraf/autogen/46/000000184-00000000
Jun 14 12:31:59 raspberrypi influxd[711]: [I] 2017-06-14T15:31:59Z compacting level 1 group (0) /var/lib/influxdb/data/telegraf/autogen/46/000000185-00000000
Jun 14 12:31:59 raspberrypi influxd[711]: [I] 2017-06-14T15:31:59Z compacted level 1 group (0) into /var/lib/influxdb/data/telegraf/autogen/46/000000185-0000
Jun 14 12:31:59 raspberrypi influxd[711]: [I] 2017-06-14T15:31:59Z compacted level 1 2 files into 1 files in 102.929915ms engine=tsm1
Jun 14 12:32:01 raspberrypi influxd[711]: [I] 2017-06-14T15:32:01Z SELECT mean(temperatura) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND ti
Jun 14 12:32:01 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:32:01 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22temperatura%22
Jun 14 12:32:08 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:32:08 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” a0b5b
Jun 14 12:32:18 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:32:18 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” a6ab7
Jun 14 12:32:28 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:32:28 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” aca1c
Jun 14 12:32:30 raspberrypi influxd[711]: [I] 2017-06-14T15:32:30Z SELECT mean(humedad) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND time >
Jun 14 12:32:30 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:32:30 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22humedad%22%29+
Jun 14 12:32:38 raspberrypi influxd[711]: [I] 2017-06-14T15:32:38Z Compacting cache for /var/lib/influxdb/data/_internal/monitor/52 engine=tsm1
Jun 14 12:32:38 raspberrypi influxd[711]: [I] 2017-06-14T15:32:38Z Snapshot for path /var/lib/influxdb/data/_internal/monitor/52 deduplicated in 3.32566ms en
Jun 14 12:32:38 raspberrypi influxd[711]: [I] 2017-06-14T15:32:38Z Removing /var/lib/influxdb/wal/_internal/monitor/52/_00282.wal engine=tsm1 service=wal
Jun 14 12:32:38 raspberrypi influxd[711]: [I] 2017-06-14T15:32:38Z Snapshot for path /var/lib/influxdb/data/_internal/monitor/52 written in 71.177261ms engin
Jun 14 12:32:38 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:32:38 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” b296f
Jun 14 12:32:39 raspberrypi influxd[711]: [I] 2017-06-14T15:32:39Z beginning level 1 compaction of group 0, 2 TSM files engine=tsm1
Jun 14 12:32:39 raspberrypi influxd[711]: [I] 2017-06-14T15:32:39Z compacting level 1 group (0) /var/lib/influxdb/data/_internal/monitor/52/000000281-0000000
Jun 14 12:32:39 raspberrypi influxd[711]: [I] 2017-06-14T15:32:39Z compacting level 1 group (0) /var/lib/influxdb/data/_internal/monitor/52/000000282-0000000
Jun 14 12:32:39 raspberrypi influxd[711]: [I] 2017-06-14T15:32:39Z compacted level 1 group (0) into /var/lib/influxdb/data/_internal/monitor/52/000000282-000
Jun 14 12:32:39 raspberrypi influxd[711]: [I] 2017-06-14T15:32:39Z compacted level 1 2 files into 1 files in 186.265598ms engine=tsm1
Jun 14 12:32:40 raspberrypi influxd[711]: [I] 2017-06-14T15:32:40Z beginning level 2 compaction of group 0, 2 TSM files engine=tsm1
Jun 14 12:32:40 raspberrypi influxd[711]: [I] 2017-06-14T15:32:40Z compacting level 2 group (0) /var/lib/influxdb/data/_internal/monitor/52/000000280-0000000
Jun 14 12:32:40 raspberrypi influxd[711]: [I] 2017-06-14T15:32:40Z compacting level 2 group (0) /var/lib/influxdb/data/_internal/monitor/52/000000282-0000000
Jun 14 12:32:40 raspberrypi influxd[711]: [I] 2017-06-14T15:32:40Z compacted level 2 group (0) into /var/lib/influxdb/data/_internal/monitor/52/000000282-000
Jun 14 12:32:40 raspberrypi influxd[711]: [I] 2017-06-14T15:32:40Z compacted level 2 2 files into 1 files in 230.483076ms engine=tsm1
Jun 14 12:32:48 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:32:48 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” b88ce
Jun 14 12:32:58 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:32:58 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” be82b
Jun 14 12:33:01 raspberrypi influxd[711]: [I] 2017-06-14T15:33:01Z SELECT mean(temperatura) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND ti
Jun 14 12:33:01 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:33:01 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22temperatura%22
Jun 14 12:33:08 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:33:08 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” c4789
Jun 14 12:33:17 raspberrypi influxd[711]: [I] 2017-06-14T15:33:17Z retention policy shard deletion check commencing service=retention
Jun 14 12:33:18 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:33:18 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” ca6e7
Jun 14 12:33:28 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:33:28 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” d064c
Jun 14 12:33:30 raspberrypi influxd[711]: [I] 2017-06-14T15:33:30Z SELECT mean(humedad) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND time >
Jun 14 12:33:30 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:33:30 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22humedad%22%29+
Jun 14 12:33:38 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:33:38 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” d65a4
Jun 14 12:33:48 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:33:48 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” dc502
Jun 14 12:33:58 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:33:58 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” e245f
Jun 14 12:34:01 raspberrypi influxd[711]: [I] 2017-06-14T15:34:01Z SELECT mean(temperatura) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND ti
Jun 14 12:34:01 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:34:01 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22temperatura%22
Jun 14 12:34:08 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:34:08 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” e83be
Jun 14 12:34:18 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:34:18 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” ee31b
Jun 14 12:34:28 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:34:28 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” f427a
Jun 14 12:34:30 raspberrypi influxd[711]: [I] 2017-06-14T15:34:30Z SELECT mean(humedad) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND time >
Jun 14 12:34:30 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:34:30 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22humedad%22%29+
Jun 14 12:34:38 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:34:38 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” fa1e9
Jun 14 12:34:48 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:34:48 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 00136
Jun 14 12:34:58 raspberrypi influxd[711]: [httpd] ::1 - - [14/Jun/2017:12:34:58 -0300] “POST /write?consistency=any&db=telegraf HTTP/1.1” 204 0 “-” “-” 06093
Jun 14 12:35:01 raspberrypi influxd[711]: [I] 2017-06-14T15:35:01Z SELECT mean(temperatura) FROM arduino.autogen.my_house_data WHERE SOURCE = ‘Living’ AND ti
Jun 14 12:35:01 raspberrypi influxd[711]: [httpd] ::1 - arduino [14/Jun/2017:12:35:01 -0300] "GET /query?db=arduino&epoch=s&q=SELECT+mean%28%22temperatura%22
First 2 lines shows the post. After compacting, never again
Thanks
@AgustinM That log would indicate to me that the database stopped recieving POST
requests from your client. Do you have any logs from the Arduino client? Telegraf continues to write successfully so I’m certain that this isn’t a database issue.
Thank you! As you say, its from the arduino the error, maybe the ethernet lib, im trying to find it.
Many thanks