Hi,
We were seeing “500 20” error for every write request to influxDB
[httpd] 10.44.109.97 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 16552e02-7fdd-11ea-af47-000000000000 10103657
ts=2020-04-16T12:23:42.843934Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.109.97 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 1658b178-7fdd-11ea-af4e-000000000000 10080766
ts=2020-04-16T12:23:42.844063Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.109.97 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 16550859-7fdd-11ea-af46-000000000000 10105021
ts=2020-04-16T12:23:42.844341Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.98.161 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 163208b1-7fdd-11ea-af02-000000000000 10334594
ts=2020-04-16T12:23:42.844535Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.98.161 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 16581ef4-7fdd-11ea-af4c-000000000000 10085701
ts=2020-04-16T12:23:42.845259Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.109.97 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 1650cc93-7fdd-11ea-af3c-000000000000 10135467
ts=2020-04-16T12:23:42.847036Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.98.161 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 165b357d-7fdd-11ea-af52-000000000000 10067353
ts=2020-04-16T12:23:42.847142Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.98.161 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 164bb0f2-7fdd-11ea-af32-000000000000 10169136
ts=2020-04-16T12:23:42.847246Z lvl=error msg="[500] - \"timeout\"" log_id=0MCgUHol000 service=httpd
[httpd] 10.44.98.161 - telegraf_v1 [16/Apr/2020:12:23:32 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 165ed62a-7fdd-11ea-af5e-000000000000 10050286
Things were fine but we saw few metrics were getting dropped because of this error. So we did the restart of influxDB and after that all requests were getting this error.
We tried restarting influxDB multiple time but still same issue.
Following are the details of our system
Version: 1.6.1
We are using influx-Relay to write data to 2 influxDB host at same time in HA.
As a workaround, we moved the influxDB to new host and after that things were back to normal.
Later we moved back influxDB to same hosts and it started working fine.
Please suggest if anyone has encountered this issue.
Regards,
Advait Deo
