Keep getting 500 on prometheus remote write

I am running prometheus

prometheus --version
prometheus, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
build user: root@dd72efe1549d
build date: 20200626-09:02:20
go version: go1.14.4

with Influxdb
influxd version
InfluxDB v1.8.0 (git: 1.8 781490de48220d7695a05c29e5a36f550a4568f5)

I am using remote prometheus
remote_write:

I am getting lot of 500 error occasionally I get 204

Jan 16 22:33:28 eu-monitordb1 influxd: [httpd] 10.5.5.244 - influxdb [16/Jan/2021:22:33:13 +0000] "POST /api/v1/prom/write?db=prometheus HTTP/1.1" 500 20 "-" "Prometheus/2.19.2" d1a20427-584a-11eb-a384-0aafdd456fa2 15055553

Jan 16 22:33:28 eu-monitordb1 influxd: ts=2021-01-16T22:33:28.336797Z lvl=error msg="[500] - “timeout”" log_id=0RkKJ8_G000 service=httpd
Jan 16 22:33:28 eu-monitordb1 influxd: [httpd] 10.5.5.244 - influxdb [16/Jan/2021:22:33:13 +0000] “POST /api/v1/prom/write?db=prometheus HTTP/1.1” 500 20 “-” “Prometheus/2.19.2” d1a23f58-584a-11eb-a385-0aafdd456fa2 15071190
Jan 16 22:33:28 eu-monitordb1 influxd: ts=2021-01-16T22:33:28.353972Z lvl=error msg="[500] - “timeout”" log_id=0RkKJ8_G000 service=httpd
Jan 16 22:33:28 eu-monitordb1 influxd: [httpd] 10.5.5.244 - influxdb [16/Jan/2021:22:33:13 +0000] “POST /api/v1/prom/write?db=prometheus HTTP/1.1” 500 20 “-” “Prometheus/2.19.2” d19529f0-584a-11eb-a37d-0aafdd456fa2 15214901
Jan 16 22:33:28 eu-monitordb1 influxd: ts=2021-01-16T22:33:28.411938Z lvl=error msg="[500] - “timeout”" log_id=0RkKJ8_G000 service=httpd
Jan 16 22:33:28 eu-monitordb1 influxd: [httpd] 10.5.5.244 - influxdb [16/Jan/2021:22:33:21 +0000] “POST /api/v1/prom/write?db=prometheus HTTP/1.1” 204 0 “-” “Prometheus/2.19.2” d69cb367-584a-11eb-a5f9-0aafdd456fa2 6784903
Jan 16 22:33:28 eu-monitordb1 influxd: [httpd] 10.5.5.244 - influxdb [16/Jan/2021:22:33:21 +0000] “POST /api/v1/prom/write?db=prometheus HTTP/1.1” 204 0 “-” “Prometheus/2.19.2” d6907416-584a-11eb-a5c0-0aafdd456fa2 6869115

Tried different setting in influxdb.conf setting but dont see any difference, any ideas where to start?

Hello @vipin ,
Welcome! Unfortunately, I don’t know the answer. I appreciate your patience while I try to find someone who can help.
Thanks :slight_smile: