E! [outputs.influxdb] Failed to write metric (will be dropped: 400 Bad Request): partial write: points beyond retention policy dropped=1

InfluxDB 1
1

Looking for some assistance with the following error in my logs:

E! [outputs.influxdb] Failed to write metric (will be dropped: 400 Bad Request): partial write: points beyond retention policy dropped=1

This is happening every few minutes. My lack of knowledge with influx makes troubleshooting a like more challenging for me unfortunately. I did some research however, and read a few things about shards overlapping (but don’t know what that means or how to look for this issue).

I can provide with some information about my databases:

InfluxDB shell version: 1.8.10
> show shards
name: _internal
id    database  retention_policy shard_group start_time           end_time             expiry_time          owners
--    --------  ---------------- ----------- ----------           --------             -----------          ------
23814 _internal monitor          23814       2023-01-09T00:00:00Z 2023-01-10T00:00:00Z 2023-01-24T00:00:00Z 
23848 _internal monitor          23848       2023-01-10T00:00:00Z 2023-01-11T00:00:00Z 2023-01-25T00:00:00Z 
23877 _internal monitor          23877       2023-01-11T00:00:00Z 2023-01-12T00:00:00Z 2023-01-26T00:00:00Z 
23906 _internal monitor          23906       2023-01-12T00:00:00Z 2023-01-13T00:00:00Z 2023-01-27T00:00:00Z 
23935 _internal monitor          23935       2023-01-13T00:00:00Z 2023-01-14T00:00:00Z 2023-01-28T00:00:00Z 
23966 _internal monitor          23966       2023-01-14T00:00:00Z 2023-01-15T00:00:00Z 2023-01-29T00:00:00Z 
23972 _internal monitor          23972       2023-01-15T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23978 _internal monitor          23978       2023-01-16T00:00:00Z 2023-01-17T00:00:00Z 2023-01-31T00:00:00Z 

name: telegraf
id    database retention_policy shard_group start_time           end_time             expiry_time          owners
--    -------- ---------------- ----------- ----------           --------             -----------          ------
23607 telegraf autogen          23607       2023-01-02T00:00:00Z 2023-01-03T00:00:00Z 2023-01-17T00:00:00Z 
23641 telegraf autogen          23641       2023-01-03T00:00:00Z 2023-01-04T00:00:00Z 2023-01-18T00:00:00Z 
23670 telegraf autogen          23670       2023-01-04T00:00:00Z 2023-01-05T00:00:00Z 2023-01-19T00:00:00Z 
23699 telegraf autogen          23699       2023-01-05T00:00:00Z 2023-01-06T00:00:00Z 2023-01-20T00:00:00Z 
23728 telegraf autogen          23728       2023-01-06T00:00:00Z 2023-01-07T00:00:00Z 2023-01-21T00:00:00Z 
23757 telegraf autogen          23757       2023-01-07T00:00:00Z 2023-01-08T00:00:00Z 2023-01-22T00:00:00Z 
23786 telegraf autogen          23786       2023-01-08T00:00:00Z 2023-01-09T00:00:00Z 2023-01-23T00:00:00Z 
23815 telegraf autogen          23815       2023-01-09T00:00:00Z 2023-01-10T00:00:00Z 2023-01-24T00:00:00Z 
23849 telegraf autogen          23849       2023-01-10T00:00:00Z 2023-01-11T00:00:00Z 2023-01-25T00:00:00Z 
23878 telegraf autogen          23878       2023-01-11T00:00:00Z 2023-01-12T00:00:00Z 2023-01-26T00:00:00Z 
23907 telegraf autogen          23907       2023-01-12T00:00:00Z 2023-01-13T00:00:00Z 2023-01-27T00:00:00Z 
23936 telegraf autogen          23936       2023-01-13T00:00:00Z 2023-01-14T00:00:00Z 2023-01-28T00:00:00Z 
23967 telegraf autogen          23967       2023-01-14T00:00:00Z 2023-01-15T00:00:00Z 2023-01-29T00:00:00Z 
23973 telegraf autogen          23973       2023-01-15T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23979 telegraf autogen          23979       2023-01-16T00:00:00Z 2023-01-17T00:00:00Z 2023-01-31T00:00:00Z 

name: graphite
id    database retention_policy shard_group start_time           end_time             expiry_time          owners
--    -------- ---------------- ----------- ----------           --------             -----------          ------
23608 graphite autogen          23608       2023-01-02T00:00:00Z 2023-01-03T00:00:00Z 2023-01-17T00:00:00Z 
23642 graphite autogen          23642       2023-01-03T00:00:00Z 2023-01-04T00:00:00Z 2023-01-18T00:00:00Z 
23671 graphite autogen          23671       2023-01-04T00:00:00Z 2023-01-05T00:00:00Z 2023-01-19T00:00:00Z 
23700 graphite autogen          23700       2023-01-05T00:00:00Z 2023-01-06T00:00:00Z 2023-01-20T00:00:00Z 
23729 graphite autogen          23729       2023-01-06T00:00:00Z 2023-01-07T00:00:00Z 2023-01-21T00:00:00Z 
23758 graphite autogen          23758       2023-01-07T00:00:00Z 2023-01-08T00:00:00Z 2023-01-22T00:00:00Z 
23787 graphite autogen          23787       2023-01-08T00:00:00Z 2023-01-09T00:00:00Z 2023-01-23T00:00:00Z 
23816 graphite autogen          23816       2023-01-09T00:00:00Z 2023-01-10T00:00:00Z 2023-01-24T00:00:00Z 
23850 graphite autogen          23850       2023-01-10T00:00:00Z 2023-01-11T00:00:00Z 2023-01-25T00:00:00Z 
23879 graphite autogen          23879       2023-01-11T00:00:00Z 2023-01-12T00:00:00Z 2023-01-26T00:00:00Z 
23908 graphite autogen          23908       2023-01-12T00:00:00Z 2023-01-13T00:00:00Z 2023-01-27T00:00:00Z 
23937 graphite autogen          23937       2023-01-13T00:00:00Z 2023-01-14T00:00:00Z 2023-01-28T00:00:00Z 
23968 graphite autogen          23968       2023-01-14T00:00:00Z 2023-01-15T00:00:00Z 2023-01-29T00:00:00Z 
23974 graphite autogen          23974       2023-01-15T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23980 graphite autogen          23980       2023-01-16T00:00:00Z 2023-01-17T00:00:00Z 2023-01-31T00:00:00Z 

name: sabnzbd
id    database retention_policy shard_group start_time           end_time             expiry_time          owners
--    -------- ---------------- ----------- ----------           --------             -----------          ------
23609 sabnzbd  autogen          23609       2023-01-02T00:00:00Z 2023-01-03T00:00:00Z 2023-01-17T00:00:00Z 
23643 sabnzbd  autogen          23643       2023-01-03T00:00:00Z 2023-01-04T00:00:00Z 2023-01-18T00:00:00Z 
23672 sabnzbd  autogen          23672       2023-01-04T00:00:00Z 2023-01-05T00:00:00Z 2023-01-19T00:00:00Z 
23701 sabnzbd  autogen          23701       2023-01-05T00:00:00Z 2023-01-06T00:00:00Z 2023-01-20T00:00:00Z 
23730 sabnzbd  autogen          23730       2023-01-06T00:00:00Z 2023-01-07T00:00:00Z 2023-01-21T00:00:00Z 
23759 sabnzbd  autogen          23759       2023-01-07T00:00:00Z 2023-01-08T00:00:00Z 2023-01-22T00:00:00Z 
23788 sabnzbd  autogen          23788       2023-01-08T00:00:00Z 2023-01-09T00:00:00Z 2023-01-23T00:00:00Z 
23817 sabnzbd  autogen          23817       2023-01-09T00:00:00Z 2023-01-10T00:00:00Z 2023-01-24T00:00:00Z 
23851 sabnzbd  autogen          23851       2023-01-10T00:00:00Z 2023-01-11T00:00:00Z 2023-01-25T00:00:00Z 
23880 sabnzbd  autogen          23880       2023-01-11T00:00:00Z 2023-01-12T00:00:00Z 2023-01-26T00:00:00Z 
23909 sabnzbd  autogen          23909       2023-01-12T00:00:00Z 2023-01-13T00:00:00Z 2023-01-27T00:00:00Z 
23938 sabnzbd  autogen          23938       2023-01-13T00:00:00Z 2023-01-14T00:00:00Z 2023-01-28T00:00:00Z 
23969 sabnzbd  autogen          23969       2023-01-14T00:00:00Z 2023-01-15T00:00:00Z 2023-01-29T00:00:00Z 
23975 sabnzbd  autogen          23975       2023-01-15T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23981 sabnzbd  autogen          23981       2023-01-16T00:00:00Z 2023-01-17T00:00:00Z 2023-01-31T00:00:00Z 

name: unifi
id    database retention_policy shard_group start_time           end_time             expiry_time          owners
--    -------- ---------------- ----------- ----------           --------             -----------          ------
23610 unifi    autogen          23610       2023-01-02T00:00:00Z 2023-01-03T00:00:00Z 2023-01-17T00:00:00Z 
23644 unifi    autogen          23644       2023-01-03T00:00:00Z 2023-01-04T00:00:00Z 2023-01-18T00:00:00Z 
23673 unifi    autogen          23673       2023-01-04T00:00:00Z 2023-01-05T00:00:00Z 2023-01-19T00:00:00Z 
23702 unifi    autogen          23702       2023-01-05T00:00:00Z 2023-01-06T00:00:00Z 2023-01-20T00:00:00Z 
23731 unifi    autogen          23731       2023-01-06T00:00:00Z 2023-01-07T00:00:00Z 2023-01-21T00:00:00Z 
23760 unifi    autogen          23760       2023-01-07T00:00:00Z 2023-01-08T00:00:00Z 2023-01-22T00:00:00Z 
23789 unifi    autogen          23789       2023-01-08T00:00:00Z 2023-01-09T00:00:00Z 2023-01-23T00:00:00Z 
23818 unifi    autogen          23818       2023-01-09T00:00:00Z 2023-01-10T00:00:00Z 2023-01-24T00:00:00Z 
23852 unifi    autogen          23852       2023-01-10T00:00:00Z 2023-01-11T00:00:00Z 2023-01-25T00:00:00Z 
23881 unifi    autogen          23881       2023-01-11T00:00:00Z 2023-01-12T00:00:00Z 2023-01-26T00:00:00Z 
23910 unifi    autogen          23910       2023-01-12T00:00:00Z 2023-01-13T00:00:00Z 2023-01-27T00:00:00Z 
23939 unifi    autogen          23939       2023-01-13T00:00:00Z 2023-01-14T00:00:00Z 2023-01-28T00:00:00Z 
23970 unifi    autogen          23970       2023-01-14T00:00:00Z 2023-01-15T00:00:00Z 2023-01-29T00:00:00Z 
23976 unifi    autogen          23976       2023-01-15T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23982 unifi    autogen          23982       2023-01-16T00:00:00Z 2023-01-17T00:00:00Z 2023-01-31T00:00:00Z 

name: octoprint
id database retention_policy shard_group start_time end_time expiry_time owners
-- -------- ---------------- ----------- ---------- -------- ----------- ------

name: vsphere
id    database retention_policy shard_group start_time           end_time             expiry_time          owners
--    -------- ---------------- ----------- ----------           --------             -----------          ------
23820 vsphere  autogen          23820       2023-01-09T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23983 vsphere  autogen          23983       2023-01-16T00:00:00Z 2023-01-23T00:00:00Z 2023-02-06T00:00:00Z 

name: cloudflare
id    database   retention_policy shard_group start_time           end_time             expiry_time          owners
--    --------   ---------------- ----------- ----------           --------             -----------          ------
23821 cloudflare autogen          23821       2023-01-09T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23984 cloudflare autogen          23984       2023-01-16T00:00:00Z 2023-01-23T00:00:00Z 2023-02-06T00:00:00Z 

name: homeassistant
id    database      retention_policy shard_group start_time           end_time             expiry_time          owners
--    --------      ---------------- ----------- ----------           --------             -----------          ------
23614 homeassistant autogen          23614       2023-01-02T00:00:00Z 2023-01-09T00:00:00Z 2023-01-23T00:00:00Z 
23822 homeassistant autogen          23822       2023-01-09T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23985 homeassistant autogen          23985       2023-01-16T00:00:00Z 2023-01-23T00:00:00Z 2023-02-06T00:00:00Z 

name: speedtest
id    database  retention_policy shard_group start_time           end_time             expiry_time          owners
--    --------  ---------------- ----------- ----------           --------             -----------          ------
23615 speedtest autogen          23615       2023-01-02T00:00:00Z 2023-01-09T00:00:00Z 2023-01-23T00:00:00Z 
23823 speedtest autogen          23823       2023-01-09T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23986 speedtest autogen          23986       2023-01-16T00:00:00Z 2023-01-23T00:00:00Z 2023-02-06T00:00:00Z 

name: nvidia
id    database retention_policy shard_group start_time           end_time             expiry_time          owners
--    -------- ---------------- ----------- ----------           --------             -----------          ------
23616 nvidia   autogen          23616       2023-01-02T00:00:00Z 2023-01-09T00:00:00Z 2023-01-23T00:00:00Z 
23824 nvidia   autogen          23824       2023-01-09T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23987 nvidia   autogen          23987       2023-01-16T00:00:00Z 2023-01-23T00:00:00Z 2023-02-06T00:00:00Z 

name: varken
id    database retention_policy shard_group start_time           end_time             expiry_time          owners
--    -------- ---------------- ----------- ----------           --------             -----------          ------
23965 varken   autogen          23965       2023-01-13T00:00:00Z 2023-01-14T00:00:00Z 2023-01-28T00:00:00Z 
23971 varken   autogen          23971       2023-01-14T00:00:00Z 2023-01-15T00:00:00Z 2023-01-29T00:00:00Z 
23977 varken   autogen          23977       2023-01-15T00:00:00Z 2023-01-16T00:00:00Z 2023-01-30T00:00:00Z 
23988 varken   autogen          23988       2023-01-16T00:00:00Z 2023-01-17T00:00:00Z 2023-01-31T00:00:00Z

All of my retention policies are set for 14 days (if it helps disclosing this).

Thank you.

2

The issue is in some of the data coming into the system… whatever their timestamp is older than 14d (your retention policy).

You won’t find much on the DB side, you can look in the log files but I don’t expect to find a sample of the lost/invalid data there

3

@Giovanni_Luisotto What would be the best way to handle this? Increasing retention policy?

4

The best way is to understand what is sending data that are more than 14d old…
You can increase your RP (if that makes sense to you) but by how much? you can’t tell without knowing the data

5

What is the best way to see this data that is older than 14d?

6

Sorry to bump this and pardon my ignorance in advance but what would be the best way to determine what is sending data beyond my retention policy?

7

no worries, but the answer is pretty simple… I have no idea.

Someone should know who/what is sending data… then someone must check what they are actually sending looking for “wrong” timestamps…
If you are using Telegraf to send the data to InfluxDB it has to be one of the input plugins… but anything can send data to InflxDB…

The topic is so broad and generic that I have nothing more to say…

To “reduce” the scope of your search you might search in the logs (meaning any log, not strictly InfluxDB related) looking for an IP or a hostname or whatever or look into your data looking for a tag (host/device/… ) telling you what is sending data… but if all the data sent by a device are rejected your data won’t help, rather the opposite

8

I should have given more details. My apologies. My data is coming through to InfluxDB via Telegraf input plugins and I am fully aware of who is sending data. I am looking to view the raw data and figure out who is responsible for the old data. I can login to InfluxDB via SSH but don’t know which commands to issue to view this data, ideally, with timestamps.

9

In this case add an output file to Telegraf, so you can see all the data there