Data stops being written after a couple of minutes

Store
1

We are seeing a strange phenomenon with our self-hosted InfluxDB 1.7.2 instance. The symptoms are:

  1. Data stops “being written” to a particular measurement with a particular tag value after ~5 minutes of restarting the InfluxDB service. That is: when we query for data the query doesn’t return data after a certain point in time.
  2. After restarting the service, all the historical data seems to be restored somehow. That is: we execute the same query we did before and suddenly the data is there. After 5 minutes we stop seeing data.
  3. We use the REST api to write data; this seems to fo flawlessly. All calls get a 204 response, as we expect.
  4. This only seems to happen for data being written with a particular tag. Other data in the same measurement with a different tag value seem to be unaffected.

It looks like somehow WAL is not being processed correctly, but we’re completely stumped on what this issue might be.

This started when we started writing data to the measurement in question from another source and introduced a new tag value. The old data does not have any of these tag values; maybe this is a problem? Does anyone have any clues for us? Thanks!

2

Indeed strange … if on linux ,do you have errors in /var/log/messages ?

3

There doesn’t appear to be anything relevant in /var/log/messages unfortunately.

4

It is difficult to help you , I would check the wal-fsync-delay setting ,
and check journalctl -u influxdb.service …

5

@MarcV’s suggestion for journalctl should help.

If you filter it for the tag value that is failing, or the source where the values are coming from you should be able to see what happens when it hits InfluxDB

I’m not sure if the 20x codes imply a succesful write, i think it just means that the influx instance is up, running and contactable.

If it is the one tag value that is causing the issues then it is most likely invalid syntax, in past experience when this happens (even if it is valid) Influx will reject it.

can you provide an example of the data as it appears when you insert it?