Database corruption?

Store
1

I have a database in some strange state. This is on an InfluxDB 1.2.2 server with multiple databases and it’s only affecting one of the databases. What I am seeing:

  • the line protocol looks fine and is accepted by the DB
  • querying out the same records (by measurement and timestamp) returns empty or strange/corrupt looking tag values
  • if I do a query with a “WHERE” for the tag values, I do get the correct results (what was in the line protocol) but the tag values returned are still wrong
  • the field values and timestamps seem to be fine

I know this isn’t a great description, but I am hoping someone with more knowledge of the internals could hazard a guess about where the problem might be and help direct my investigation.

2

@sloach Can you share the version you are running, some sample strange tag values returned as well as the corresponding writes. Also if you could provide a reproduction case that would help greatly.

3

Turns out a restart of influxdb cleared this. So it must have been some sort of index corruption I guess, in the in-memory index. Seems like certain series, and any new series, were impacted. Most of the tag values returned were null and a few looked like maybe they were part of a line protocol line.

I have no idea what triggered this or how to reproduce it, sorry. I was hoping someone with more knowledge of the internals would have an idea how this could happen: the lookup seemed to work fine (tag values in the WHERE) but the output was all wrong.

1 Like
4

@sloach Glad you were able to clear that up. I would still highly advise that you upgrade to the newest version. We won’t be able to help you much if you stay on the 0.9.x bits. What version exactly are you running?

5

I’m running 1.2.2.

Scot

6

The same database got into that same state again today. Is there anything I should to to investigate this when it happens?

7

@sloach :facepalm: Sorry that response was supposed to be to a different thread. I would try upgrading the 1.2.4 release. Can you post some of the examples of the query output when the DB is in a weird state? Also any relevant graphs would help.