InfluxDB doesn't write data after manual intervention

InfluxDB 1
1

Hey @all,
I have an docker InfluxDB v1.8 Instance with data of almost one year. I wanted to migrate the data to another retentionPolicy (to autogen). While migrating the influx crashed because of memory limitations (go error, malloc). Because of this, I manually moved some directories, so the DB could work on less data. With these modifications, the migration worked just fine, but it seems to have corrupted my database.

I can query the old data, but new data is no longer written into the DB. My log looks perfectly normal:

[httpd] 172.19.0.17 - - [05/Sep/2022:09:43:53 +0000] "POST /write?db=MyDB&rp=autogen&precision=n&consistency=one HTTP/1.1 " 204 0 "-" "okhttp/4.10.0" 40a60ac7-2cff-11ed-826d-0242ac130007 6955

But I can’t query the data, or there is none.
Apart form that I can insert more data into unrelated measurements within the same database.

I tried exporting the data into a portable backup and import that into a new DB (fresh docker volume) but it didn’t fix the problem. Also I have checked the tsm files:

influx_inspect verify -dir /var/lib/influxdb
/var/lib/influxdb/data/MyDB/autogen/10/000000027-000000003.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/11/000000020-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/12/000000025-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/13/000000024-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/14/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/15/000000021-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/16/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/17/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/18/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/19/000000021-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/2/000000033-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/20/000000025-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/21/000000018-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/22/000000022-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/23/000000039-000000003.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/24/000000026-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/25/000000024-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/26/000000017-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/27/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/28/000000015-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/29/000000013-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/3/000000016-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/30/000000013-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/31/000000020-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/32/000000026-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/33/000000002-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/34/000000002-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/35/000000003-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/36/000000001-000000001.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/4/000000012-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/5/000000033-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/6/000000028-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/7/000000027-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/8/000000027-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/autogen/9/000000033-000000002.tsm: healthy
/var/lib/influxdb/data/MyDB/policy_360d/71/000000001-000000001.tsm: healthy
Broken Blocks: 0 / 122527, in 1.489326344s

I have no idea why I can’t write more data. Can anyone help me narrowing down the problem? I will happily provide more data or logs if needed.

Thanks, Arne

2

Hello @arne.s,
Whoa I’ve never seen this before. Just to verify that I understand you correctly:

  1. You moved old data to a new location to minimize the migration size.
  2. The migration to a new RP worked now.
  3. You can:
  • write new data to a different DB and RP AND query it
  • query old data from the new RP/measurement
  1. You can’t:
  • write data to the new RP/measurement (or aren’t sure whether you can nor can’t?)
  • query new data from the new RP/measurement

Is that correct?

Please bear with me as we try to verify assumptions.
How are you writing new data?
What errors do you get if any?
What errors do you get when you try to query it? How are you querying the new data in the new RP?
Can you tried writing a dummy point with cURL?

curl -i -XPOST 'http://localhost:8086/write?db=mydb'
--data-binary '<new RP>,test=write value=foo'

And then querying it?

Thank you!