Dear All!
I am using influxdb2 for few years together with openhab for my home automation.
I upgraded my Synology NAS and in this step I want to upgrade also the influx database to V3.
Influxdb3 is running on my NAS (with 32GB memory) as a Docker image.
But then I ran into several problems importing my old data from influxdb2 …
What I did:
-
First I tried to exported it into a big lp.gz file (about 240MByte (from 2022 …)); this was one of my first try, but it is not working on my NAS with limited (home license) cpu/memory.
So I exported the data via script day by day to a lp file (from the old influxdb2) …
…
influxd inspect export-lp \
–bucket-id “$BUCKET_ID” \
–engine-path “$ENGINE_PATH” \
–start “$start_ts” \
–end “$end_ts” \
–output-path “${OUTPUT_DIR}/export_${current_date}.lp.gz” \
–compress
…
Working well, no problems. -
I tried importing it several times, but I encountered multiple issues (memory, hanging …) so I made my last attempt:
-
-
I wrote a smal script to merge the daily files into monthly files
-
I setup the memory threshold to about 250MByte - INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD=256
For less imported data, the compressor should be able to work more easily … -
The I imported it in chunks of about 80000 lines (INFLUXDB3_MAX_REQUEST_SIZE_BYTES=104857600) …, to get larger L1 files !
export_2022-71.lp to export_2023-05.lp (about 216MByte of lp lines), then I stoped the import to check the compaction process! -
This wrote me L1/GEN1 parquet files (enterprise version) without problems …, about 227000 parquet files!
-
But if I look at the compactor I found many plans but only few run_completed …
-
In the meantime I switched from the home license temporarly to the trial licens to have more cpu’s available, but it doesn’t help.
-
-
I have tried several times to start over with changed settings, but now I have no idea what the actual problem is.
It looks like the compactor is not selecting the old data sets for compression.
Setups/logs:
-
influxdb3 show nodes --token apiv3_
| node01 | 0 | 30b7abd4-545d-4926-b82a-d4264b6d0991 | [all] | 6 | running | 2026-05-01T17:50:24.255 | 063b9e0efa36:8181 | {“exec-mem-pool-bytes”:“8000000000”,“num-datafusion-threads”:“4”,“license-email”:“*******”,“gen1-lookback-duration”:“5y”,“mode”:“all”,“license-type”:“trial”,“query-file-limit”:“1000”,“parquet-mem-cache-size”:“100”,“retention-check-interval”:“24h”,“table-index-cache-max-entries”:“200”,“serve-invocation-method”:“docker-hub”,“cluster-id”:“cluster01”,“object-store”:“file”,“last-cache-eviction-interval”:“1m”,“datafusion-max-parquet-fanout”:“50000”,“compaction-gen2-duration”:“20m”,“plugin-dir”:“/var/lib/influxdb3/plugins”,“compaction-max-num-files-per-plan”:“1000”,“node-id”:“node01”,“wal-replay-concurrency-limit”:“1”,“data-dir”:“/var/lib/influxdb3/data”,“log-filter”:“info,influxdb3_enterprise=debug,influxdb3_compactor=debug”,“wal-snapshot-size”:“300”,“compaction-check-interval”:“30m”,“num-io-threads”:“2”,“force-snapshot-mem-threshold”:“256”} | -
docker inspect influxdb3-enterprise
see attached file … -
compose.yaml
see attached file … -
current docker_logs_influxdb3-enterprise.txt
see attached file …
I think it can only be a small thing why the compactor is bypassing my old imported data!
Does anyone have any ideas?
Many greetings from Vienna, HG
PS: I could not upload files, nor allowed for new user ?!?
