Follow-up: retry-upgrade-to-pacha-tree worked, but the migration still can’t complete — compactor orphans ~89% of what it writes
Follow-up to PachaTree migration failed part-way — is there any supported way to resume it?
Setup: InfluxDB 3 Enterprise (home licence), single node, object-store=file,
shard-count=1, 12 cores / 62 GB RAM, one SATA SSD. Now on 3.11.2.
First: thank you — 3.11.2 shipped exactly what I asked for in that thread, and the stale-gen1 reference handling matches the diagnosis I posted on 2026-08-12. The retry command does what it says. The migration still doesn’t complete, for a different reason, and that’s what this
post is about.
1. The retry command works
Run on 2026-08-22 05:34 UTC:
$ influxdb3 manage retry-upgrade-to-pacha-tree
Sources re-checked: 11
Skipped (source gone): 0
Skipped (table dropped): 11
Requeued: 0
Unresolved: 0
Upgrade status: upgrading
Migration state reset. Restart the compactor node with --upgrade-pacha-tree to resume.
Then upgrade-pacha-tree="true" + restart. The ledger drained in about nine minutes:
| status |
before |
after |
imported |
6344 |
6974 |
failed |
14 |
0 |
skipped_table_dropped |
0 |
14 |
not_started |
630 |
0 |
No data loss at this stage — four fixed-window row-count probes (a settled 1 h window, a
settled 24 h window, a 2020 month in an infinite-retention DB, and a pre-cutover hour that
existed only in the legacy tree) all returned byte-identical counts afterwards.
2. But the migration never completes
Eight hours later, and across five restarts:
SELECT node_id, mode, status FROM system.upgrade_parquet_node;
| node_id |
mode |
status |
| collect-eu |
ingest |
completed |
| collect-eu |
compactor |
upgrading |
$ influxdb3 manage cleanup-parquet --dry-run
409 Conflict: catalog storage mode is ParquetAndPachaTree; cleanup requires PachaTree
The bulk import scheduler is idle and says so — No work remaining; waiting for new coordinators. — yet the compactor node never leaves upgrading.
Q1: what advances the compactor node from upgrading to completed? With the ledger fully drained and the scheduler reporting no work, I can’t find the remaining precondition.
3. The reason, I think: ~89% of compactor output is never referenced
.pt files on disk (find /data/influxdb3/*/pr-eu/cv2/ -name '*.pt'): 126,302
files referenced by system.pt_compaction_files: 13,809
orphaned: 112,493 (89.1%)
Oldest orphan mtime is 2026-08-01 19:12 — hours after the original failed migration. The
newest are being written continuously right now.
This matches everything else I can measure. Over a 40-minute window sampling
system.pt_compaction_files every 5 minutes:
|
value |
change |
| referenced files |
13,809 |
constant |
| distinct windows |
2,152 |
constant |
| rows |
234.1M → 234.4M |
+7,357/min |
and measured ingest over the same period was 5,876 lines/min — i.e. the only growth is live
data arriving. No imported legacy content is being adopted into the catalog.
Meanwhile the compactor reads far more than it writes:
sum(rate(influxdb3_compactor_input_bytes_total[20m])) ~4,800 GiB/day
sum(rate(influxdb3_compactor_output_bytes_total[20m])) ~270 GiB/day
output ÷ input ~0.06
Pre-migration that input rate was 2–3.5 GiB/day. It has been ~1,400× that, steadily, for
eight hours, on a 15 GB dataset. influxdb3_compactor_windows reports 20,688 and has not
changed in eight hours, while system.pt_compaction_files reports 2,152 distinct windows —
those two appear to count different things, which made the metric misleading as a progress
signal.
Q2: is a ~89% orphan rate expected during or after a migration? If those files are pending
adoption, what drives it? If they’re garbage, what reclaims them — files_queued_for_cleanup
oscillates between ~1,500 and ~4,300 and never drains.
4. system.upgrade_parquet reports “imported” for data that is not queryable in PachaTree
This is the part I’d most like corrected, because it’s the one that could cost someone their
data.
The ledger reads imported=6974, not_started=0. It also names the affected tables directly —
electricity_meter 82 files, adsb_daily 353 files, all imported. But setting
--disable-hybrid-query=true (so reads come only from PachaTree) for ~100 seconds:
| table |
retention |
hybrid ON |
hybrid OFF |
invisible |
electricity.electricity_meter |
infinite |
533,971 |
133,153 |
75% |
tracking_lt.adsb_daily |
infinite |
1,482 |
87 |
94% |
Reverting restored both counts exactly. Nothing was written or deleted — this was read-path
only.
So the legacy Parquet files are still the only readable copy of that data, and
manage cleanup-parquet deletes them. The 409 in §2 is currently the only thing preventing
that. If I had succeeded in reaching PachaTree mode, the documented next step would have destroyed roughly 400,000 rows.
Worth flagging for anyone else verifying a migration: 7 of my 9 probes stayed green during
that loss. Only long-history, infinite-retention tables showed it. A representative sample is
not a safety check here.
Q3: should status = imported in system.upgrade_parquet imply the data is queryable from PachaTree? If not — and evidently it doesn’t — is there a system table that reports actual
PachaTree coverage per table, so a migration can be verified before cleanup-parquet is run?
5. Separately: the default merge concurrency livelocks a single-node local-storage deployment
--compactor-max-concurrent-merges defaults to 64 × available parallelism = 768 here.
After the migration (when compactor_windows went 2,154 → 20,688 and the scheduler began
emitting bursts of 22 plans), that default deadlocked the compactor completely:
compaction plans produced 15
compaction plans COMPLETED 0 ← in 12 minutes
lease renewal timeouts 10
V2 checkpoint reloads 9 (20,688 windows each, ~every 80 s)
with
ERROR ...coordinator::lease_holder: lease renewal failed with error
e=Generic LocalFileSystem error: conditional put: lock acquisition timed out
after 50 attempts: Resource temporarily unavailable (os error 11)
WARN ...coordinator::lease_holder: primary lease lost during renewal
The node expires its own lease and immediately reacquires it — expired_holder and the
reacquiring node_id are the same UUID — reloading the entire checkpoint each time, so no plan ever survives to completion.
This drive’s fsync latency is a sustained 70–78 ms (QLC, no power-loss protection), and I
want to be clear that this is pre-existing and not caused by the migration — node_exporter
shows the same 70–78 ms for hours before it, rising only to ~83 ms after. What changed was
the volume of compaction work.
Capping concurrency restored completions, but did not change the underlying behaviour:
| cap |
compactor input |
output ÷ input |
completed ÷ dispatched |
compactor_windows |
| 768 (default) |
— |
— |
0 plans completed / 12 min |
20,688 |
| 8 |
~2,400 GiB/day |
0.041 |
0.50 |
20,688 |
| 2 |
~4,850 GiB/day |
0.055 |
0.89 |
20,688 |
| 1 |
~4,286 GiB/day |
0.061 |
0.92 |
20,688 |
An 8× reduction in concurrency changed total disk writes by ~5% (441 → 462 GiB/day). The
orphaning is invariant across a 768× range of this setting.
Q4: is 64 × available parallelism intended for object-store=file single-node
deployments? On anything without deep request parallelism it appears able to starve the
primary lease renewal, which needs a durable conditional put on the same filesystem.
What I’ve ruled out
- Concurrency — invariant across 768 / 8 / 2 / 1 (table above).
--final-compaction-age — set to 15m (legacy default is 72h) hoping to force
finalisation to L4. Zero L3/L4 activity across ten samples over 30 minutes. I believe the
idle clock never elapses because Import and Retention plans touch the data continuously.- Waiting — eight hours, five restarts, coverage static throughout.
downgrade-to-parquet — would discard three weeks of writes. Not acceptable.
Where I am now
Running happily in hybrid mode: ingest is correct, every query returns complete results, and
the four correctness probes are exact. The costs are ~460 GiB/day of write amplification on a
consumer SSD, and a legacy tree I can’t retire.
I’m not blocked in any urgent sense and I’m not asking for a fix on any timeline — but I’d
value knowing whether Q1–Q3 indicate a defect or a misunderstanding on my side, and Q3 in
particular seems worth a docs warning regardless.
Happy to supply the full ledger, system.pt_compaction_files extracts, state files, logs or
the orphan file listing.
