PachaTree migration failed part-way — is there any supported way to resume it?

InfluxDB 3
1

Setup: InfluxDB 3 Enterprise (home licence), single node, object-store=file, shard-count=1. Now on 3.11.1.

What happened

On 2026-08-01 I upgraded 3.10.5 → 3.11.0 and restarted with --upgrade-pacha-tree. The migration ran for 14 minutes and stopped at status: failed.

The trigger was on my side: 14 catalog entries pointed at Parquet files that no longer existed on disk — deleted by retention months before the upgrade, i.e. a pre-existing catalog-integrity issue. Every failure reads:

External error: External: Object at location
/data/influxdb3/<node>/dbs/25/1/2026-06-11/08-50/00000000000015021636.parquet
not found: No such file or directory (os error 2)

I dropped the 11 affected tables the next day, so the original blocker is gone.

Where that left the cluster

system.upgrade_parquet, unchanged for 10 days:

status count
imported 6344
failed 14
not_started 630 (2.0 GiB)

The catalog is in hybrid ParquetAndPachaTree mode. No data loss — every table is queryable and hybrid reads across the cutover are exact. The problem is that ~21 GiB of pre-upgrade data now sits in a frozen legacy tree: retention still hides out-of-retention rows from queries, but no longer deletes the underlying files. Nothing has been written to or removed from that tree since the cutover instant, so “wait for it to expire” isn’t an option.

What I’ve tried

1. Restarting with --upgrade-pacha-tree again (3.11.1). The flag is parsed and honoured, and then:

upgrade_coordinator: Skipping enterprise upgrade coordinator.
                     compactor=<node> import=migration status=Failed
bulk_import_scheduler: No work remaining; waiting for new coordinators.

So a Failed import is deliberately never auto-resumed. Both state files came back byte-identical.

2. influxdb3 manage cleanup-parquet --dry-run (new in 3.11.1):

409 Conflict: catalog storage mode is ParquetAndPachaTree; cleanup requires PachaTree

Correctly gated — but unreachable, because I have no way to get to PachaTree.

3. influxdb3 import upload per file. This works well and is safe — additive, dedupes on time+tags, no duplicate rows. I used it to make one table PachaTree-native. But it runs under its own import name, so the migration ledger still reads failed and the legacy files stay on disk.

Questions

  1. Is there a supported way to reset or retry a bulk import stuck at Failed, so the coordinator picks up the 630 not_started files? I can’t find one in serve, manage or import, or in the docs.
  2. Should a missing source file be skippable rather than terminal? One dangling catalog reference stopped 630 unrelated files from ever being migrated, and the failed status is sticky even after the offending tables are dropped.
  3. Is the catalog storage mode readable anywhere other than by triggering that 409? It’s the authoritative “did the migration complete” signal, but it appears in no system table.

The only documented exit I can see is manage downgrade-to-parquet, which discards everything written since the upgrade — 10 days now, and growing daily. I’d rather not.

Happy to supply the full ledger, state files or logs.

Many thanks,
Peter

2

Hey @pradig do you have any retention period set up on your database/tables?

If you’re not sure, you could take the db and table ID from the path in the error for the missing file:

/data/influxdb3/<node>/dbs/25/1/

is database with ID 25 and table with ID 1.

You could see which database/table these correspond to via system table queries:

select * from system.databases
select * from system.tables where database_id = 25

It is possible that retention removed a file while the migration was taking place. We have opened a ticket internally to look into this.

3

Thanks @hiltontj — yes, retention is configured, and the hypothesis is half right: retention
did delete those files. But not during the migration. Every retention-caused deletion here
predates the upgrade by weeks to months, and the remaining five refs are in a database that had
no retention at all.

Here’s the mapping you asked for.

The three databases behind the 14 failed refs

path in the error database retention on 2026-08-01 refs
dbs/25/1/ network, table vigor 30d 1
dbs/32/{9..16}/ proxmox, 8 stray tables 30d 8
dbs/lorawan-7/lorawan_{up,rx}-{0,1}/ lorawan none 5

I didn’t need the ID lookup in the end — system.upgrade_parquet carries plain
database/table names. Worth flagging for anyone else following this thread, though:
select * from system.tables where database_id = 25 no longer resolves for me, because I dropped those 11 tables on 2026-08-02. The table column in upgrade_parquet also comes back NULL for them now, while database still resolves.

Retention deleted the files — long before the upgrade

The parent directory mtimes date the deletions precisely, and both land exactly one retention
period after the partition timestamp named in the dangling reference:

db partition in the failed ref + 30d actual dir mtime vs. upgrade (2026-08-01 14:30)
25 network 2026-06-11 08:50 2026-07-11 08:50 dbs/25/ 2026-07-11 08:58 21 days earlier
32 proxmox 2026-03-13 18:10 2026-04-12 18:10 dbs/32/ 2026-04-12 18:26 111 days earlier

Table dir 1 is absent under dbs/25/ (0, 2–8 present), and 9–16 are absent under dbs/32/
(0, 1, 2, 5–8 present) — consistent with a retention sweep removing each table’s whole
directory at those moments. So there was no race with the migration: the catalog had been
carrying references to files that retention removed 3 weeks and 3.6 months earlier.

The other five can’t be retention at all

lorawan had no retention period configured on 2026-08-01 — I only set 30d on 2026-08-03, two
days after the failure. Those five refs are also the only ones in the whole ledger using the
old <name>-<id> directory scheme (dbs/lorawan-7/lorawan_up-0/…) instead of bare numeric ids. dbs/lorawan-7/ doesn’t exist on this node at all; the live path is dbs/7/. Whatever removed those files, retention wasn’t it.

One useful difference: before I dropped those two tables, any query whose time predicate
overlapped 2025-08-11 hard-failed mid-body with the same NotFound. So the file was genuinely
gone, not merely mis-pathed. The network and proxmox orphans instead returned 0 rows
silently at any time range, because retention prunes them out of the query plan — which is
exactly why they sat unnoticed for months.

What I think this means

The condition the migration tripped on isn’t “retention raced the import”. It’s that the
catalog can retain references to files that no longer exist, apparently indefinitely, and
normal querying hides that from the operator whenever retention is active. Any long-lived
instance with retention enabled could therefore hit this on its first PachaTree upgrade with no
prior warning.

The cost is also disproportionate to the damage. All 11 failed jobs recorded
bytes_written=0 row_count=0 — nothing was actually being lost — yet those 14 refs stopped 630 unrelated, healthy files (2.0 GiB) from ever being migrated, and the Failed status is sticky.
On 3.11.1 the coupling is now explicit in the logs:

upgrade_coordinator: Skipping enterprise upgrade coordinator.
                     compactor=<node> import=migration status=Failed

One dangling reference from months ago → permanently frozen migration → 21 GiB stuck in the
legacy tree, with no route to PachaTree and therefore none to cleanup-parquet either.

Two things would have prevented it: a pre-flight catalog-integrity check before the migration
starts, and treating a missing source file as skippable-and-recorded rather than terminal —
that alone would have let the other 6,974 files through.

But I still need a way out of the state I’m actually in, which is the one question I can’t
answer from the docs: is there a supported way to reset a bulk import from Failed back to a resumable state, now that the offending tables are gone? downgrade-to-parquet costs me 11 days of writes today and grows daily.

Happy to run any query, or hand over the ledger, state files or logs.

Minor reporting nit while I’m here: the ledger has one row per source file, but last_message
appears to be per job. My three lorawan rows for the 12-30 and 12-40 partitions all
carry the message naming the 12-50 path, which is misleading when you’re trying to work out
precisely which files are missing.