feat: Unified Migration System Design by quake · Pull Request #1323 · nervosnetwork/fiber

quake · 2026-05-05T03:19:46Z

Impl of #1256

Copilot

Pull request overview

This PR implements a unified, callback-driven database migration flow in crates/fiber-store (usable across node/CLI and WASM), and archives the legacy standalone fnn-migrate tool for pre-INIT_DB_VERSION upgrades.

Changes:

Refactors fiber-store migrations to run via auto_migrate() with confirm/progress callbacks, and wires this into native (fiber-bin) and WASM (fiber-wasm) startup.
Archives the old migrate/ tool as migrate_archive/ and removes it from the Makefile check target.
Updates migration-related tests and sample fixture version strings to align with the new migration epoch/versioning.

Reviewed changes

Copilot reviewed 21 out of 52 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
migrate_archive/src/util.rs	Legacy helper utilities for archived migrator (bincode convert + prompt).
migrate_archive/src/tests/sample_tests.rs	Archived tool tests for v0.7.0 sample round-trips and determinism.
migrate_archive/src/tests/mod.rs	Archived tool test module wiring.
migrate_archive/src/tests/migration_tests.rs	Archived tool tests validating v0.6.1→v0.7.0 migration behavior.
migrate_archive/src/migrations/sample.rs	Archived placeholder migration object example.
migrate_archive/src/migrations/mod.rs	Archived migration module registry list.
migrate_archive/src/migrations/mig_20260302_channel_replay_fields.rs	Archived migration appending new replay-related trailing fields.
migrate_archive/src/migrations/mig_20260302_channel_open_record.rs	Archived migration deleting ephemeral ChannelOpenRecord entries.
migrate_archive/src/migrations/mig_20260301_network_state_pubkey.rs	Archived migration clearing legacy network actor state entries.
migrate_archive/src/migrations/mig_20260204_pubkey_channel_index.rs	Archived migration rebuilding pubkey-based channel index.
migrate_archive/src/migrations/mig_20260203_trampoline.rs	Archived migration for trampoline-related schema changes (channel/payment/attempt).
migrate_archive/src/migrations/mig_20260203_channel_index.rs	Archived migration building channel→attempt index for non-final payments.
migrate_archive/src/migrations/mig_20251219.rs	Archived migration v0.6.0→v0.6.1 channel state conversion.
migrate_archive/src/migrations/mig_20250724.rs	Archived “breaking change” migration marker.
migrate_archive/src/migrations_backup/mig_20250617.rs	Archived backup migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250507.rs	Archived backup invoice migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250428.rs	Archived backup channel/payment migrations (historical reference).
migrate_archive/src/migrations_backup/mig_20250407.rs	Archived backup broadcast-message cleanup migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250318.rs	Archived backup v0.4.0→v0.4.1 channel migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250308.rs	Archived backup timestamp key consolidation migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250307.rs	Archived backup payment-session migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250227.rs	Archived backup watchtower schema migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250123.rs	Archived backup v0.3.1→v0.4.0 channel migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250115.rs	Archived backup v0.2.1→v0.3.0 channel migration (historical reference).
migrate_archive/src/migrations_backup/mig_20250114.rs	Archived backup v0.2.0→v0.2.1 channel migration (historical reference).
migrate_archive/src/main.rs	Archived standalone migrator binary entrypoint.
migrate_archive/src/lib.rs	Archived migrator crate module exports.
migrate_archive/README.md	Documents that `migrate_archive/` is reference-only and how to upgrade old DBs.
migrate_archive/Cargo.toml	Archived migrator crate manifest (not in workspace).
migrate_archive/build.rs	Archived build script generating migration registration code.
Makefile	Removes `cd migrate && cargo check` from the `check` target.
docs/plans/2026-04-09-unified-migration-system.md	Implementation plan for the unified migration system.
crates/fiber-wasm/src/lib.rs	Uses `open_store_with_migration` and adds WASM confirm/progress callbacks.
crates/fiber-types/src/sample/mod.rs	Updates fixture path examples to v0.8.0.
crates/fiber-types/src/sample/broadcast.rs	Updates sample broadcast message version string to 0.8.0.
crates/fiber-store/src/migrations/mod.rs	Adds migrations module with build-time registration include.
crates/fiber-store/src/migration.rs	Introduces callback-based auto-migration, new errors, and new INIT epoch.
crates/fiber-store/src/lib.rs	Exposes the new `migrations` module.
crates/fiber-store/src/db_migrate.rs	Simplifies DbMigrate to delegate to `Migrations::auto_migrate()`.
crates/fiber-store/Cargo.toml	Removes `indicatif`/`console` dependencies from fiber-store.
crates/fiber-store/build.rs	Updates build script to scan `src/migrations` and generate registration/latest version.
crates/fiber-lib/src/tests/test_utils.rs	Updates store opening call sites (formatting/compat with new API).
crates/fiber-lib/src/store/tests/store.rs	Updates store opening call sites (formatting/compat with new API).
crates/fiber-lib/src/store/tests/migrate.rs	Reworks migration tests for the new `auto_migrate` flow.
crates/fiber-lib/src/store/store_impl/mod.rs	Adds `open_store_with_migration` and wires unified auto-migrate into store open.
crates/fiber-lib/src/store/mod.rs	Re-exports `open_store_with_migration` and migration plan/progress types.
crates/fiber-lib/src/fiber/tests/gossip.rs	Updates store opening call sites (formatting/compat with new API).
crates/fiber-lib/src/build.rs	Removes build-time scanning of legacy `migrate/` directory.
crates/fiber-lib/Cargo.toml	Removes no-longer-needed deps (e.g. indicatif/ouroboros).
crates/fiber-bin/src/main.rs	Wires CLI confirm/progress callbacks into store open at startup.
Cargo.lock	Updates lockfile to reflect dependency graph changes (plus additional version shifts).

quake · 2026-05-05T03:54:28Z

@copilot apply changes based on the comments in this thread

- Fix stderr flush compile error in fiber-bin cli_confirm - Remove unused StorageBackend import in migration test - Include db version and migration guidance in check_validate messages

Zhangcy0x3

I've left a couple of technical suggestions regarding WASM error handling and some minor API improvements for BTreeMap. Please take a look when you have time.

Detailed plan covering 4 phases (12 tasks) for migrating from the standalone migrate/ crate to a unified migration system in fiber-store that works across all backends (RocksDB, SQLite, IndexedDB).

…ystem)

…catif/console deps

…ration code

…gration

…migration tests

…stem - Fix fiber-bin/fiber-wasm imports to use fnn::store re-exports instead of direct fiber_store::migration (fiber-bin doesn't depend on fiber-store) - Add MigrationPlan/MigrationProgress re-exports in fiber-lib/src/store/mod.rs - Remove obsolete migration scanning code from fiber-lib/src/build.rs - Fix migration.rs auto_migrate() to use effective_latest_version() that considers dynamically registered migrations (fixes test failures) - Fix test_run_migration open_db call to pass &Path reference - Remove unused mut on DbMigrate variable in test

- Add open_store(path) that auto-confirms migration (for tests/simple setups) - Rename old 3-param version to open_store_with_migration (for CLI/WASM) - Fix all v0.7.x references to v0.8.x in migration messages, samples, docs

- Fix stderr flush compile error in fiber-bin cli_confirm - Remove unused StorageBackend import in migration test - Include db version and migration guidance in check_validate messages

…t methods)

- Use BTreeMap::range() instead of .iter().filter() for O(log n) lookup - Replace .expect() with .ok() in get_db_version to avoid WASM trap - Remove stale migrate/ references from release workflow - Introduce MigrationStore object-safe trait backed by StorageBackend to keep Migration dyn-compatible while avoiding concrete Store type

quake added this to the v0.9 milestone May 5, 2026

quake mentioned this pull request May 5, 2026

feat: Unified Migration System Design #1268

Closed

quake requested review from chenyukang, Copilot, doitian and jjyr May 5, 2026 03:45

Copilot started reviewing on behalf of quake May 5, 2026 03:46 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Comment thread crates/fiber-store/src/migration.rs

Comment thread crates/fiber-bin/src/main.rs

Comment thread crates/fiber-lib/src/store/tests/migrate.rs

Comment thread crates/fiber-lib/src/store/store_impl/mod.rs

Zhangcy0x3 reviewed May 6, 2026

View reviewed changes

Comment thread crates/fiber-store/src/migration.rs

Comment thread crates/fiber-store/src/migration.rs Outdated

chenyukang reviewed May 6, 2026

View reviewed changes

Comment thread migrate_archive/src/migrations_backup/mig_20250114.rs

chenyukang reviewed May 7, 2026

View reviewed changes

Comment thread crates/fiber-store/src/migration.rs Outdated

quake added 17 commits May 7, 2026 12:21

docs: add unified migration system implementation plan

b78e9a1

Detailed plan covering 4 phases (12 tasks) for migrating from the standalone migrate/ crate to a unified migration system in fiber-store that works across all backends (RocksDB, SQLite, IndexedDB).

chore: archive migrate/ to migrate_archive/ (ref: unified migration s…

c8a8628

…ystem)

chore: remove old migrate/ check from Makefile

2b16232

refactor: migration trait to use callback-based progress, remove indi…

4fcb7a2

…catif/console deps

refactor: simplify DbMigrate to delegate to Migrations::auto_migrate()

5763084

feat: add empty migrations directory with auto-registration support

d4a9b47

refactor: update build.rs to scan src/migrations/ and generate regist…

cadb619

…ration code

refactor: open_store() accepts confirm/progress callbacks for auto-mi…

dc06b17

…gration

feat: CLI confirm/progress callbacks for database migration

1208180

feat: WASM auto-confirm callback for database migration

dc53bfc

test: update open_store() calls with auto-confirm callbacks, rewrite …

670b8fb

…migration tests

chore: rebase

4a43566

chore: cargo fmt

6be51c4

chore: update Cargo.lock

bbd2fe3

fix: apply copilot review suggestions for PR nervosnetwork#1323

2ef4dfb

- Fix stderr flush compile error in fiber-bin cli_confirm - Remove unused StorageBackend import in migration test - Include db version and migration guidance in check_validate messages

quake added 3 commits May 7, 2026 12:23

fixup: restore StorageBackend import in migrate test (needed for trai…

691d80f

…t methods)

chore: rebase

c9e0250

quake force-pushed the quake/rebase-on-develop branch from 5f9ebc2 to c9e0250 Compare May 7, 2026 04:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Unified Migration System Design#1323

feat: Unified Migration System Design#1323
quake wants to merge 20 commits intonervosnetwork:developfrom
quake:quake/rebase-on-develop

quake commented May 5, 2026 •

edited by chenyukang

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

quake commented May 5, 2026

Uh oh!

Zhangcy0x3 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

quake commented May 5, 2026 • edited by chenyukang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

quake commented May 5, 2026

Uh oh!

Zhangcy0x3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

quake commented May 5, 2026 •

edited by chenyukang

Loading