fix: resolve payment on RemoveTlc(Fulfill) for force-closed channels (#1222) by doitian · Pull Request #1254 · nervosnetwork/fiber

doitian · 2026-04-07T08:48:03Z

Summary

When a channel is force-closed, the channel actor stops and any RemoveTlc(Fulfill) from the peer is silently dropped, leaving the sender's payment session stuck in Inflight forever. Fix handle_peer_message to intercept these messages: look up the TLC from persisted channel state, relay the fulfillment upstream for intermediate nodes, or emit TlcRemoveReceived for the payment sender.
Add unit tests for one-hop (A force-closes, B settles) and two-hop (B force-closes B→C, C settles, fulfill relays back to A) on-chain settlement scenarios.
Add force-close-preimage-settled-by-recipient Bruno e2e test and register it in CI.

Test plan

cargo nextest run -p fnn -p fiber-bin — 767 tests pass, 0 failures
test_one_hop_payment_payee_settles_onchain passes
test_two_hop_payment_payee_settles_onchain passes
CI e2e watchtower/force-close-preimage-settled-by-recipient passes
Payee should update invoice status to Paid
Payer should update TLC status to RemoteRemoved

gpBlockchain · 2026-04-08T06:55:39Z

The current test has passed, but whether the results of list_channel and get_invoice need to be synchronizedly modified ？

lnd0->lnd1->fiber1->fiber2

fiber2 successfully settle on chain
fiber2.get_invoice -> Received

{"jsonrpc": "2.0", "id": 42, "result": {"invoice_address": "fibd1000001py803aefk6mczxzat8lhtajtd96je3ufftyatpn5tj7g7u3c933f8hzwguvwzu6575f6072fjah0gup4axf4ctf4lfjxnk4ztc7wteyymdrar0arre2sxqt5nljy4dfwnjutmhfuj6phrljsm6dn0cw3ws8atfv57zz7uk8kuqcv0su70celywz829ngu6wg44rpxpqsqdpsm2eckevnta9mqgylugwmaukud5unepswgpc9h8xfs2myah34qyykymt92hctzaw3z2z7zpn6j7m9ms8ldrhmjyn5l0585f000qx5g548092e30d98j473tmws9zkez9x7yx7d7aytd3zzxejzu6728t4qr07yxu85qgz6rzpuvxtjulafasqvx8nywfjj0c927g58lft489mqdxefshnu4faasz57gymh8d9s2rpaxrm4ua2cv2ga84agg9g0rzjjvu6phhxpdaq4kgz567hqaw6jcqp6vkkhr", "invoice": {"currency": "Fibd", "amount": "0x186a0", "signature": "0c060713040e0912120f18050a1e0814071f090b1507051b000d0619091017131c15091d1d1002141e08041b17070d05100a03011d06031b151c1d0a180c0a081d07151d080805080f030212120c1c1a01171706010d1d0015160802141a1e17001d0e1a12180001", "data": {"timestamp": "0x19d6bcc50e6", "payment_hash": "0xb00ce15c1a83174e2ea4576a4d7cfbcbe79ce1a6c364501c2910403ff635a116", "attrs": [{"description": "already settled invoice"}, {"final_htlc_minimum_expiry_delta": "0x927c00"}, {"udt_script": "0x55000000100000003000000031000000102583443ba6cfe5a3ac268bbb4475fb63eb497dce077f126ad3b148d4f4f8f8012000000032e555f3ff8e135cece1351a6a2971518392c1e30375c1e006ad0ce8eac07947"}, {"hash_algorithm": "sha256"}, {"payee_public_key": "03362ada447f3b1334fb1ee5786641c8deaba43e1dcc8e44eca3e2118872d1aba7"}]}}, "status": "Received"}}

fiber2.list_channel->"status": {"Inbound": "LocalRemoved"}}

{"jsonrpc": "2.0", "id": 42, "result": {"channels": [{"channel_id": "0xb346fe2c42093cec58eb420b66530feddbfd0003098ebd51c3b0e30dbd91a316", "is_public": true, "is_acceptor": false, "is_one_way": false, "channel_outpoint": "0xde25fb461c194c04237afb7380ca37a77a8a997927976ee49947a0136c28ef8d00000000", "pubkey": "02fd4cfea10f398c5e3a27c5f383ee9aa4d11fd82f35f28a3d9ce2994fe4b7c98f", "funding_udt_type_script": {"code_hash": "0x102583443ba6cfe5a3ac268bbb4475fb63eb497dce077f126ad3b148d4f4f8f8", "hash_type": "type", "args": "0x32e555f3ff8e135cece1351a6a2971518392c1e30375c1e006ad0ce8eac07947"}, "state": {"state_name": "Closed", "state_flags": "UNCOOPERATIVE_REMOTE"}, "local_balance": "0x19968ceb00", "offered_tlc_balance": "0x0", "remote_balance": "0x174876e800", "received_tlc_balance": "0x186a0", "pending_tlcs": [{"id": "0x0", "amount": "0x186a0", "payment_hash": "0xb00ce15c1a83174e2ea4576a4d7cfbcbe79ce1a6c364501c2910403ff635a116", "expiry": "0x19d94b5f5ff", "forwarding_channel_id": null, "forwarding_tlc_id": null, "status": {"Inbound": "LocalRemoved"}}], "latest_commitment_transaction_hash": "0x46cb82891d8bef1a1abe805614edd0b5c2221f72dcfd7852bd49168cd2e298f1", "created_at": "0x19d6bcc154e", "enabled": true, "tlc_expiry_delta": "0xdbba00", "tlc_fee_proportional_millionths": "0x3e8", "shutdown_transaction_hash": "0x9865f79e0da81cb20a73be1e99d04335e3ff10666cc35514e050f16c818e3922", "failure_detail": null}]}}

fiber1.list_channel -> "status": {"Outbound": "Committed"}

{"jsonrpc": "2.0", "id": 42, "result": {"channels": [{"channel_id": "0xb346fe2c42093cec58eb420b66530feddbfd0003098ebd51c3b0e30dbd91a316", "is_public": true, "is_acceptor": true, "is_one_way": false, "channel_outpoint": "0xde25fb461c194c04237afb7380ca37a77a8a997927976ee49947a0136c28ef8d00000000", "pubkey": "03362ada447f3b1334fb1ee5786641c8deaba43e1dcc8e44eca3e2118872d1aba7", "funding_udt_type_script": {"code_hash": "0x102583443ba6cfe5a3ac268bbb4475fb63eb497dce077f126ad3b148d4f4f8f8", "hash_type": "type", "args": "0x32e555f3ff8e135cece1351a6a2971518392c1e30375c1e006ad0ce8eac07947"}, "state": {"state_name": "Closed", "state_flags": "UNCOOPERATIVE_LOCAL"}, "local_balance": "0x174876e800", "offered_tlc_balance": "0x186a0", "remote_balance": "0x19968ceb00", "received_tlc_balance": "0x0", "pending_tlcs": [{"id": "0x0", "amount": "0x186a0", "payment_hash": "0xb00ce15c1a83174e2ea4576a4d7cfbcbe79ce1a6c364501c2910403ff635a116", "expiry": "0x19d94b5f5ff", "forwarding_channel_id": null, "forwarding_tlc_id": null, "status": {"Outbound": "Committed"}}], "latest_commitment_transaction_hash": "0x3a965b46b376ab99d592dad3ef02e1fb9107174cfc5d9b6bfa3d7f4c826abb28", "created_at": "0x19d6bcc1552", "enabled": true, "tlc_expiry_delta": "0xdbba00", "tlc_fee_proportional_millionths": "0x3e8", "shutdown_transaction_hash": "0x9865f79e0da81cb20a73be1e99d04335e3ff10666cc35514e050f16c818e3922", "failure_detail": null}]}}

doitian · 2026-04-08T08:49:44Z

+                                            )
+                                        })
+                                    {
+                                        self.store


TODO: check the invoice is fully paid.

Copilot

Pull request overview

Fixes a long-standing edge case in Fiber’s payment lifecycle where a peer’s RemoveTlc(Fulfill) can arrive after a channel has been force-closed and the channel actor is already gone, leaving the sender (and downstream CCH tracking) stuck in Inflight. The PR adds network-level handling to recover fulfillment from persisted channel state, plus tests to validate one-hop and two-hop on-chain settlement flows and a new watchtower Bruno e2e scenario.

Changes:

Intercept RemoveTlc(Fulfill) for force-closed channels in NetworkActor::handle_peer_message, recover TLC info from persisted channel state, and propagate fulfillment upstream / to the payment actor.
Ensure invoices are marked Paid when a fulfilled received-TLC is locally removed but the commitment round-trip never completes (common around force-closes).
Add unit tests and register a new Bruno e2e watchtower scenario in CI.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`crates/fiber-lib/src/fiber/network.rs`	Adds fallback handling for fulfill removes on closed channels and additional invoice-status reconciliation in periodic channel checks.
`crates/fiber-lib/src/store/store_impl/mod.rs`	Emits `StoreChange::PutPreimage` when inserting watchtower preimages so downstream watchers (e.g. CCH) can react.
`crates/fiber-lib/src/fiber/tests/payment.rs`	Adds one-hop and two-hop unit tests covering on-chain settlement after force-close.
`.github/workflows/e2e.yml`	Registers the new Bruno e2e scenario in CI.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/01-node1-connect-node2.bru`	New e2e step: connect Node1↔Node2.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/02-node2-connect-node3.bru`	New e2e step: connect Node2↔Node3.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/03-node1-node2-open-channel.bru`	New e2e step: open Node1→Node2 channel.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/04-node2-get-auto-accepted-channel.bru`	New e2e step: discover N1–N2 channel id.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/05-ckb-generate-blocks.bru`	New e2e step: mine epochs for N1–N2 channel confirmation.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/06-node2-node3-open-channel.bru`	New e2e step: open Node2→Node3 channel.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/07-node3-get-auto-accepted-channel.bru`	New e2e step: discover N2–N3 channel id.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/08-ckb-generate-blocks.bru`	New e2e step: mine epochs for N2–N3 channel confirmation.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/09-get-node1-funding-script.bru`	New e2e step: read Node1 funding lock script.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/10-get-node3-funding-script.bru`	New e2e step: read Node3 funding lock script.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/11-get-node1-balance.bru`	New e2e step: snapshot Node1 chain balance.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/12-get-node3-balance.bru`	New e2e step: snapshot Node3 chain balance.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/13-node3-gen-invoice.bru`	New e2e step: create hold invoice / preimage pair for the scenario.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/14-node1-send-payment-with-invoice.bru`	New e2e step: Node1 initiates payment via invoice.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/15-node2-force-close-channel.bru`	New e2e step: Node2 force-closes N2–N3 to force on-chain resolution path.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/16-node2-disconnect-node3.bru`	New e2e step: disconnect Node2 from Node3.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/17-ckb-generate-blocks-for-force-close-tx.bru`	New e2e step: mine epochs for force-close tx confirmation.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/18-node3-remove-tlc.bru`	New e2e step: Node3 removes TLC with preimage (fulfill).
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/19-ckb-generate-blocks-for-settlement-tx-preimage.bru`	New e2e step: mine epochs for settlement tx/preimage discovery.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/20-ckb-generate-blocks-for-final-settlement-tx.bru`	New e2e step: mine epochs to reach final settlement.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/21-ckb-generate-blocks-for-final-settlement-tx-commit.bru`	New e2e step: mine epochs to commit final settlement.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/22-check-channel1-balance.bru`	New e2e assertion: N1–N2 channel balance reflects claimed amount.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/23-check-payment-status.bru`	New e2e assertion: payment status becomes `Success`.
`tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/24-disconnect.bru`	New e2e teardown: disconnect peers.

When a channel is force-closed the channel actor stops, so any RemoveTlc(Fulfill) arriving from the peer is silently dropped. This leaves the sender's payment session stuck in Inflight forever. Handle this in handle_peer_message: when a RemoveTlc(Fulfill) arrives for a channel whose actor is gone, look up the TLC from persisted state. If the TLC has a forwarding_tlc (intermediate node), relay the fulfillment upstream. Otherwise (payment sender), emit TlcRemoveReceived so the payment actor can finalise. Also emit StoreChange::PutPreimage from insert_watch_preimage so watchtower preimage discoveries are observable by store watchers. Made-with: Cursor

Add two tests that verify payment session resolution after a force close when the payee settles the TLC by revealing the preimage: - test_one_hop_payment_payee_settles_onchain: A force-closes, B settles invoice, A's payment should reach Success. - test_two_hop_payment_payee_settles_onchain: B force-closes B→C, C settles invoice, the fulfill relays through B back to A. Made-with: Cursor

Bruno e2e scenario: A→B→C payment where B force-closes the B→C channel and C settles the TLC on-chain with the preimage. Verifies that A's payment status reaches Success and B's channel balance reflects the settled amount. Made-with: Cursor

…lement After a force-closed channel's TLC is settled on-chain: - Payee: CheckChannels now scans received TLCs with LocalRemoved + Fulfill reason on both ChannelReady and Closed(UNCOOPERATIVE) channels and updates the invoice status to Paid. - Payer: the RemoveTlc(Fulfill) handler for force-closed channels now calls set_offered_tlc_removed() to transition the TLC to RemoteRemoved and persists the updated channel state. - Tests: added invoice Paid and TLC RemoteRemoved assertions to both one-hop and two-hop on-chain settlement tests.

Gate the persisted-state fallback path to channels that are actually closed and whose persisted remote pubkey matches the incoming peer, so a reconnect race or unrelated peer cannot drive fulfillment handling for a channel that does not belong to it. Also re-validate the preimage against the TLC payment hash and verify the offered TLC is in Committed status before calling `set_offered_tlc_removed`, which asserts on the status and would otherwise panic the network actor on duplicate or out-of-order removes.

Replace fixed 3s sleeps with `wait_until_success` so the assertions wait as long as needed up to the helper's timeout, reducing CI flakiness under load and producing clearer diagnostics when settlement takes longer than expected.

The payer reaching PaymentStatus::Success and the payee marking its invoice as Paid (and the offered TLC transitioning to RemoteRemoved) are independent events, so asserting them synchronously after wait_until_success races with their propagation and produces flaky CI failures. Wrap the post-settlement invoice and TLC checks in wait_until_async_timeout so they converge within the helper's bound, matching the polling pattern already used elsewhere in the test.

doitian changed the title ~~fix: resolve payment on RemoveTlc(Fulfill) for force-closed channels~~ fix: resolve payment on RemoveTlc(Fulfill) for force-closed channels (#1222) Apr 7, 2026

doitian marked this pull request as draft April 8, 2026 08:46

doitian requested a review from Copilot April 8, 2026 08:47

Copilot started reviewing on behalf of doitian April 8, 2026 08:48 View session

doitian force-pushed the on-chain-tlc-settlement branch from 4fc682d to 85ec855 Compare April 8, 2026 08:48

doitian commented Apr 8, 2026

View reviewed changes

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/fiber-lib/src/fiber/network.rs Outdated

Comment thread crates/fiber-lib/src/fiber/network.rs Outdated

Comment thread crates/fiber-lib/src/fiber/tests/payment.rs

Comment thread crates/fiber-lib/src/fiber/tests/payment.rs

quake added this to the v0.9 milestone May 5, 2026

ian added 4 commits May 5, 2026 15:12

doitian force-pushed the on-chain-tlc-settlement branch from 85ec855 to 0cb440d Compare May 5, 2026 07:35

doitian marked this pull request as ready for review May 5, 2026 07:35

ian added 3 commits May 5, 2026 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: resolve payment on RemoveTlc(Fulfill) for force-closed channels (#1222)#1254

fix: resolve payment on RemoveTlc(Fulfill) for force-closed channels (#1222)#1254
doitian wants to merge 7 commits intonervosnetwork:developfrom
doitian:on-chain-tlc-settlement

doitian commented Apr 7, 2026 •

edited

Loading

Uh oh!

gpBlockchain commented Apr 8, 2026

Uh oh!

doitian Apr 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

doitian commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

gpBlockchain commented Apr 8, 2026

Uh oh!

doitian Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

doitian commented Apr 7, 2026 •

edited

Loading