Skip to content

(2) sync facade + shuttle/loom backends + workspace sweep#1542

Draft
daniel-noland wants to merge 1 commit into
pr/daniel-noland/concurrency-stack/1-absorb-quiescentfrom
pr/daniel-noland/concurrency-stack/4-workspace-transition
Draft

(2) sync facade + shuttle/loom backends + workspace sweep#1542
daniel-noland wants to merge 1 commit into
pr/daniel-noland/concurrency-stack/1-absorb-quiescentfrom
pr/daniel-noland/concurrency-stack/4-workspace-transition

Conversation

@daniel-noland
Copy link
Copy Markdown
Collaborator

@daniel-noland daniel-noland commented May 15, 2026

Summary

Introduce a parking_lot-shaped facade over Mutex / RwLock, ship the four backends together (std / parking_lot / shuttle / loom), and rewire the data-path crates through it in a single atomic step.

The four pieces are bundled because a partial sweep leaves the workspace in a state where the facade and its consumers disagree on whether Mutex::lock returns LockResult<Guard> or a naked guard. The shuttle/loom wrappers strip LockResult in exactly the same poison-as-panic shape as the std wrapper, so splitting them re-introduces that disagreement under the test/shuttle and test/loom CI jobs.

std::sync wrapper (poison-as-panic)

  • New concurrency::sync module exposing Mutex / RwLock with a parking_lot-shaped naked-guard surface backed by a poison-as-panic wrapper around std::sync. Workspace policy treats a crashed thread as a crashed process: poison surfacing inside a lock acquire panics through concurrency::sync::poisoned() rather than handing back a possibly-torn LockResult.
  • One cold #[inline(never)] fn poisoned() -> ! keeps the hot path branch-free; the wrapper is a single match per acquire.
  • RwLock::upgradable_read is implemented as exclusive write() for now -- sound but lossy. Documented at the backend module level.
  • Arc, Weak, atomics, Condvar, mpsc, OnceLock, etc. are re-exported from std::sync unchanged.

parking_lot as the production default

  • Adds a parking_lot Cargo feature (enabled by default) and concurrency/src/sync/parking_lot_backend.rs -- a zero-cost re-export of parking_lot::{Mutex, RwLock} plus the std re-exports for everything parking_lot doesn't ship.
  • Backend routing in concurrency/src/sync/mod.rs picks parking_lot_backend when feature = "parking_lot" is on, falls back to std_backend otherwise.
  • _strict_provenance feature overrides parking_lot and routes through std_backend: parking_lot_core::word_lock uses integer-to-pointer casts that -Zmiri-strict-provenance rejects, and the CI miri job (--features=_strict_provenance) exercises the fallback slot, which needs std::sync underneath.
  • Workspace Cargo.toml gets a tokio parking_lot override entry so the tokio parking_lot feature is enabled workspace-wide.

shuttle backend

  • concurrency/src/sync/shuttle_backend.rs mirrors std_backend.rs: a thin poison-as-panic wrap around shuttle::sync::{Mutex, RwLock} returning naked guards, plus re-exports for Arc / Weak / atomics / Condvar / mpsc that pass through unchanged. OnceLock falls back to std::sync because shuttle doesn't ship one.
  • Three feature flags share one dep:shuttle, arranged as a chain (shuttle_dfs -> shuttle_pct -> shuttle):
    • shuttle -- random scheduler (default first choice).
    • shuttle_pct -- PCT, biases toward rare interleavings.
    • shuttle_dfs -- DFS, exhaustive small-state exploration.
      Same backend, scheduler chosen at runtime (by concurrency::stress, added in the follow-on PR). The chain means a single feature = "shuttle" cfg check is true under every variant, so the routing in sync/mod.rs collapses to single-feature checks (no any(shuttle, shuttle_pct, shuttle_dfs) chains).

loom backend

  • concurrency/src/sync/loom_backend.rs mirrors the std and shuttle wrappers: poison-as-panic around loom::sync::{Mutex, RwLock} returning naked guards, with LockResult / TryLockError / OnceLock etc. re-exported from std::sync because loom doesn't ship them.
  • Loom 0.7 has no Weak<T> and no Arc::downgrade. The backend adds a local Arc<T> wrapper that adds downgrade plus a Weak<T> shim that holds a strong clone of the inner loom::sync::Arc until the Weak itself drops. Documented quirks (Weak::upgrade always succeeds after a downgrade, Arc::weak_count panics rather than silently returning 0, last Arc drop does not free when a Weak is live) are explicit at the module level -- silent fallbacks would let assertions pass for the wrong reason on every backend.
  • sync/mod.rs routes the loom feature ahead of every other backend.

Workspace transition

  • Sweep the data-path crates (flow-entry, flow-filter, nat, net, pipeline, routing, stats) to consume dataplane_concurrency::sync::{Arc, Mutex, RwLock, atomic, ...} instead of std::sync::* / parking_lot::* directly.
  • Call sites drop the .unwrap() / .expect("poisoned") noise the LockResult-shaped surface forced, since the facade returns naked guards. Same shape as the existing parking_lot::Mutex consumers.
  • concurrency_mode(std) annotations at QSBR-touching test sites in nat::stateful::apalloc::test_alloc, routing::fib::test, and flow-entry::flow_table::table::tests keep the existing std-only feature gating intact.
  • nat::portfw::flow_state::get_packet_port_fw_state simplification: the explicit drop(guard) dance before debug! calls is removed (per the recorded author intent that it isn't worth the code complexity); the guard now drops implicitly at the end of scope.

The sweep is mostly mechanical (use rewrites and .unwrap() removal). The semantic change is one step: every consumer now gets the facade-routed primitives, so flipping the loom/shuttle features at the top of the workspace is now meaningful.

PR 2 of 3 in the concurrency facade stack (the original 4-PR plan collapsed the shuttle-only and loom-only PRs into this one because the wrappers, the workspace sweep, and the consumer changes can't be independently merged without leaving CI red). Depends on #1539 -- absorb-quiescent. The follow-on PR (#1544) adds #[concurrency::test], the stress dispatcher, and the thread::scope shim.

Test plan

  • cargo build --workspace passes.
  • cargo nextest run --workspace passes.
  • cargo nextest run -p dataplane-concurrency --no-default-features passes (the std-backend path).
  • cargo nextest run -p dataplane-concurrency passes (parking_lot default).
  • cargo nextest run -p dataplane-concurrency --features shuttle passes.
  • cargo nextest run -p dataplane-concurrency --features loom --release passes.
  • cargo build -p dataplane-concurrency --no-default-features --features _strict_provenance compiles via std_backend.
  • cargo clippy --workspace --all-targets passes.
  • CI check/miri/powerpc64 passes (covers the _strict_provenance strict-provenance path).
  • CI features/release/shuttle passes.
  • CI features/release/loom passes.

@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/3-facade-parking-lot branch from d560e77 to aa509ce Compare May 15, 2026 21:17
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/4-workspace-transition branch from 03fec49 to 8878b3c Compare May 15, 2026 21:17
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/3-facade-parking-lot branch from aa509ce to cdf3507 Compare May 15, 2026 21:22
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/4-workspace-transition branch 2 times, most recently from 9632629 to 2be0637 Compare May 15, 2026 21:36
@daniel-noland daniel-noland changed the title refactor(workspace): route data-path concurrency through the facade feat(concurrency): parking_lot-shaped sync facade with workspace sweep May 15, 2026
@daniel-noland daniel-noland changed the base branch from pr/daniel-noland/concurrency-stack/3-facade-parking-lot to pr/daniel-noland/concurrency-stack/1-absorb-quiescent May 15, 2026 21:37
@daniel-noland daniel-noland changed the title feat(concurrency): parking_lot-shaped sync facade with workspace sweep (2) parking_lot-shaped sync facade with workspace sweep May 15, 2026
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/4-workspace-transition branch 2 times, most recently from 1932e47 to 1ce8d26 Compare May 15, 2026 22:20
@daniel-noland daniel-noland changed the title (2) parking_lot-shaped sync facade with workspace sweep (2) sync facade + shuttle/loom backends + workspace sweep May 15, 2026
@daniel-noland daniel-noland added ci:+vlab Enable VLAB tests ci:+cross run cross compile jobs labels May 15, 2026
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/1-absorb-quiescent branch from c87ff7f to e292f12 Compare May 15, 2026 23:01
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/4-workspace-transition branch from 1ce8d26 to 8d86891 Compare May 15, 2026 23:02
Introduce a `parking_lot`-shaped facade over `Mutex` / `RwLock`, ship
the four backends together (std / parking_lot / shuttle / loom), and
rewire the data-path crates through it in a single atomic step.

The four pieces are bundled because a partial sweep leaves the
workspace in a state where the facade and its consumers disagree on
whether `Mutex::lock` returns `LockResult<Guard>` or a naked guard.
The shuttle/loom wrappers strip `LockResult` in exactly the same
poison-as-panic shape as the std wrapper, so splitting them re-
introduces that disagreement under the test/shuttle and test/loom CI
jobs.

## std::sync wrapper (poison-as-panic)

New `concurrency::sync` module exposing `Mutex` / `RwLock` with a
`parking_lot`-shaped naked-guard surface backed by a poison-as-panic
wrapper around `std::sync`. Workspace policy treats a crashed thread
as a crashed process: poison surfacing inside a lock acquire panics
through `concurrency::sync::poisoned()` rather than handing back a
possibly-torn `LockResult`.

* One cold `#[inline(never)] fn poisoned() -> !` keeps the hot path
  branch-free; the wrapper is a single `match` per acquire.
* `RwLock::upgradable_read` is implemented as exclusive `write()`
  for now -- sound but lossy. Documented at the backend module
  level.
* `Arc`, `Weak`, atomics, `Condvar`, `mpsc`, `OnceLock`, etc. are
  re-exported from `std::sync` unchanged.

## parking_lot as the production default

Adds a `parking_lot` Cargo feature (enabled by default) and
`concurrency/src/sync/parking_lot_backend.rs` -- a zero-cost
re-export of `parking_lot::{Mutex, RwLock}` plus the std re-exports
for everything `parking_lot` doesn't ship.

* Backend routing in `concurrency/src/sync/mod.rs` picks
  `parking_lot_backend` when `feature = "parking_lot"` is on, falls
  back to `std_backend` otherwise.
* `_strict_provenance` feature overrides `parking_lot` and routes
  through `std_backend`: `parking_lot_core::word_lock` uses
  integer-to-pointer casts that `-Zmiri-strict-provenance` rejects,
  and the CI miri job (`--features=_strict_provenance`) exercises
  the fallback slot, which needs `std::sync` underneath.
* Workspace `Cargo.toml` gets a `tokio` parking_lot override entry
  so the `tokio` parking_lot feature is enabled workspace-wide.

## shuttle backend

`concurrency/src/sync/shuttle_backend.rs` mirrors `std_backend.rs`:
a thin poison-as-panic wrap around `shuttle::sync::{Mutex, RwLock}`
returning naked guards, plus re-exports for `Arc` / `Weak` / atomics
/ `Condvar` / `mpsc` that pass through unchanged. `OnceLock` falls
back to `std::sync` because shuttle does not ship one (it is
pure-std machinery and does not need model-checker integration).

Three feature flags share one `dep:shuttle`, arranged as a chain
(`shuttle_dfs` -> `shuttle_pct` -> `shuttle`):

  * `shuttle`     -- random scheduler (default first choice).
  * `shuttle_pct` -- PCT, biases toward rare interleavings.
  * `shuttle_dfs` -- DFS, exhaustive small-state exploration.

Same backend, scheduler chosen at runtime (by `concurrency::stress`,
added in the follow-on PR). The chain means a single `feature =
"shuttle"` cfg check is true under every variant, so the routing in
`sync/mod.rs` and the macro gates collapse to single-feature checks
(no `any(shuttle, shuttle_pct, shuttle_dfs)` chains).

## loom backend

`concurrency/src/sync/loom_backend.rs` mirrors the std and shuttle
wrappers: poison-as-panic around `loom::sync::{Mutex, RwLock}`
returning naked guards, with `LockResult` / `TryLockError` /
`OnceLock` etc. re-exported from `std::sync` because loom does not
ship them.

Loom 0.7 has no `Weak<T>` and no `Arc::downgrade`. The backend ships
a local `Arc<T>` wrapper that adds `downgrade` plus a `Weak<T>` shim
that holds a *strong* clone of the inner `loom::sync::Arc` until the
`Weak` itself drops. Documented quirks (`Weak::upgrade` always
succeeds after a `downgrade`, `Arc::weak_count` panics rather than
silently returning 0, last `Arc` drop does not free when a `Weak`
is live) are explicit at the module level -- they bound what loom
can model, and silent fallbacks would let assertions pass for the
wrong reason on every backend.

`sync/mod.rs` routes the loom feature ahead of every other backend.

## Workspace transition

Sweep the data-path crates (`flow-entry`, `flow-filter`, `nat`,
`net`, `pipeline`, `routing`, `stats`) to consume
`dataplane_concurrency::sync::{Arc, Mutex, RwLock, atomic, ...}`
instead of `std::sync::*` / `parking_lot::*` directly.

* Call sites drop the `.unwrap()` / `.expect("poisoned")` noise the
  `LockResult`-shaped surface forced, since the facade returns
  naked guards. Same shape as the existing `parking_lot::Mutex`
  consumers.
* `concurrency_mode(std)` annotations at QSBR-touching test sites
  in `nat::stateful::apalloc::test_alloc`,
  `routing::fib::test`, and `flow-entry::flow_table::table::tests`
  keep the existing std-only feature gating intact.
* `nat::portfw::flow_state::get_packet_port_fw_state`
  simplification: the explicit `drop(guard)` dance before `debug!`
  calls is removed (per the recorded author intent that it isn't
  worth the code complexity); the guard now drops implicitly at
  the end of scope.

Mechanical-by-design -- the sweep is mostly `use` rewrites and
`.unwrap()` removal. The semantic change is one step: every
consumer now gets the facade-routed primitives, so flipping the
loom/shuttle features at the top of the workspace is now
meaningful. The follow-on PR adds `#[concurrency::test]`, the
`stress` dispatcher, and the `thread::scope` shim that let a
single test source file run under any backend.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/concurrency-stack/4-workspace-transition branch from 8d86891 to 0364eb1 Compare May 15, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:+cross run cross compile jobs ci:+vlab Enable VLAB tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant