Skip to content

feat(zeam): thread --chain-worker flag from ZEAM_CHAIN_WORKER env#170

Merged
ch4r10t33r merged 1 commit intomainfrom
feat/zclawz/chain-worker-flag
May 6, 2026
Merged

feat(zeam): thread --chain-worker flag from ZEAM_CHAIN_WORKER env#170
ch4r10t33r merged 1 commit intomainfrom
feat/zclawz/chain-worker-flag

Conversation

@zclawz
Copy link
Copy Markdown
Contributor

@zclawz zclawz commented May 6, 2026

Adds an env-gated --chain-worker {on|off} flag for the slice c-2c part 2 burn-in on devnet4 (zeam #803, #828). Threaded through both deployment paths so the live ansible-driven devnet4 playbooks pick it up:

  1. client-cmds/zeam-cmd.sh — reads ZEAM_CHAIN_WORKER env, validates against {on, off, empty}, appends --chain-worker <val> to node_binary and node_docker only when explicitly set. Used by spin-node.sh / any script-driven run.
  2. ansible/roles/zeam/{defaults,tasks}/main.yml — new zeam_chain_worker ansible variable (empty default), threaded into the docker run templated command. Without this, the script-side change wouldn't take effect on the actual devnet4 ansible deployment because the role re-derives the docker run line itself rather than sourcing the script. (Caught by @ch4r10t33r reviewing this PR.)

Default behavior: strict no-op

ZEAM_CHAIN_WORKER (script) and zeam_chain_worker (ansible) both default to empty → no flag emitted. This PR is a strict no-op against any zeam build, including the currently-published blockblaz/zeam:devnet4 (v0.4.13, pre-c-1) which does not recognise --chain-worker at all. Land safely; flip-on later via env / -e / group_vars.

How to enable burn-in

Once a zeam build with chain-worker support is published as blockblaz/zeam:devnet4 (target: v0.4.14, zeam release PR):

# via script:
export ZEAM_CHAIN_WORKER=on
./spin-node.sh ...

# via ansible:
ansible-playbook ... -e zeam_chain_worker=on
# or set in inventory group_vars/zeam.yml:
#   zeam_chain_worker: "on"

To explicitly run the legacy synchronous path on a c-1+ build (kill-switch): substitute off for on above.

Burn-in target (per zeam #803 c-2c plan)

  • ≥24h with --chain-worker=on
  • Watch zeam_lock_hold_seconds{lock="states", site="onBlock.commit"} p99 — should drop dramatically vs slice (b) baseline (rwlock no longer load-bearing under chain-worker single-writer regime).
  • Watch lean_chain_state_refcount_distribution — typical=1, occasional 2-4, never >16 (entries stuck >16 indicate a leaked reader acquire).
  • Watch lean_chain_queue_dropped_total — should be 0 under nominal load; non-zero means producer/consumer mismatch worth investigating.

Validation

$ bash -n client-cmds/zeam-cmd.sh && echo OK
OK

# Default (no env): no --chain-worker flag in node cmd line ✅
# ZEAM_CHAIN_WORKER=on:  --chain-worker on appended ✅
# ZEAM_CHAIN_WORKER=off: --chain-worker off appended ✅
# ZEAM_CHAIN_WORKER=hello: WARN logged, no flag emitted ✅

Ansible side is parallel to the existing --db-backend plumbing (same template idiom: {{ ('--flag ' + var) if (var | length > 0) else '' }}).

Followups

After burn-in clean, two separate PRs:

  1. zeam side: flip the compiled-in default of --chain-worker from offon (one-line, in zeam repo).
  2. lean-quickstart side: can drop the env-gating from this file once the zeam default flips.

Base automatically changed from devnet4 to main May 6, 2026 16:14
@zclawz zclawz force-pushed the feat/zclawz/chain-worker-flag branch from 576883d to 52a577a Compare May 6, 2026 16:16
Copy link
Copy Markdown
Contributor

@ch4r10t33r ch4r10t33r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are needed in ansible/roles as well

@zclawz zclawz force-pushed the feat/zclawz/chain-worker-flag branch from 52a577a to 0b26232 Compare May 6, 2026 16:24
Adds an env-gated --chain-worker {on|off} flag for the slice c-2c part 2
burn-in on devnet4 (zeam #803). Threaded through both deployment paths:

  1. client-cmds/zeam-cmd.sh: reads ZEAM_CHAIN_WORKER env, validates
     against {on, off, empty}, appends --chain-worker <val> to
     node_binary and node_docker only when explicitly set. Used by
     spin-node.sh and any direct script-driven runs.

  2. ansible/roles/zeam (defaults + tasks): new zeam_chain_worker
     ansible variable (empty default), threaded into the docker run
     templated command so the live devnet4 ansible playbooks can pass
     --chain-worker via -e zeam_chain_worker=on or via inventory
     group_vars. Without this, the script-side change wouldn't take
     effect on the actual devnet4 deployment because the role
     re-derives the docker run line itself rather than sourcing the
     script. (Noted by @ch4r10t33r reviewing PR #170.)

Default empty in BOTH paths: no flag is emitted unless explicitly set,
so this PR is a strict no-op against any zeam build -- including the
currently-published blockblaz/zeam:devnet4 (v0.4.13, pre-c-1) which
does not recognise --chain-worker at all.

To enable the burn-in once v0.4.14+ is published as blockblaz/zeam:devnet4:

  # via script:
  export ZEAM_CHAIN_WORKER=on
  ./spin-node.sh ...

  # via ansible:
  ansible-playbook ... -e zeam_chain_worker=on
  # or set zeam_chain_worker: "on" in group_vars

To explicitly run the legacy synchronous path on a c-1+ build (kill-switch):
substitute 'off' for 'on' above.

Burn-in target per the c-2c plan in zeam #803:
  * >=24h with --chain-worker=on
  * watch zeam_lock_hold_seconds{lock="states", site="onBlock.commit"} p99
    (should drop dramatically vs slice (b) baseline)
  * watch lean_chain_state_refcount_distribution (typical=1, never >16)
  * watch lean_chain_queue_dropped_total (should be 0 under nominal load)

Verified bash -n green on the script and tested all four script
branches (unset, on, off, invalid) emit the expected node command line.
@zclawz zclawz force-pushed the feat/zclawz/chain-worker-flag branch from 0b26232 to 70544a9 Compare May 6, 2026 16:25
Copy link
Copy Markdown
Contributor

@ch4r10t33r ch4r10t33r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ch4r10t33r ch4r10t33r merged commit c2b557d into main May 6, 2026
4 checks passed
@ch4r10t33r ch4r10t33r deleted the feat/zclawz/chain-worker-flag branch May 6, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants