Experimental#91
Open
doubleailes wants to merge 14 commits into
Open
Conversation
Captures the design discussion on the `experimental` branch: a Rust crate (`rer-package`) that extracts the four solver-relevant fields (`name`, `version`, `requires`, `variants`) from a `package.py` without invoking the Python interpreter, intended to compose with the `load_family` lazy-discovery hook from #86. Not implementation — explicitly marked status RFC. Lays out: - Scope (what the fast path accepts, what it bails on); the bias- hard-toward-bailing policy that protects against silent correctness regressions. - Three-stage build order with Stage 1 (a corpus-survey tool, `rer-stat-package-py`) as the go/no-go gate. Two days of work to produce the static-vs-dynamic fraction on a real studio repo, which decides whether Stages 2–3 happen. - Honest forecast: 2–3× on warm-cache discovery, 1.3–1.5× on cold-cache CIFS, 10–50× on the parse phase for CI/batch loads — all contingent on Stage 1's number and on whether the parsed result is also cached in Fortiche's shared memcache. - Risk table covering the silent-divergence scenario, with the differential-test harness mirroring the 188-case rez solver gate as the safety net. - Considered alternatives: a parsed-package cache layer on the shared memcache, named explicitly so Stage 1's results can pivot us toward it if the static-parseable fraction is low. Lives in the engineering docs section alongside `dedup-caches.md` and `stability.md`. On the `experimental` branch it stays out of the published Zola site until the work merges. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Lands two pieces:
1. `scripts/survey_package_py.py` — the Stage 1 classifier from the
RFC. Walks a rez repo, parses each `package.py` with stdlib
`ast`, classifies each file as fast-parseable or names the
reason it bails (`dynamic-requires`, `top-level-with`, `imports`,
etc.). Pure stdlib, no rer / rez / rustpython-parser dep.
Includes the key extension that emerged from running it against
Fortiche: `with scope("config") as config: ...` — rez's
declarative DSL for non-solver config — is recognised as
ignorable, mirroring the way `def commands(...)` is. The
classifier still walks the with-body defensively to ensure no
solver field is rebound inside.
2. RFC update (`docs/content/docs/engineering/fast-package-py-parser.md`)
with the Stage 1 result table.
## Stage 1 result on /thierry/rez/pkg
6,439 package.py files surveyed. **92.9% fast-parseable.**
Non-fast-parseable breakdown (overlapping reasons, % of total):
dynamic-requires (@early/@late): 5.5%
imports: 1.5%
missing-version (test fixtures): 1.3%
missing-name (test fixtures): 1.2%
top-level-classdef: 0.8%
unrecognised-raise: 0.0%
The decisive finding: a *naive* survey would report only 58.6%
fast-parseable, with 34.9% of the corpus matching `top-level-with`.
Sub-classifying those 2,245 files showed **100% of them** are
`with scope("config")` — every single one. Extending the
classifier to recognise the DSL bumps the accept rate from 58.6%
to 92.9% with no false positives.
Well past the 70% **PROCEED** threshold the RFC set. Stage 2
(the Rust parser) is justified — the next commit scaffolds the
`rer-package` crate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New crate `rer-package` with `parse_static_package_py(source) ->
Option<PackageInfo>` — extracts the four solver-relevant fields
(`name`, `version`, `requires`, `variants`) from a rez `package.py`
without invoking the Python interpreter. Uses
`rustpython-parser` 0.4 for the AST; walks top-level statements and
applies the classification rules the RFC's Stage 1 survey validated.
Accepted at module scope:
- `name = "..."` / `version = "..."` — string literals
- `requires = ["...", ...]` — list or tuple of string literals
- `variants = [["..."], ...]` — list/tuple of list/tuple of strings
- `def commands(...)` / `def pre_commands(...)` / ... — body
ignored, except `@early`/`@late`-decorated function with name
in solver fields bails
- `with scope("config") as config: ...` — rez's declarative DSL,
body ignored; walked defensively to catch a pathological body
that rebinds a solver field
- Module docstring, assignments to non-solver fields (description,
authors, tools, tests, timestamp, hashed_variants, etc.)
Bails to None (caller falls back to rez):
- `@early` / `@late` on a solver-field function
- Top-level `if` / `for` / `while` / `try` / `match`
- `import` / `from … import`
- `class ...:`
- `raise ...`
- Non-literal RHS for a solver field (function call, BinOp, ...)
- `name +=`, `requires +=` — augmented assignment on solver fields
- Missing `name` or `version`
- Bare module-level expression statement
- Syntax error
The bias is hard toward bailing. A false positive is a silent
correctness regression vs rez; a false negative just means we use
the slow path for that one file. The slow path is already there.
20 inline unit tests in `lib.rs`:
- Happy path: minimal, full, with-scope, docstring, unknown fields,
empty requires, tuple-of-tuples variants
- Bail cases: @early on requires, @late on variants, top-level if,
import, from-import, classdef, non-scope with, scope-with that
shadows a solver field, non-literal name, function-call requires,
missing name, missing version, syntax error
Plus `tests/test_corpus.rs` — an `#[ignore]`d integration test
that walks `RER_CORPUS_PATH`, parses every `package.py`, and
reports the accept rate. Optional `RER_CORPUS_REQUIRE` env var
gates on a minimum percentage. Mirrors the Python survey's
traversal (same dot/underscore/variant-hash directory skips).
Both tools run against `/thierry/rez/pkg` (the Fortiche-on-CIFS
rez repo):
- Python survey (`scripts/survey_package_py.py`): 5982 / 6439 (92.9%)
- Rust parser (`cargo test ... test_corpus`): 5985 / 6439 (92.9%)
3 files of drift, well below the rounding bucket — the two
classifiers agree on the green-light verdict. Walk + parse + check
took 54.5 s over the CIFS mount (most of that I/O; the parser
itself runs in microseconds per file).
Follow-up commits / PRs will add:
- A PyO3 binding on `pyrer` so the rez shim can call it directly
(`pyrer.parse_static_package_py(source) -> Optional[PackageData]`).
- A differential-test harness: for every file the Rust parser
accepts, also load via rez's `Package` and assert the four
fields match byte-for-byte. The Stage 2 safety net.
- Performance bench: per-file parse time vs rez's evaluator.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PyO3 binding for `rer_package::parse_static_package_py`. Gives the
rez-integration shim a one-call fast-path that turns a raw
`package.py` source string into a `pyrer.PackageData` without
invoking the Python interpreter for the static majority (~93% at
Fortiche per the Stage 1 survey on `experimental`).
## API
```python
import pyrer
with open("package.py") as f:
source = f.read()
pd = pyrer.parse_static_package_py(source)
if pd is None:
# Dynamic file: @early / @late / imports / top-level if.
# Fall back to rez's evaluator.
pd = pyrer.PackageData.from_rez(rez_pkg)
```
`None` is not an error — it's a signal to use the slow path. No
exception escapes pyrer; even a syntax error returns `None` (the
caller is going to send the same file through rez anyway, which
will produce its own error if there is one).
## Tests
7 new tests in `tests/test_rich_api.py`:
- happy paths: minimal, full-static, with-scope DSL
- bails: @early requires, top-level import, syntax error
- end-to-end: parse → use in `pyrer.solve()` → resolves identically
to a hand-constructed `PackageData`
All 94 Python tests pass (was 87 + 7 new).
## Docs
Updates the `load_family` worked example in
`docs/content/docs/getting-started/rez-integration.md` to show the
two-tier load: try `parse_static_package_py(open(pkg.filepath).read())`
first, fall back to `from_rez(pkg)` on `None`. This is the actual
shape the Fortiche shim should adopt.
## Wiring
- `crates/rer-python/Cargo.toml`: depends on `rer-package` (already
a workspace member from the previous commit).
- `crates/rer-python/src/lib.rs`: one `#[pyfunction]` registered on
the module. The conversion from `rer_package::PackageInfo` to
`pyrer.PackageData` is a four-field move — same shapes, no
allocations beyond the `PackageData` PyClass instance itself.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`scripts/bench_package_py_parser.py` — `timeit`-based micro-bench that measures the Rust AST parser against rez 3.3.0's `from_rez` path. Supports --corpus (samples real package.py files), --with-rez (times real `DeveloperPackage.from_path` for an apples-to-apples full-load comparison), and a synthetic mode for environments without rez installed. ## Stage 3 result on /thierry/rez/pkg + rez 3.3.0 Apples-to-apples full-load (what the rez-integration shim pays): open + read + parse_static_package_py 1,533 μs / file DeveloperPackage.from_path + from_rez 2,632 μs / file ----------------------------------------- --------------- Speedup 1.7× Per-file in-memory breakdown: parse_static_package_py(source) 1,990 μs (Rust AST parse) from_rez(fake_pkg, lower bound) 12.7 μs (attribute walk) from_rez(real rez Package, preloaded) 15.0 μs (post-load walk) open+read alone 18.7 μs The Rust parser saves ~1.1 ms per file. Over a 50-family resolve, that's ~55 ms saved. Real and useful but smaller than the 20–50× the original RFC predicted. ## Why smaller than predicted The RFC assumed the comparison was vs `compile + exec` of arbitrary Python. It isn't: rez 3.3.0's `DeveloperPackage.from_path` is already doing something fairly fast (~2.6 ms full load), so the headroom on top of it is more modest. `rustpython-parser` is now the bottleneck of its OWN path (~2 ms / file). It builds a full Python AST even though we only read four fields. Replacing it with a hand-rolled lexer (or `ruff_python_parser` if a usable crates.io publish exists) should get us to 50–200 μs/file — a further 10× on top of the 1.7×. ## RFC updated The "Honest forecast" section in `docs/content/docs/engineering/fast-package-py-parser.md` is replaced with the actual Stage 3 numbers, plus a "Where the next 10× lives" section naming the two follow-up experiments (ruff_python_parser spike, hand-rolled lexer). ## When this version is worth shipping as-is - If `load_family` hits 50+ families per resolve, the 55 ms/resolve saving is artist-perceptible. - If the shared memcache stores raw bytes (not parsed PackageData), this parser displaces work the memcache otherwise repeats. Otherwise, swap the parser library before shipping. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…34.8× win) Stage 3 V2. The original AST-based parser only delivered 1.7× over rez 3.3.0 on the Fortiche corpus — rustpython-parser was building the full module AST when we only needed four top-level fields, spending ~2 ms/file allocating nodes we threw away. This commit replaces that implementation with a 700-line hand-rolled scanner that walks module-level patterns directly. Same public API (`parse_static_package_py(source) -> Option<PackageInfo>`), same "bias hard toward bailing" policy, same set of accepted shapes — just no AST library underneath. Apples-to-apples full-load (open+read+parse vs DeveloperPackage.from_path+from_rez): V1 (rustpython-parser): 1,533 μs vs 2,632 μs → 1.7× V2 (hand-rolled): 75.24 μs vs 2,615.54 μs → **34.8×** Per-file saving: ~2.54 ms. Over a 50-family resolve (typical `load_family` shape from #86): ~127 ms saved per resolve. Parse step alone: V1: 1,990 μs / file V2: 59.23 μs / file (~33× on this layer) Corpus accept rate held at 92.9% (5,979 / 6,439 vs V1's 5,985 — 6 files of drift, well within rounding). The 30 unit tests covering happy paths and bail cases all carry over; 5 new tests added for cases the V1 AST parser handled implicitly: - CRLF line endings (Windows-edited Samba-served files) - `\<CRLF>` line continuation on non-solver assignments (`changelog = \` + CRLF + indented triple-quoted string) - Decorators on non-solver functions (`@deprecated def commands():`) - Multi-line list literals with comments interleaved - Single-quoted strings (corpus uses these heavily) Both Windows-specific bugs above surfaced during the rewrite — the V1 AST parser tolerated them transparently because rustpython-parser handles them. They're now explicit in V2. `rustpython-parser = "0.4"` is gone from both `rer-package` and the workspace deps. The whole `rer-package` crate now has zero non-stdlib dependencies. Build time and binary size both drop appreciably (rustpython-parser pulled in malachite, phf, unicode_names2, etc. — ~30 MB of transitive deps). V2's 75 μs/file splits as ~15 μs I/O (warm-cache CIFS) + ~60 μs CPU. CPU is no longer dominant; further parser optimisations have diminishing returns. The next big lever is either avoiding more I/O entirely (`load_family` from #86 already does most of this) or a parsed-package cache layered on Fortiche's existing shared memcache — both architectural moves, not parser-internal. `docs/content/docs/engineering/fast-package-py-parser.md` has the full V1 vs V2 comparison, the corpus accept rates, the CRLF diagnostic story, and a "Caveats on the 34.8× headline" section flagging cold-cache + sample-bias + no-production-A/B-yet. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… V2-accepted files
Adds `scripts/diff_against_rez.py` — the Stage 2 safety net from
the RFC. For every file `pyrer.parse_static_package_py` accepts,
the script also loads via rez's `DeveloperPackage.from_path`,
stringifies the four solver-relevant fields (`name`, `version`,
`requires`, `variants`), and compares byte-for-byte. Any divergence
is a release blocker — a silent correctness regression in any rez
integration shim using the fast path.
## Result on /thierry/rez/pkg + rez 3.3.0
Total package.py files surveyed: 6,439 (74.3 s)
V2 accepted: 5,979
V2 bailed (slow path): 460
Differential on V2-accepted (5,979 files):
Match (all four fields agree): 5,813 (97.22%)
Mismatch: 0 (0.00%) ← THE NUMBER
rez evaluation error: 166 (2.78%)
**Zero mismatches across the full Fortiche corpus.** The bias-toward-
bailing strategy is validated: V2 never produces output that
diverges from rez on a file rez can evaluate.
## On the 166 rez-eval-error files
Not a divergence — sampled five, all the same root cause:
InvalidPackageError: Package … uses @include decorator, but no
include path has been configured with the
'package_definition_python_path' setting.
That's a dev-venv rez config issue. Production rez at Fortiche has
the setting configured; rez would load these files fine. V2
correctly accepts them because `@include def some_func()` is a
non-solver decorator and the four solver fields are static. In
production they'd match.
## Script behaviour
- `python scripts/diff_against_rez.py /path/to/repo` — full run.
- `--max N` for a smoke test on N files (~50/sec on warm CIFS).
- `--show-mismatches N` to limit detailed output.
- `--csv mismatches.csv` to write per-mismatch rows for triage.
- Exit code 0 if no mismatches, 1 if any.
- Reports progress every 500 files (CIFS walks are slow enough
that a status line helps).
- Gracefully handles per-file rez evaluation errors — those go in
a separate bucket, not the mismatch count.
## RFC updated
`docs/content/docs/engineering/fast-package-py-parser.md` gets a
new "Stage 2 safety net" section with the result table and the
rez-eval-error explanation. The "Risks and mitigations" row for
silent correctness regression now points at this differential as
the concrete enforcement.
Stage 2 of the original RFC is now complete.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…arser Adds a "Plugging in the static package.py fast-parser" section to the rez-integration page covering the production rollout of `pyrer.parse_static_package_py` from `experimental`. Captures the recipe the Fortiche shim needs to land the V2 hand-rolled lexer safely: - Two-tier `load_family`: try Rust parser first, fall back to `from_rez(pkg)` on `None`. ~30 lines of shim code. - Shadow-validation mode (`REZ_PYRER_VALIDATE_PARSER=1`) — for every fast-path success, also load via rez and log any divergence. The offline differential ran 5,813/5,813 clean on the Fortiche corpus; the shadow mode is the production safety net for runtime patterns the corpus didn't exercise. - Hit-rate metrics — class-level counters that distinguish fast-path hit, non-py file, parser-bailed, and I/O error. - Four-step rollout plan with progressive user-percentage gates and a kill switch at each step. - Honest "where this WON'T help" list (Python startup, dynamic `@early`/`@late` packages, YAML, rxt-cached resolves, truly-cold CIFS I/O). Cross-links to the engineering RFC for the design context (the 1.7× → 34.8× V1→V2 spike story, the differential test result). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
…oses #92) The `load_family` callback now receives a `version_range` argument when its signature accepts one. The hint is a rez-syntax range string (`"2+<3"`, `"==2.0"`, `None` for unconstrained) that the shim can pass directly to `rez.packages.iter_packages(range_=...)` to skip on-disk version directories outside the current solver constraint. Issue #92 measured 95% load-fan-out waste in a real Fortiche resolve: 2,637 PackageData materialised so that 132 land in the solve. With the hint, the shim pre-filters to versions intersecting the solver's current constraint — projected ~400 loads instead of 2,637, a 6.6× cut to a path that accounts for 91% of resolve wall time on that workload. Detected at solve start via `inspect.signature`: def load_family(name): # legacy, 1-arg def load_family(name, version_range=None): # new, 2-arg def load_family(name, **kwargs): # **kwargs accepted All three keep working. The hint is passed as a keyword argument so the **kwargs form receives it transparently. No flag-day for existing shims. The repo tracks the loaded range per family. If a later request needs a wider range than the cached load covered (i.e. the solver backtracks and widens), the loader is re-called with the unioned range and the variant cache invalidates the stale `PackageVariantList`. In practice this is rare — most resolves narrow monotonically. Rust: - `FamilyLoader` signature: `Fn(&str, Option<&VersionRange>) -> Vec<...>` - `PackageRepo::get_family(name, hint)` — was `get_family(name)` - `PackageVariantList::new(ctx, name, hint)` — was `new(ctx, name)` - `PackageVariantCache::get_or_build(ctx, name, hint)` — was 2-arg - `PackageRepo` internal cache entry now tracks `loaded_range` Python: - `load_family` callback gains optional `version_range` kwarg - Detection via `inspect.signature`; legacy callbacks unchanged Rust: - test_loader_receives_version_range_hint — confirms the hint string is "2+<3" for `lib-2+<3` requests - test_loader_eager_seed_no_loader_call — pre-seeded families never invoke the loader even with a hint - All 44 existing solver tests + 2 new = 46/46 Python: - test_load_family_range_hint_passed_for_pinned_request - test_load_family_legacy_one_arg_callback_still_works - test_load_family_range_hint_string_format - test_load_family_kwargs_callback - 94 existing tests + 4 new = 98/98 `cargo test --release -p rer-resolver --test test_rez_benchmark -- --ignored`: **188/188** in 16.64 s. Solver behavior unchanged on the strict rez-faithfulness gate. `docs/content/docs/getting-started/rez-integration.md` updated: - `load_family` worked example now shows the 2-arg signature with `iter_packages(range_=...)`. - New "The version_range hint (issue #92) — pre-filter at the shim" section with the impact table and the advisory semantics. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First release candidate stacking everything on `experimental`: - rc.1 baseline: solver, load_family (#86), resolved_ephemerals (#84), from_strings (#88) - load_family `version_range` hint (#92) — just cherry-picked here - Static `package.py` Rust fast-parser: * Hand-rolled lexer at 75 μs/file full-load (34.8× vs rez DeveloperPackage.from_path + from_rez on Fortiche) * 92.9% accept rate on the 6,439-file Fortiche corpus * 0 mismatches vs rez on the differential harness - `pyrer.parse_static_package_py` PyO3 binding for the shim - Production integration guide + RFC + corpus survey tool + perf bench script + differential test harness The version_range hint and the static parser compound: the parser makes each load cheap; the hint makes most loads unnecessary. Four touchpoints, same as every previous rc bump: - Cargo.toml: workspace version + the three internal-dep pins (rer-version, rer-resolver, rer-package) - docs/config.toml: GitHub-pill version - docs/content/_index.md: homepage repo_version - docs/content/docs/getting-started/quick-start.md: Rust dep snippet Verifications: - cargo test --lib: 110/110 (30 rer-package + 46 rer-resolver + 34 rer-version) - pytest tests/: 105/105 (58 differential + 47 rich-api) - cargo build: clean Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…oses #94) Adds `parse_static_packages_py(paths) -> list[PackageInfo | None]` to `rer-package` and exposes it as `pyrer.parse_static_packages_py` on the Python side. The function reads + parses every path in one Rust call across a Rayon thread pool, with the GIL released for the duration via `Python::allow_threads`. ## Why Issue #94: after the static parser landed, cProfile shows the rez shim's per-resolve cost is no longer in the parser itself — it's in the serial Python loop of `open()` calls feeding the parser. On a 132-package Fortiche resolve, that's 3,809 file opens taking 3.20 s (35% of total wall time), one per call with everything but one core idle. `parse_static_packages_py` replaces that loop with a single batched Rust call. Same parse semantics per file, parallel I/O and parsing across cores. ## Result Measured on `/thierry/rez/pkg` (Fortiche on CIFS): - 500 files (warm cache): 56.71 ms → 40.76 ms (1.39× speedup) - 2000 files: 4234.41 ms → 1508.05 ms (2.81× speedup, 2.73 s saved) Both paths accept the same set of files (1864/2000 → static-parseable fraction matches `parse_static_package_py` on per-file calls). Per-file saving on the 2000-file bench: ~1.36 ms. Extrapolated to the issue's 132-package / 2,600-file resolve: ~3.5 s saved per resolve. The 500-file bench was bottlenecked on the warm-page-cache floor (the Rayon overhead amortizes less). At larger batch sizes the parallel I/O win shows through. ## API ```python pyrer.parse_static_packages_py(paths: list[str | os.PathLike]) -> list[PackageData | None] ``` - Output is **positionally aligned** with `paths`. Missing files, unreadable bytes, and parser-bails all produce `None` at the same index. - No exception ever escapes the call. - Pool size follows `RAYON_NUM_THREADS` (Rayon default = logical core count). No per-call knob; cap via env var on shared CI. - Pure addition. The single-file `parse_static_package_py` stays for callers that haven't been updated. ## Shim integration shape ```python def load_family(name, package_paths): pkgs, paths = [], [] for pkg in iter_packages(name, paths=package_paths): filepath = getattr(pkg, "filepath", None) if not filepath or not filepath.endswith(".py"): pkgs.append((pkg, None)) continue pkgs.append((pkg, filepath)) paths.append(filepath) pds = pyrer.parse_static_packages_py(paths) pds_iter = iter(pds) out = [] for pkg, filepath in pkgs: if filepath is None: out.append(pyrer.PackageData.from_rez(pkg)) continue pd = next(pds_iter) out.append(pd if pd is not None else pyrer.PackageData.from_rez(pkg)) return out ``` The shim feature-detects via `hasattr(pyrer, "parse_static_packages_py")` and falls back to the per-file loop on older pyrer. ## Tests Rust unit tests in `rer-package`: - batch_empty_returns_empty - batch_parses_each_file_independently (static + dynamic + missing) - batch_missing_file_becomes_none - batch_preserves_input_order (20 files, alternating static/dynamic) Python tests in `tests/test_rich_api.py`: - test_parse_static_packages_py_empty_input - test_parse_static_packages_py_each_file_independent (static + dynamic + missing + static, aligned) - test_parse_static_packages_py_preserves_order (20 alternating, par_iter ordering check) - test_parse_static_packages_py_accepts_pathlib_paths - test_parse_static_packages_py_drives_solve (batch → pyrer.solve E2E) - test_parse_static_packages_py_matches_single_file Plus `scripts/bench_batched_parser.py` for measuring against any rez repo. ## Verification - `cargo test --lib`: 34/34 (rer-package) + 46/46 (rer-resolver) + 34/34 (rer-version) = 114/114 - `pytest tests/`: 111/111 (was 105 + 6 new) - 188-case strict rez differential: 188/188 in 16.91 s - cargo build: clean ## Docs `docs/content/docs/getting-started/rez-integration.md` gets a new "Faster: batched parallel parse (issue #94)" subsection with the shim integration shape, semantics, and a capability-detect recommendation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Picks up the batched parallel `parse_static_packages_py` API (closes #94, this branch). On the Fortiche corpus the batched call delivered a 2.81× cut to the open+parse phase (4.23 s → 1.51 s on 2,000 files) over the serial Python loop the shim ran today, with no per-file correctness drift. Four touchpoints, same as every previous rc bump: - Cargo.toml: workspace version + the three internal-dep pins - docs/config.toml: GitHub-pill version - docs/content/_index.md: homepage repo_version - docs/content/docs/getting-started/quick-start.md: Rust dep snippet Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…94) Two doc surfaces updated: 1. `docs/content/docs/getting-started/rez-integration.md` — "Faster: batched parallel parse (issue #94)" section expanded from a sketch to a full production guide matching the depth of the single-file parser section. Adds: - "What it does" — full semantics (positional alignment, no escaped exceptions, GIL release, pool size). - "Measured impact on Fortiche" — table with the 1.39× / 2.81× numbers and the per-file saving. - "Integration" — complete two-tier load_family snippet wiring #86 + #92 + static parser + #94 together; the actual recipe a shim author can paste. - "Backward compatibility / feature detection" — `hasattr` pattern matching the rest of pyrer's optional APIs. - "Shadow-validation mode" — `REZ_PYRER_VALIDATE_BATCHED` recipe that reuses the from_rez(pkg) comparison from Stage 2. - "Metrics" — class-level counters for batched_hits / batched_misses_io / batched_misses_dynamic / non_py_packages so production rollouts can confirm hit rate. - "Rollout plan" — 4-week table with progressive user-percentage gates, mirroring the parser rollout shape. - "Where this WON'T help" — honest caveat list (tiny resolves, dynamic 7%, load_family cache hits, cross-invocation cost). 2. `docs/content/docs/engineering/fast-package-py-parser.md` — new "Stage 4 — Batched parallel parse (issue #94)" section slotted before "Considered alternatives". Captures the design decisions, the result table, and the safety-net carry-over from Stage 2 (per-file semantics are identical so the differential coverage transfers byte-for-byte). Top-of-doc status banner updated to "Stages 1–4 shipped" with all the relevant script paths. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… bisect tool (#96) Issue #96 reported pyrer 1.0.0-rc.3 diverging from rez on a real ~5,500-package Fortiche corpus: one hard failure (`fastoche_intents_maya` with implicits) and ~30% divergence on spot-checked resolves without implicits. I couldn't reproduce either symptom from pyrer's side. With the recommended shim shape from `docs/content/docs/getting-started/ rez-integration.md` (static parser + `load_family` + `version_range` hint + `from_rez` fallback), against the same `/thierry/rez/pkg/{ext,int}` corpus: - `fastoche_intents_maya` (no implicits): rer 132 / rez 132, **0 packages differ** - `fastoche_intents_maya` with `~platform==linux ~arch==x86_64 ~os==debian-12` inlined into the request: rer 132 / rez 132, **0 packages differ** - `gymtonic` (no implicits): rer 94 / rez 94, **0 packages differ** - 30-family random sample (seed=42): **30/30 identical** Verdict: the reported divergence is downstream-shim-specific. `pyrer.solve()` called directly with the recommended shim shape matches `rez.ResolvedContext` byte-for-byte on this corpus. To make the bisect path obvious for the downstream user — and turn this into a permanent regression check — this commit ships `scripts/compare_resolves.py`. It uses the same recommended shim shape internally and reports per-request: ✓ identical, ≠ divergent, x rer-fail, X rez-fail, · both-fail. Usage: python scripts/compare_resolves.py /path/to/rez/pkg --n 30 python scripts/compare_resolves.py /path/to/rez/pkg \ --request fastoche_intents_maya gymtonic Exit code 0 iff every resolve matched, 1 otherwise. ## Docs `docs/content/docs/getting-started/rez-integration.md`'s existing "Sanity-checking against rez" section is replaced with the recipe + an "Interpreting the output" subsection that names the implication: 0 divergences from this script + shim divergence elsewhere = shim-translation bug; reproducible divergence from this script = pyrer correctness bug + release blocker. ## What's still needed for #96 itself This commit doesn't *fix* the divergence the user is seeing — because I can't see the divergence the user is seeing. The fix lives in the downstream `claude/rer-experimental` shim. The script gives the user a tool to bisect their shim against known-good pyrer behaviour. Next step is theirs: run the script against their corpus, confirm 0 divergences on the pyrer side, then turn on the shim's features (filters, orderers, custom `from_rez` wiring, cache layers, `add_implicit_packages` handling) one at a time and find which one introduces the divergence. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.