phase 3 day 1: RLV harness skeleton + project doc + integration gate

Project documentation:
- docs/phase3_rlv_challenge.md (canonical source of truth, 350+ lines)
  - Problem framing (RAG silent hallucination + long-context cliff)
  - 5-stage architecture mapped from human cognitive retrieval pattern
  - Why this is feasible *now* with quant.cpp specifically
  - Day 1-7 plan with Karpathy gates per day
  - Files-and-directory layout
  - Reading order if a future Claude Code session loads this project
  - Day 1 Karpathy log section
- ~/.claude/projects/.../memory/project_phase3_rlv.md (memory entry)
- MEMORY.md updated with Phase 3 RLV as the top item

Harness skeleton (bench/rlv/):
- README.md pointing back to the project doc
- rlv_orchestrator.py: 5-stage flow controller, end-to-end answer_question
- stages/_llm.py: shared HTTP client for quant-server
  - start_server() / stop_server() lifecycle
  - check_cliff_budget() enforcing the cliff invariant
  - llm_call() via /v1/chat/completions with tolerant system prompt
- stages/gist.py: Stage 1 — chunked summarisation (~500 char chunks)
- stages/locator.py: Stage 2 — outline + question -> chunk pointer
- stages/lookup.py: Stage 3 — region + question -> answer (minimal prompt)
- stages/verifier.py: Stage 4 — gist + answer -> {confident, unsure, contradicted}
- stages/researcher.py: Stage 5 — retry with different region (max 3)
- tests/smoke_test.py: D1 gate — orchestrator on a 4-section synthetic doc

D1 gate result:
- Integration ✅: pipeline runs end-to-end without crashing
- Accuracy ❌: smoke test picks wrong entity due to gist summaries being
  too vague for the locator to discriminate. The model picks the first
  entity mentioned in the read region (Maria Santos / CEO) instead of
  the question target (John Williams / CFO). This is exactly the Phase
  2B primacy-bias failure mode — and it's what RLV is supposed to fix
  by isolating chunks to single-section reads.

Lessons surfaced and embedded in code comments:
- Subprocess stdout/stderr need stderr=STDOUT to merge in bash 2>&1
  order so the model output parser can find the --- delimiters.
- Llama-3.2-3B-Q4 in chat mode emits "## Step 1: ..." reasoning chains
  unless given a short, direct system prompt. Fighting structured
  formats (TOPICS:/CHUNK:/VERDICT:) is counterproductive — use direct
  natural-language questions and tolerant parsers instead.
- Server-based architecture (quant-server) is mandatory: per-call
  subprocess start = ~50s model reload overhead = 5 min per question
  for a 5-stage pipeline. With the server it's ~10s per call.
- Primacy bias kicks in at sub-cliff sizes too. Chunking even small
  docs to ~500 chars is necessary for RLV's locator to have anything
  to choose between.

D2 plan (next): pivot the locator to use first-100-char chunk snippets
as the index instead of model-written summaries. Direct text extraction
beats LLM summarization for the indexing signal that the locator needs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: unamedkr

bench/rlv/README.md

@@ -0,0 +1,38 @@
+# RLV — Read-Locate-Verify document QA
+
+A 5-stage human-cognition-inspired document QA architecture built on top of `quant.cpp`. The challenge, motivation, architecture, and project plan are in **[`docs/phase3_rlv_challenge.md`](../../docs/phase3_rlv_challenge.md)** at the repo root — read that first if you've never seen this work.
+
+## Quickstart
+
+```bash
+# From the repo root
+python3 bench/rlv/rlv_orchestrator.py \
+    --doc bench/data/wikitext2_test.txt \
+    --question "Who is Robert Boulter?" \
+    --model models/Llama-3.2-3B-Instruct-Q8_0.gguf
+```
+
+## Layout
+
+```
+bench/rlv/
+├── README.md                    # this file
+├── rlv_orchestrator.py          # main entry point
+├── stages/
+│   ├── __init__.py
+│   ├── gist.py                  # Stage 1: chunked summarisation → outline
+│   ├── locator.py               # Stage 2: outline + question → region pointer
+│   ├── lookup.py                # Stage 3: region.kv + question → answer
+│   ├── verifier.py              # Stage 4: gist + answer → verdict
+│   └── researcher.py            # Stage 5: retry with a different region
+├── prompts/                     # template prompts (gist/locator/lookup/verify)
+├── eval/
+│   ├── eval_acme.py             # D3: v0.12 Acme reproduction
+│   └── eval_stress.py           # D5: 8000-token stress test
+└── tests/
+    └── smoke_test.py
+```
+
+## Cliff invariant
+
+Every stage's prompt MUST be ≤ **1024 tokens** for Llama-3.2-3B-Q4 (the cliff measured in Phase 1B). The orchestrator enforces this in `_check_cliff_budget()`.

bench/rlv/rlv_orchestrator.py

@@ -0,0 +1,169 @@
+#!/usr/bin/env python3
+"""RLV (Read-Locate-Verify) document QA orchestrator.
+
+Implements the 5-stage architecture from docs/phase3_rlv_challenge.md:
+
+    Stage 1 GIST     — chunked summarisation pass → structured outline
+    Stage 2 LOCATOR  — outline + question → region pointer
+    Stage 3 LOOKUP   — region + question → tentative answer
+    Stage 4 VERIFY   — gist + answer → {confident, unsure, contradicted}
+    Stage 5 RESEARCH — retry with different region if verify fails
+    Stage 6 OUTPUT   — calibrated final answer (confident or explicit uncertainty)
+
+Cliff invariant (see docs/phase3_rlv_challenge.md §3.2): every stage's
+prompt must be ≤ 1024 tokens for Llama-3.2-3B-Q4. The harness enforces
+this through stages._llm.check_cliff_budget().
+
+Usage:
+    python3 bench/rlv/rlv_orchestrator.py \\
+        --doc bench/data/wikitext2_test.txt \\
+        --question "Who is Robert Boulter?"
+
+For evals see bench/rlv/eval/.
+"""
+import argparse
+import json
+import sys
+import time
+from dataclasses import asdict
+from pathlib import Path
+
+# Make 'stages' importable when running from anywhere
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from stages import gist as gist_stage
+from stages import locator as locator_stage
+from stages import lookup as lookup_stage
+from stages import verifier as verifier_stage
+from stages import researcher as researcher_stage
+
+
+def answer_question(
+    doc_text: str,
+    question: str,
+    *,
+    doc_id: str = "doc",
+    cached_gist: gist_stage.Gist = None,
+    verbose: bool = True,
+) -> dict:
+    """Run the full RLV pipeline. Returns a dict with the final answer
+    and per-stage diagnostic info."""
+    t_start = time.time()
+    timings = {}
+
+    # Stage 1: GIST (or use cached one)
+    t0 = time.time()
+    if cached_gist is not None:
+        gist = cached_gist
+        if verbose:
+            print(f"[stage 1] using cached gist ({len(gist.chunks)} chunks)")
+    else:
+        if verbose:
+            print(f"[stage 1] building gist for doc_id={doc_id}, len={len(doc_text)} chars")
+        gist = gist_stage.build_gist(doc_text, doc_id=doc_id, verbose=verbose)
+    timings["stage1_gist"] = time.time() - t0
+
+    # Stage 2: LOCATOR
+    t0 = time.time()
+    if verbose:
+        print(f"[stage 2] locating question: {question!r}")
+    region = locator_stage.locate(question, gist, verbose=verbose)
+    timings["stage2_locator"] = time.time() - t0
+    if verbose:
+        print(f"[stage 2] -> chunk {region.chunk_id} (confidence={region.confidence})")
+
+    # Stage 3: LOOKUP
+    t0 = time.time()
+    if verbose:
+        print(f"[stage 3] reading chunk {region.chunk_id}")
+    look = lookup_stage.lookup(question, region, doc_text, verbose=verbose)
+    timings["stage3_lookup"] = time.time() - t0
+    if verbose:
+        print(f"[stage 3] -> answer: {look.answer[:80]!r}")
+
+    # Stage 4: VERIFY
+    t0 = time.time()
+    if verbose:
+        print(f"[stage 4] verifying against gist")
+    ver = verifier_stage.verify(question, look.answer, gist, verbose=verbose)
+    timings["stage4_verifier"] = time.time() - t0
+    if verbose:
+        print(f"[stage 4] -> verdict: {ver.verdict} ({ver.reason})")
+
+    # Stage 5: RESEARCH (only if verify failed)
+    t0 = time.time()
+    research = researcher_stage.research(
+        question, look, ver, gist, doc_text, verbose=verbose,
+    )
+    timings["stage5_research"] = time.time() - t0
+
+    # Stage 6: OUTPUT — format the final answer based on the verdict
+    if research.final_verdict == "CONFIDENT":
+        final_text = research.final_answer
+        confidence = "high"
+    elif research.final_verdict == "EXHAUSTED":
+        final_text = (
+            f"I'm not fully confident in any answer to your question. The closest "
+            f"information I found is: {research.final_answer}"
+        )
+        confidence = "low"
+    else:
+        final_text = research.final_answer
+        confidence = "medium"
+
+    timings["total"] = time.time() - t_start
+
+    return {
+        "question": question,
+        "final_answer": final_text,
+        "confidence": confidence,
+        "research": {
+            "verdict": research.final_verdict,
+            "n_retries": research.n_retries,
+            "attempts": research.attempts,
+        },
+        "timings": timings,
+        "gist_n_chunks": len(gist.chunks),
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--doc", required=True, type=Path,
+                        help="Path to the document text file")
+    parser.add_argument("--question", required=True, type=str,
+                        help="The question to answer")
+    parser.add_argument("--doc-id", default=None, type=str)
+    parser.add_argument("--quiet", action="store_true",
+                        help="Suppress per-stage diagnostics")
+    parser.add_argument("--json", action="store_true",
+                        help="Output JSON instead of human text")
+    args = parser.parse_args()
+
+    doc_text = args.doc.read_text(encoding="utf-8", errors="replace")
+    doc_id = args.doc_id or args.doc.stem
+
+    result = answer_question(
+        doc_text, args.question,
+        doc_id=doc_id, verbose=not args.quiet,
+    )
+
+    if args.json:
+        print(json.dumps(result, indent=2, default=str))
+    else:
+        print("\n" + "=" * 70)
+        print(f"QUESTION:    {result['question']}")
+        print(f"ANSWER:      {result['final_answer']}")
+        print(f"CONFIDENCE:  {result['confidence']}")
+        print(f"VERDICT:     {result['research']['verdict']}")
+        print(f"RETRIES:     {result['research']['n_retries']}")
+        print(f"GIST CHUNKS: {result['gist_n_chunks']}")
+        print(f"TOTAL TIME:  {result['timings']['total']:.1f}s")
+        print("  " + " | ".join(
+            f"{k}={v:.1f}s" for k, v in result["timings"].items() if k != "total"
+        ))
+        print("=" * 70)
+
+
+if __name__ == "__main__":
+    main()

bench/rlv/stages/__init__.py

@@ -0,0 +1,16 @@
+"""RLV stages — see docs/phase3_rlv_challenge.md §3.1 for the full architecture.
+
+Stage layering (each stage depends on the previous):
+    gist     → produces a structured outline (Stage 1)
+    locator  → outline + question → region pointer (Stage 2)
+    lookup   → region + question → tentative answer (Stage 3)
+    verifier → gist + answer → verdict (Stage 4)
+    researcher → retry locator with a different region (Stage 5)
+"""
+from . import gist
+from . import locator
+from . import lookup
+from . import verifier
+from . import researcher
+
+__all__ = ["gist", "locator", "lookup", "verifier", "researcher"]