BigQuery Agent Analytics Roadmap

[caohy1988]

BigQuery Agent Analytics — Roadmap

A tiered roadmap built from a survey of the 19 open issues on this repo, the
22 sections of the current SDK.md user manual, the 38 Python modules in
src/bigquery_agent_analytics/, and the ADK plugin
(google/adk/plugins/bigquery_agent_analytics_plugin.py, ~3,500 LOC, 14
lifecycle callback hooks, BigQuery Storage Write API path with GCS offload).

Filed for discussion. Estimates are calibrated to a single experienced
SDK / plugin engineer; multiply for parallel streams. Impact and effort
are best-effort; the implementer of each item will refine them.

Ground rules for this doc. Impact = H/M/L on a "downstream user
adoption + DevX uplift" axis, not revenue. Effort = engineer-weeks
(full-time equivalent), confidence band in parens. Items marked
strategic decision pending need a maintainer call before
sequencing.

---

Director summary

For an exec audience, the actionable shape is:

• Next 30 days (1 engineer): P0 (~2 eng-weeks) plus Quality Scorecard
  Phase 1 (~2 eng-weeks). That's the realistic month. Don't promise
  more on a single track.
• Next quarter (1 engineer): P0 + most/all of P1 (~15 eng-weeks
  total). No P2 unless P1 scope is cut. If a strategic bet
  matters more than completing P1, the realistic trade is to drop
  one P1 item (typically inheritance compilation) and pull
  ReasoningBank up — but pick one, not both.
• Next quarter (2 engineers in parallel): ship all P0/P1 work and
  fund ReasoningBank as the first strategic P2 bet — it converts
  the existing agent_improvement_cycle demo into a real product
  surface and has lower execution risk than ontology consolidation.
  Defer bigquery_ontology migration (Migrate SDK ontology pipeline to bigquery_ontology package #38) until its ADR is approved;
  treat it as the next quarter's strategic bet, not this one.
• Always P3: auto-skills, advanced resolvers, Spanner backends,
  V5 context graph. Research arm or partner-team work; don't block
  production roadmap on them.

Reality check on calendar math: P0 ≈ 2 eng-weeks, P1 ≈ 13 eng-weeks,
P0+P1 ≈ 15 eng-weeks ≈ 1 quarter for 1 engineer, not 1 month.
Earlier drafts of the resource table had this wrong; the table below
is corrected.

---

TL;DR

• Three workstreams already mature: trace reconstruction, deterministic
  • LLM-judge + categorical evaluation, CLI gating. Don't re-prioritize them;
    keep them stable.
• One workstream is the next center of gravity: the self-improvement
  loop — quality scorecard (Proposal: Automated Agent Quality Scorecard #63), automated benchmarking (Data Science Features for BQ Agent Analytics #95), agent
  improvement cycle (already shipped as a demo), ReasoningBank (BigQuery Agent Analytics SDK — ReasoningBank #49). This
  is the user-facing differentiator over the next quarter.
• One workstream is high-leverage but quietly under-invested: plugin
  telemetry. The cache-hit-rate metric (Improve token efficiency in ADK via BigQuery Agent Analytics #32) is a one-week win that
  unlocks the cost-optimization narrative and ties directly into the
  existing CodeEvaluator surface.
• One workstream is over-extended: ontology / context graph (8 open
  issues across V5 Context Graph: TTL import, mixed extraction, temporal lineage — implementation design #12, feat: implement inheritance (extends) compilation support #30, Migrate SDK ontology pipeline to bigquery_ontology package #38, Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57, Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58, Epic: Compile-time code generation for structured trace extractors (scoped rework) #75, Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76, Epic: Address Remaining Ontology Platform Gaps (Live Agent Resolution, Advanced Resolvers, SHACL, Spanner/MAKO) #93). Multiple design
  proposals are gating implementation. Constraint: at most one ontology
  bet per quarter (see resource table below).
• One workstream needs a strategic decision before any code: the
  ontology pipeline migration to upstream bigquery_ontology (Migrate SDK ontology pipeline to bigquery_ontology package #38) —
  runtime contract migration, not a module swap; needs a go/no-go.

---

Method

How I built this:

1. Read each open issue's body + first 2-3 comments. Categorized by
   workstream.
2. Surveyed src/bigquery_agent_analytics/ (38 modules) and SDK.md
   (22 sections, ~1700 lines) to understand what already ships.
3. Read the ADK plugin to understand the upstream telemetry contract
   the SDK consumes.
4. Tiered each item by impact × effort, then sequenced for parallel
   workstreams.
5. Flagged items that need a strategic call before scoping (marked
   strategic decision pending).

---

Workstream map (where the open issues live)

Workstream	What ships today	Open issues
Plugin telemetry	14 lifecycle hooks → agent_events; GCS offload; PyArrow schema	#32
Trace consumption	Client.get_session_trace, list_traces, tree render	(none open — mature)
Evaluation	Code, LLM-as-Judge, categorical, trajectory, multi-trial, grader pipeline, eval suite, eval validator	#84
Self-improvement loop	agent_improvement_cycle example, quality_report.py	#63, #95, #49
Insights / drift / memory	client.insights(), drift detection, memory_service	#74
Ontology / context graph	V5 pipeline, gm CLI, OWL importer, DDL compiler, materializer, compile_concept_index	#12, #30, #38, #57, #58, #75, #76, #93
DevX / CLI / docs	bq-agent-sdk CLI, SDK.md, examples/, blog series	#10 (wishlist), #51, #53, #77, #82 (series)

---

P0 — ship in ≤2 weeks (low-risk, asked-for)

Item	Issue	Impact	Effort	Notes
Quote-escape feedback="..." snippet in evaluate --exit-code FAIL output	#84	L	0.25 wk (high)	One-line escape pass; add unit test with embedded " and \. Surfaced from blog #3 live capture.
Context cache hit-rate evaluator (SDK side; plugin already does the telemetry)	#32	H	1 wk (high)	Upstream google/adk-python@main already extracts cached_content_token_count from usage_metadata and exposes usage_cached_tokens in plugin-generated views (verified at bigquery_agent_analytics_plugin.py:1829). Remaining SDK work: add CodeEvaluator.context_cache_hit_rate(min_rate), document the minimum required ADK plugin version, fall back gracefully when the column is missing on older agent_events tables, write tests. Strong tie-in to post #2's cost narrative.
Close out blog series posts #1-#3 in #51 / #53 / #77 / #82	#51 #53 #77 #82	M	0.25 wk	Posts #1, #2, and #3 are all live on the Google Cloud Medium publication. #82 closed; #51 updated through slot 3. Remaining: confirm #53 / #77 final-state checkbox sweep.

P0 total: ~2 eng-weeks. (Item-level estimates round up to ~2.25 because the cache-hit-rate evaluator is still a small SDK polish even with the plugin work done.)

---

P1 — ship in 1 quarter (high leverage, 1 eng) or 1 month (2 eng parallel)

Item	Issue	Impact	Effort	Notes
Quality Scorecard Phase 1: pre-built rubric factories (evaluation_rubrics.py) over existing CategoricalEvaluator	#63	H	2 wk (med)	Already in flight — Gayathri uploaded evaluation_rubrics.py for review. Locks in three pillars (response_usefulness, task_grounding, policy_compliance) using existing categorical vocabulary. Additive, reuses existing dashboard views.
Quality Scorecard Phase 2: persist root_agent_name + region on categorical_results + new categorical_fleet_leaderboard view	#63	H	2 wk (med)	Schema-affecting; needs ALTER TABLE migration plan. Bridges the eval results → fleet ranking gap that the SDK doesn't have today.
Quality Scorecard Phase 3: Client.triage_low_score_sessions(...) + hitl_triage_queue table	#63	M	2 wk (med)	The genuinely net-new piece. Decide upfront: idempotent MERGE vs append-only? resolved_at lifecycle column? Worth a small design note before implementation.
Inheritance (extends) compilation in ontology DDL compiler	#30	M	2 wk (low)	Three candidate strategies named in the issue (fan-out / union view / label-referenced edges). Decide one, implement, ship as gm compile --emit-extends-as=… flag.
Compile-time extractor prerequisite: ontology-aware validate_extracted_graph(spec, graph)	#76	M	2 wk (med)	Independent shipping value; gates #75 epic. Adds field/node/edge/event-level fallback classification.
evaluate --suggest-thresholds baseline helper (deferred from blog #2 polish)	(no issue yet — file one)	M	1 wk (high)	Reads last N days of prod, prints suggested per-metric thresholds with a buffer. Halves the prose burden of blog #2's "how do I pick thresholds" sidebar.
Unified evaluation_results persistence layer (design RFC + Phase 1 schema)	#10 (item 6)	H	2 wk (med) for the RFC + Phase 1 schema; implementation rolls into Scorecard Phase 2	Promoted from #10's wishlist because Scorecard / ReasoningBank / fleet trends / SxS all assume a unified persistence shape. Today only categorical results have a stable persisted schema (categorical_results); LLM-judge and trajectory-match reports are returned in-memory. RFC needs: shared columns (session_id, metric_name, score, passed, prompt_version, endpoint, execution_mode, created_at), evaluator-type discriminator, view conventions, migration plan from categorical_results (don't fork). Without this, Scorecard Phase 2's "fleet leaderboard" only works for categorical metrics.

P1 total: ~13 eng-weeks (P0+P1 ≈ 15 eng-weeks total). For a single engineer, that's a full quarter, not a month — see resource table below.

---

P2 — ship in 1 quarter (strategic, larger effort)

Item	Issue	Impact	Effort	Notes
ReasoningBank: per-user/per-session memory of past distilled outcomes, loaded as initial agent context	#49	H	4 wk (low)	Storage layer (BQ table for memories), distillation pipeline (LLM-as-Judge + summarization), retrieval API (MemoryService.load_relevant_memories(...)), agent integration shape (callable from plugin or app). Needs a small design RFC first because memory shape affects every downstream consumer.
Compile-time code generation for structured extractors (Phase 1) — only extract_bka_decision_event and the structured-event registry	#75	M	3 wk (med)	Gated on #76 landing. Phase 1 scope is deliberate: known structured event schemas only, no free-text. Server-side AI.GENERATE stays as semantic fallback until precision/recall is measured.
Ontology pipeline migration to bigquery_ontology upstream package (strategic decision pending)	#38	H	4 wk (low)	Runtime contract migration, not a module swap. Maintainer needs to decide: full migration vs. keep SDK pipeline as a thin wrapper. Risk is high (consumed across 5+ modules); upside is dropping ~5K LOC of duplicate code.
SKOS import support alongside OWL	#57	M	2 wk (med)	Design proposal phase; needs feedback round resolved. Follows #38 because it should land in bigquery_ontology not in this repo if migration goes ahead.
Runtime entity resolution primitives — OntologyRuntime, concept index lookups, EntityResolver protocol	#58	M	4 wk (low)	Design proposal currently. Quoted user feedback: ~85% of brief-validation value sits at runtime, not schema time. Big payoff for production-agentic users; design surface needs to land first.
Auto-benchmark from traces — extract high-signal success/failure pairs to seed eval suites	#95 (Pillar 1)	H	3 wk (med)	Builds on existing quality_report.py + agent_improvement_cycle. Generalize the cycle into a reusable extractor. Cross-links to Vertex AI Prompt Optimizer integration (post #4 in the blog series).
Streaming evaluation — Pub/Sub + continuous query path that scores sessions as events arrive	#10 (item 2)	M	3 wk (low)	Partial scaffolding exists at _streaming_evaluation.py. Productization needs an architectural call: on-arrival vs. micro-batch, latency budgets, schema for "in-flight" partial sessions.

P2 total: ~23 eng-weeks. With two engineers parallel-streaming, ~12 weeks (one quarter).

---

P3 — research / future (defer until P0-P2 ships)

Item	Issue	Impact	Effort	Notes
Auto-skills loop based on AutoSkill paper (arxiv 2603.01145) — agents learn reusable skills from interaction history	#95 (Pillar 2)	H	6+ wk (low)	Cite trace as skill-formation source; abstract patterns into reusable skill objects. Big bet, mostly research.
Behavioral diff (SxS) analysis between two agent runs on identical tasks — "divergence point" detection	#10 (item 1)	M	4 wk (low)	Needs trajectory representation that supports semantic diff (more than character-level). Cross-cuts trajectory matching + insights.
Live agent resolution — turn-time EntityResolver for live agents (extends #58)	#93 (Gap 1)	M	6 wk (low)	Out-of-scope for #58; needs an agent-facing package extension of ADK. Latency budget: <50ms target. Big architecture decision.
Advanced LLM + embedding-based resolvers	#93 (Gap 2)	M	4 wk (low)	Builds on #58's EntityResolver protocol. Embedding store + cosine-similarity layer + LLM-disambiguation tier.
SHACL constraint validation	#93 (Gap 3)	L	3 wk (low)	Requires SHACL parser; integrates with validate_extracted_graph from #76. Niche but needed for governance-heavy verticals.
Spanner / MAKO backends for ontology storage	#93 (Gap 4)	L	8+ wk (low)	Alternative storage to BigQuery for ontology data. Significant — multiple concurrent backends multiply maintenance. Defer unless an ADK-team partner needs it.
V5 Context Graph — TTL import + mixed extraction + temporal lineage (BigQuery-only V5 demo)	#12	M	6 wk (low)	Existing design doc. Sequence after bigquery_ontology migration decision (#38).
Insights from latest research papers — three arxiv refs, currently unscoped	#74	?	1 wk just to triage	Needs a triage pass to convert into concrete proposals before estimating real effort.
Blog series posts #4-#10 — analyst views, agent quality scorecard publication, real-time dashboards, ontology, HITL safety	#51 (slots 4-10)	H	2 wk per post (high)	Cadence: ~one per 2-3 weeks. Critical for SDK adoption. Each post drafted by SDK lead + reviewed against live demo.

---

Items to deprecate / explicitly close

Item	Status
_LEGACY_LLM_JUDGE_BATCH_QUERY (ML.GENERATE_TEXT path inside the LLM-judge cascade)	Audit usage and define deprecation criteria. AI.GENERATE works without a connection now (per post #3 finding), but ML.GENERATE_TEXT is also intentionally referenced in _bqml_judge (client.py:1141), insights extraction (client.py:2080), and feedback paths, plus there's an active "legacy model reference" warning at client.py:602 that supports customers still passing fully-qualified BQML model refs. Don't mark deprecated until: (a) telemetry on the cascade reports how often the BQML tier actually fires, (b) documented migration path for BQML-model customers exists. Track as a follow-up issue, not a "next release" removal.
--strict for API-fallback judge errors	Already documented as a no-op in this case. Consider auto-disabling and warning rather than silently no-op'ing.
Combined --spec-path flag (gm compile --spec-path ...)	Already deprecated in favor of --ontology PATH --binding PATH. Schedule removal for 0.4.x.

---

Strategic decisions pending (need maintainer call)

Three decisions gate P2 work. Each is small to write but blocks larger implementation; sign-off needed before the matching P2 item enters a sprint.

#	Decision	Owner	Required artifact	Blocks
1	Ontology pipeline migration to upstream bigquery_ontology (#38)	Ontology workstream lead	ADR (Architecture Decision Record) — short doc, options table (full migration vs. thin wrapper vs. status quo), recommendation, expected risk surface, migration steps. ~1 day to draft.	#38, #57, #75, #76, #93 sequencing
2	Runtime entity resolution surface boundary (#58)	SDK lead + ADK plugin maintainer (joint)	API boundary note — single page naming what lives in this repo vs. ADK plugin vs. a new package, with the proposed EntityResolver protocol pinned. ~half day to draft.	#58 implementation, #93 Gap 1 (live agent resolution)
3	ReasoningBank storage shape (#49)	SDK lead	Storage schema RFC — BQ table shape (memory rows, distillation provenance, retrieval index), API surface (MemoryService.load_relevant_memories(...)), interaction with the existing agent_improvement_cycle demo so the two don't fork. ~1 day to draft.	#49 implementation, future agent_improvement_cycle productization

Recommended sequence: write all three artifacts in parallel over the same week (~3 person-days total), review together, sign off in one batch. Decision-fatigue on the maintainer side is lower if the three land as one review pass rather than three separate ones.

---

Resource budget summary

Calibrated against P0 ≈ 2 eng-weeks, P1 ≈ 13 eng-weeks (P0+P1 ≈ 15 eng-weeks total), P2 ≈ 23 eng-weeks. Hard constraint: at most one ontology bet per quarter because the workstream has 8 open issues and shared design surface — concurrent ontology work invites churn.

Headcount × time	What ships
1 eng × 1 month (~4 eng-weeks)	All of P0 + Quality Scorecard Phase 1. That's it. Don't promise Scorecard Phase 2-3 in the same month with one owner.
1 eng × 1 quarter (~12 eng-weeks)	P0 + most of P1 (Scorecard Phases 1-3, baseline helper, extractor prerequisite, persistence RFC). Inheritance compilation slips.
2 eng × 1 quarter parallel	P0 + all of P1 + one strategic P2 bet (ReasoningBank or bigquery_ontology migration). Recommend ReasoningBank because it has the highest "demo-becomes-product" leverage.
2 eng × 2 quarters parallel	Everything in P2 except deep-research items. Sequence ontology migration in Q2 once its ADR is signed.
3 eng × 2 quarters	Everything in P0+P1+P2. P3 research items still deferred.
3 eng × 1 year	Includes P3 research items (auto-skills, V5 context graph, Spanner/MAKO) one at a time.

---

Sequencing rationale (short version)

• P0 first because it's small, asked-for, and unblocks the cost-narrative thread of post Revamp README, enhance documentation navigation, and fix CI #2 readers (cache hit rate is exactly the "tune your token budget" follow-up). Reframed against the current ADK plugin state — the plugin already does the telemetry; SDK adds the evaluator + version pin + fallback.
• Quality Scorecard before everything else in P1 because it's already in flight (Gayathri's PR coming) and it converts the existing categorical evaluator surface from "I have to design my own metrics" → "here's a known-good rubric." Adoption boost.
• Persistence RFC promoted into P1 (was P3 wishlist as BigQuery Agent Analytics SDK Feature Wish List #10 item 6) because Scorecard Phase 2's fleet leaderboard, ReasoningBank's memory-load lookups, future SxS analysis, and trend tracking all assume a shared evaluation_results shape. Decide the shape once, reuse it everywhere — otherwise each consumer forks its own table.
• Ontology consolidation in P2 not P1 because it's the most expensive single decision and we don't want to block the scorecard / extractor work behind it. The ADR for Migrate SDK ontology pipeline to bigquery_ontology package #38 can run in parallel with P1 implementation.
• ReasoningBank in P2 not P3 because it ties the agent_improvement_cycle demo to a real product surface. Without ReasoningBank, the demo is "look, agents can self-improve in this contained example"; with ReasoningBank, it's "your agent's memory, in a queryable BQ table."
• At most one ontology bet per quarter is a hard constraint, not a preference: 8 open issues share design surface and concurrent work invites churn.
• Auto-skills, V5 context graph, Spanner backends to P3 because they're large bets that need a research arm or a partner team to justify the investment. Don't block production work on them.

---

What this roadmap deliberately doesn't do

• No prioritization of bigquery_ontology ↔ BigQuery-Agent-Analytics-SDK repo splits beyond the existing Migrate SDK ontology pipeline to bigquery_ontology package #38 decision. That's an organizational call.
• No commitment to specific calendar dates. Sequencing is relative; absolute dates depend on engineer count + DevRel review cycles + GA gates.
• No mention of internal-only Google Cloud product integrations (Vertex AI Agent Engine, etc.) beyond what the public surface already covers. Those would be a parallel internal roadmap.

---

How to use this issue

• Maintainer: leave reactions on items you agree with the priority of; comment with re-rankings on items you disagree with; resolve the three strategic decisions above so P2 can sequence.
• Contributors: pick a P0 or P1 item that matches your interest, drop a "I can take this" comment, and the maintainer can hand the linked issue over.
• Quarterly review: this roadmap should be re-checked every ~6 weeks. The shape of evaluation work (scorecard, auto-benchmarking) is moving fastest right now and may push items between tiers.

---

Generated 2026-04-28 from a survey of the 19 open issues, the SDK surface
at target/main, and the ADK plugin at google/adk/plugins/bigquery_agent_analytics_plugin.py.
Open to revision; this is a starting point, not a contract.

[caohy1988]

Revision pass — addressing all six review findings:

P1 (resource math):

• Added a Director Summary at the top with corrected calendar math: P0+P1 ≈ 13 eng-weeks ≈ 1 quarter for 1 engineer, not 1 month.
• Rewrote the Resource Budget table — single-engineer/single-month now ships only P0 + Scorecard Phase 1, not the whole P1 stack.
• Fixed the P1-summary footer ("~13 eng-weeks") and the duplicated "1 month (1 eng)" claims that conflicted with item totals.

P1 (cache hit-rate stale framing):

• Reframed Improve token efficiency in ADK via BigQuery Agent Analytics #32 entry against the current ADK plugin (verified at bigquery_agent_analytics_plugin.py:1829 — cached_content_token_count and usage_cached_tokens already exist upstream).
• Effort dropped from 1.5 wk → 1 wk; scope narrowed to SDK evaluator, version pin, fallback for older plugin tables, tests/docs.

P2 (premature deprecation):

• Changed _LEGACY_LLM_JUDGE_BATCH_QUERY row from "mark deprecated, remove next release" to "audit usage and define deprecation criteria." Cited the three intentional fallback paths (_bqml_judge at client.py:1141, insights at client.py:2080, the legacy-model-ref warning at client.py:602). Removal needs telemetry on cascade-tier hit rates and a customer migration path first.

P2 (missing persistence dependency):

• Promoted BigQuery Agent Analytics SDK Feature Wish List #10 item 6 (unified evaluation_results schema) into a P1 design RFC. Scorecard Phase 2 fleet leaderboard, ReasoningBank lookups, and SxS analysis all assume this exists; without the RFC each consumer forks its own table. Added to the P1 table and to the sequencing rationale.

P3 (ontology count):

• Fixed the TL;DR claim from "~6 open issues" to "8 open issues" (the actual list always had 8: V5 Context Graph: TTL import, mixed extraction, temporal lineage — implementation design #12, feat: implement inheritance (extends) compilation support #30, Migrate SDK ontology pipeline to bigquery_ontology package #38, Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57, Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58, Epic: Compile-time code generation for structured trace extractors (scoped rework) #75, Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76, Epic: Address Remaining Ontology Platform Gaps (Live Agent Resolution, Advanced Resolvers, SHACL, Spanner/MAKO) #93).
• Added explicit constraint "at most one ontology bet per quarter" to the TL;DR and the Resource Budget table — the original draft mentioned this only in passing.

P3 (P0 wording):

• P0 total now reads "~2 eng-weeks" with a footnote that the line-item rounding ticks to ~2.25 because the cache-hit-rate evaluator is still a small SDK polish even after the plugin work landed upstream.

Suggested edits:

• ✅ Director summary added with explicit 30-day / quarter recommendations.
• ✅ Strategic decisions table now has owner + required artifact + blocked work per row (ADR for Migrate SDK ontology pipeline to bigquery_ontology package #38, API boundary note for Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58, storage RFC for BigQuery Agent Analytics SDK — ReasoningBank #49). Recommendation: write all three in parallel over a single week, review as one batch.
• ✅ Resource table makes the "one ontology bet per quarter" constraint explicit.

[caohy1988]

Three small consistency fixes per the second review pass:

• Director Summary "Next quarter (1 engineer)" — pulled back to match the resource table. New wording: "P0 + most/all of P1 (~15 eng-weeks). No P2 unless P1 scope is cut." Names the realistic trade-off explicitly (drop inheritance compilation, pull ReasoningBank up — pick one, not both) instead of implying a P1-plus-P2 quarter is feasible solo.
• P1 section heading — changed from "P1 — ship in 1 month" (which conflicted with the corrected ~13 eng-week total) to "P1 — ship in 1 quarter (1 eng) or 1 month (2 eng parallel)." Heading and resource table now agree.
• P0+P1 total in Director Summary — was "≈ 13 eng-weeks," now "≈ 15 eng-weeks" to match the P1-section footer ("~13 eng-weeks for P1, ~15 for P0+P1"). One source of truth across the doc.

No content changes elsewhere; this is purely consistency cleanup over the corrected math from the previous revision.

[caohy1988]

One-pager (leadership)

The detailed roadmap above is the appendix. Below is the decision-oriented one-pager for an exec audience.

---

BigQuery Agent Analytics Roadmap: from observability to self-improving agents

The ask. Approve the strategic quarter shape: 2 engineers for one quarter. Engineer 1 lands the Quality Scorecard and unified evaluation persistence. Engineer 2 lands ReasoningBank, turning the current self-improvement demo into a reusable product surface. Fallback: with 1 engineer, ship P0 + most/all of P1 and defer ReasoningBank.

Where we are. The base is shipped and being exercised through the SDK, sandbox CI demos, and the Medium series. The ADK plugin captures agent interactions into BigQuery `agent_events`; the SDK supports trace reconstruction, deterministic CI budgets, LLM-as-Judge, categorical evaluation, and `--exit-code` CI gates. The AI.GENERATE judge path is live-verified in SDK 0.2.3 on PyPI. The first three Medium posts are live, with post #3 (Scalable LLM-as-Judge in BigQuery) anchoring the self-improvement narrative.

What we'd do next

Horizon	Investment	Outcome
30 days	1 eng	Quality Scorecard Phase 1, CI output polish, cache hit-rate evaluator.
1 quarter, baseline	1 eng	P0 + most/all of P1: Scorecard Phases 1–3, unified evaluation persistence, threshold helper, extractor prerequisite.
1 quarter, strategic	2 eng	Baseline + ReasoningBank: queryable per-user/per-session memory distilled from prior traces.

Why ReasoningBank first

1. Demo becomes product. The agent-improvement-cycle demo already exists; ReasoningBank turns it into a reusable BigQuery-backed memory surface.
2. Clear narrative arc. We move from "see traces" to "gate on traces" to "score traces" to "learn from traces."
3. Lower execution risk than ontology consolidation. ReasoningBank is one storage/API/distillation surface. Ontology consolidation is a runtime contract migration across multiple modules and should wait for an ADR.

Decisions needed this month

Decision	Why it matters
Fund baseline vs. strategic quarter shape	Sets whether ReasoningBank lands this quarter.
Confirm ReasoningBank as the first strategic bet	Prevents split focus with ontology migration.
Approve a lightweight RFC week — not implementation	Produces the `bigquery_ontology` ADR, runtime-resolver boundary note, and ReasoningBank storage RFC in one review cycle. Approval here is to write the docs, not to start the code.

Top risks

• Evaluation persistence slips. Scorecard leaderboard, ReasoningBank, and future SxS analysis all need one shared `evaluation_results` schema.
• Ontology overextension. 8 open ontology/context-graph issues; limit to one ontology bet per quarter.
• No strategic-bet decision. Without a ReasoningBank-vs-ontology call, the quarter ships only incremental P0/P1 work.

Bottom line

Invest next quarter in the self-improvement loop. With 2 engineers, ship Scorecard + persistence on one track and ReasoningBank on the second. Defer ontology consolidation until its ADR is approved and treat it as the next strategic bet.

[caohy1988]

One-pager: BigQuery Agent Analytics 2026 staffing plan

Recommendation

Assign 2 engineers as BigQuery Agent Analytics owners for the remainder of 2026.

The committed 2026 outcome is:

• Engineer 1: Evaluation Platform

  • Make evaluation results durable, queryable, and dashboard-ready.
  • Ship Quality Scorecard, unified evaluation_results, cache hit-rate evaluator, CI/reporting polish, and threshold helper.

• Engineer 2: Self-Improvement Loop

  • Turn prior trace outcomes into reusable agent memory.
  • Ship ReasoningBank, productize agent_improvement_cycle, and add auto-benchmark generation from traces.

This plan assumes normal interrupt load: reviews, releases, demos, DevRel support, and customer-driven fixes. With protected full-time allocation, remaining 2026 capacity should go to one ADR-approved ontology/runtime track. With only 1 engineer, commit to the Evaluation Platform track and defer ReasoningBank.

---

Priority-ranked work plan

Rank	Work item	Concrete deliverables	Impact	Effort	Owner
1	Quality Scorecard Phase 1	Prebuilt rubric factories for response_usefulness, task_grounding, policy_compliance; docs; examples; tests	High	2 wks	Eng 1
2	Unified evaluation persistence	New shared evaluation_results schema for LLM-judge, categorical, trajectory, and future scorecard outputs; migration path from categorical_results; views for latest score per session	High	2-3 wks	Eng 1
3	Quality Scorecard Phase 2	Persist root_agent_name and region; create fleet leaderboard view; show scorecard by agent / region / metric	High	2 wks	Eng 1
4	CI/reporting polish	Escape feedback="..."; close blog-series cleanup issues; ensure red CI output is parseable and screenshot-ready	Medium	0.5 wk	Eng 1
5	Cache hit-rate evaluator	CodeEvaluator.context_cache_hit_rate(min_rate); fallback for older ADK plugin tables; docs/tests	High	1 wk	Eng 1
6	Threshold helper	bq-agent-sdk evaluate --suggest-thresholds; reads recent prod runs and recommends latency / tool-call / cost thresholds	Medium	1 wk	Eng 1
7	Quality Scorecard Phase 3	Client.triage_low_score_sessions(...); hitl_triage_queue table; unresolved / resolved lifecycle fields	Medium	2 wks	Eng 1
8	ReasoningBank RFC	BigQuery memory table shape; distillation provenance; retrieval API contract; interaction with agent_improvement_cycle	High	1 wk	Eng 2
9	ReasoningBank implementation	Memory write path, distillation job, MemoryService.load_relevant_memories(...), SDK docs, example notebook/demo	High	4-5 wks	Eng 2
10	Auto-benchmark from traces	Convert high-signal success/failure sessions into reusable eval cases; seed eval suites from real traces	High	3 wks	Eng 2
11	Extractor validation prerequisite	validate_extracted_graph(spec, graph); structured validation errors; tests for node/edge/event fallback	Medium	2 wks	Eng 1 after Scorecard, or Eng 2 after ReasoningBank
12	Ontology migration ADR	Decide full migration vs thin wrapper vs status quo for bigquery_ontology; document migration risks and module impact	High decision value	1 wk	Tech lead + ontology owner
13	Ontology migration implementation — stretch / ADR-gated	Only after ADR: runtime contract migration, package boundary cleanup, docs/tests	High but risky	4+ wks	Eng 3 or protected stretch capacity
14	Runtime resolver boundary — stretch / ADR-gated	OntologyRuntime, EntityResolver protocol, repo/package ownership boundary	Medium	4 wks	Eng 3 or protected stretch capacity
15	Research bets — future / partner-funded	Auto-skills, V5 context graph, Spanner/MAKO backends, advanced resolvers	Future	6-8+ wks each	Partner / future

---

Delivery scenarios through Dec 2026

Staffing	By end of 2026 we can commit to	Stretch goals	What slips
1 engineer	Quality Scorecard Phases 1-3, unified evaluation_results, cache hit-rate evaluator, threshold helper, CI/reporting polish	Extractor validation prerequisite	ReasoningBank, auto-benchmark, ontology migration
2 engineers recommended	Everything in 1-engineer plan plus ReasoningBank and auto-benchmark from traces	Extractor validation, ontology ADR	Ontology implementation, runtime resolver
3 engineers	Everything in 2-engineer plan plus one ADR-approved ontology/runtime track	Runtime resolver prototype	Research bets
3+ engineers / partner support	Full evaluation + ReasoningBank + ontology/runtime work	Auto-skills or V5 context graph	Depends on partner scope

---

Proposed sequencing

Month-by-month sequencing is directional; release timing depends on review latency, API design decisions, and whether live demos/blog support continue to interrupt feature work.

May

• Finish CI/reporting polish.
• Ship cache hit-rate evaluator.
• Start Quality Scorecard Phase 1.
• Write ReasoningBank RFC.
• Draft ontology migration ADR so implementation can be approved or deferred deliberately, ahead of any October capacity decision.

June

• Land Quality Scorecard Phase 1.
• Start unified evaluation_results.
• Start ReasoningBank implementation.
• Land the ontology migration ADR (review + sign-off).

July-August

• Land unified evaluation_results.
• Ship Scorecard Phase 2 leaderboard.
• Complete ReasoningBank memory schema, write path, and retrieval API.

September

• Ship Scorecard Phase 3 HITL triage queue.
• Productize agent_improvement_cycle on ReasoningBank.
• Start auto-benchmark from traces.

October-November

• Ship auto-benchmark generation.
• Add extractor validation prerequisite if capacity allows.
• If the ADR is approved and stretch capacity exists, start the ontology / runtime track; otherwise defer it to 2027 planning.

December

• Stabilization, docs, demos, and 2027 planning.
• Decide whether ontology/runtime resolver becomes next funded track.

---

Leadership decisions needed

Decision	Recommendation
Staff level	Assign 2 engineers as 2026 owners; protect enough capacity to deliver Evaluation Platform + ReasoningBank
First strategic bet	Choose ReasoningBank before ontology
Ontology work	Approve ADR now; implementation only after ADR sign-off
Success metric	Endorse: by Dec 2026, durable evaluation data, fleet-level scorecards, red/green CI gates, triage queues, and ReasoningBank-backed agent memory are all live.

Bottom line

With 2 named owners through 2026, BigQuery Agent Analytics gets a concrete product arc: durable evaluation data, fleet-level scorecards, red/green CI gates, triage queues, and ReasoningBank-backed agent memory. Ontology/runtime work should start only after the ADR is approved and only with protected stretch capacity or a third engineer.

[caohy1988]

Working plan: ontology / property-graph / extractor track (cross-issue)

Sequencing across #58, #75, #76, #104, #105 to make the user-visible story coherent. Filed here on the roadmap because it spans five issues and the calendar is shared. Individual issues remain the source of truth for scope.

Story arc

1. Populate a user-pre-defined BigQuery property graph from BQ AA traces. Foundation already exists (V5 ontology pipeline + PR feat(owl-import): SKOS support alongside OWL #62 abstract/SKOS plumbing + PR feat(ontology): Phase 1 — concept-index emission (issue #58, A1-A8 squashed) #92 concept-index emission). Phase A makes the CLI ergonomic and adds pre-flight safety.
2. Catch extractor mistakes before they hit BigQuery. Phase B inspects extracted output.
3. Reduce AI.GENERATE fallback for structured events when the schema is known. Phase C compiles deterministic extractors for events with known shape; only proceeds if measurements support it. (Phase 1 does not reduce cost vs. the existing hand-written registry — both are zero-token. The win is authoring scale, fewer fallbacks to LLM extraction, and uniform safety/validation gates.)
4. Resolve user-facing identifiers against ontology / SKOS at agent runtime. Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58 reader/runtime API track; runs in parallel.

Dependency graph

#104 ─────────────────────────────────────────────► ship independently
#105 ─────────────────────────────────────────────► ship independently
#76  ──────────► #75 (P0.1 hard prerequisite)
#58  ─────────────────────────────────────────────► parallel track (Phase 1 emission shipped in #92)

Strict dependency: only #76 → #75. Everything else is sequenceable on value/risk.

Phase A — Make the user-pre-defined-graph story usable on the CLI (1.5–2 weeks)

1. #104 — --skip-property-graph CLI flag (0.5 eng-day, single PR)

• Smallest change. Unblocks the "I own my CREATE PROPERTY GRAPH DDL, just populate the base tables" workflow without forcing users into Python.
• One CLI flag + one orchestrator branch + corrected cli.py:1277 exit-handling for skipped_reason == "user_requested".

2. #105 — pre-flight binding-vs-BQ-table validator (2–3 eng-days, recommend 2 PRs)

• PR 2a — validator core: validate_binding_against_bigquery(), BindingValidationReport, BindingValidationFailure, BindingValidationWarning, FailureCode enum, the seven default-mode failure codes plus one strict-only code (KEY_COLUMN_NULLABLE, which escalates from warning to failure under strict=True), unit tests against a fake BQ client. Default-mode regression test that SDK-create_tables() output validates clean (NULLABLE keys are fine by default — emit a BindingValidationWarning; KEY_COLUMN_NULLABLE becomes a hard failure only under --strict).
• PR 2b — CLI surface: bq-agent-sdk binding-validate [--strict] standalone command + ontology-build --validate-binding opt-in flag. Live integration test gated on RUN_LIVE_BIGQUERY_TESTS=1.
• Ships before Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76: catches the most common authoring error (binding YAML drifted from physical tables) before extraction wastes AI.GENERATE tokens. Matches the user journey — validate physical tables first, then validate extracted graph.

Phase B — Make extraction output safe and inspectable (1 week)

3. #76 — validate_extracted_graph post-extract validator (~1 week, recommend 2 PRs)

• PR 3a — validator core: validate_extracted_graph(spec, graph), ValidationReport, ValidationFailure, FallbackScope enum, all 11 defined failure codes against fake fixtures (4 NODE-scope + 3 FIELD-scope + 4 EDGE-scope per Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76), regression test that extract_bka_decision_event's output validates ok=True against its declared entity.
• PR 3b — adapter + integration: validate_extracted_graph_from_ontology(ontology, binding, graph) adapter, public export from __init__.py, docs/ontology/validation.md, opt-in pre-materialization gate on the orchestrator (mirrors Feat: validate ontology binding against existing BigQuery table schemas (pre-flight) #105's --validate-binding shape).
• Independently useful regardless of Epic: Compile-time code generation for structured trace extractors (scoped rework) #75. Hard prerequisite for Epic: Compile-time code generation for structured trace extractors (scoped rework) #75 Phase 1.

Cross-PR consistency note (revised)

#105 and #76 should expose the same public report ergonomics: report.ok, report.failures, typed failure codes, printable summaries, and non-zero CLI exit on failure. Do not force a single shared ValidationFailure too early — #105 failures need binding_path / bq_ref, #76 failures need node_id / edge_id / event_id / FallbackScope, and a unified type would become a bag of optional fields. If the second implementation duplicates obvious report boilerplate, lift only the common helper / report base into validation_common.py.

Phase C — Lower extraction cost and improve precision (3–5 weeks, milestone-gated)

Phase C is calendar-honest about uncertainty. The first milestone is the smallest unit that produces a go/no-go signal; everything after depends on it.

Milestone C1 — compile harness + one compiled extractor + measurement report (2–3 weeks, 3 PRs)

• PR 4a — P0.2 runtime-target RFC merge: short doc PR settling Option A (client-side Python) for Phase 1. Required before any compilation code lands.
• PR 4b — compile harness + smoke-test runner: template-fill pipeline, AST validation, smoke test against sample events, ontology validation via Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76. Fingerprint includes (ontology, binding, event_schema, event_allowlist, transcript_builder_version, content_serialization_rules, extraction_rules, template_version, compiler_package_version) per the issue.
• PR 4c — compile extract_bka_decision_event + measurement report: first compiled extractor. Hand-written version is the F1 ground truth for its own compiled replacement. Measurement report compares F1 / token-cost / latency / fallback-rate against hand-written and AI.GENERATE baselines on a real trace corpus.

Go/no-go gate after C1: the issue's accept bar is F1 ≥ 0.95 and fallback rate ≤ 10%. If C1 measurements clear that bar, proceed to C2. If not, Phase C contracts to "C1 only" and the epic's scope reduces accordingly.

Milestone C2 — operationalization (1–2 weeks, conditional on C1, 2–3 PRs)

• Sidecar storage + bundle-loading runtime path so compiled extractors are actually called by run_structured_extractors().
• Revalidation harness (scheduled / on-demand agreement check + per-field disagreement table).
• Doctor integration: surface "recompile recommended" when agreement drops.
• Two more hand-written baseline extractors (TOOL_COMPLETED shape + a HITL event) so F1 isn't a single-point measurement.

Phase 2 (compile session-aggregated AI.GENERATE extractors per #75) is not on this calendar. It's gated on Milestone C2's measurements continuing to support compilation at the session-aggregated tier, which has more open-ended inputs and may not survive the F1 bar.

Parallel track — #58 runtime concept-index (continuous)

Status (verified 2026-05-02): Phase 1 (concept-index emission, items A1–A8) shipped in PR #92, merged 2026-04-28. The issue body has been updated to reflect this — "Design proposal. Not yet implemented." replaced with a note that emission is live and remaining work is reader / runtime accessor APIs.

Future phases (concept-index reader API, multi-layer resolution primitives, agent-facing surface, the EntityResolver protocol + reference resolvers) ship on their own cadence. Estimate 2–3 weeks per future phase.

One coordination point: when #105 PR 2a lands, the concept-index emission tables become validation targets too. Extend #105's table list once #58's emission tables are part of the standard ontology pipeline.

Calendar (one engineer, normal interrupt load)

Week	What lands	Confidence
1	#104 (PR 1) + #105 PR 2a (validator core)	High
2	#105 PR 2b (CLI) + #76 PR 3a starts	High
3	#76 PR 3a + PR 3b	High
4	#75 PR 4a (RFC) + PR 4b starts	Medium
5–6	#75 PR 4b (compile harness) + PR 4c (first extractor + measurement report)	Medium
C1 go/no-go gate	F1 ≥ 0.95 and fallback ≤ 10% on real traces	—
7–8	(if C1 clears) C2 PRs	Lower until C1 clears

#58 reader/runtime work runs in parallel on its own track (no shared dependencies with A/B/C above except the one coordination point on emission tables).

Why this order is right (one-liner each)

• Feat: bq-agent-sdk ontology-build --skip-property-graph to populate base tables only #104 first — cheapest, immediate user win, makes Feat: validate ontology binding against existing BigQuery table schemas (pre-flight) #105 obviously useful.
• Feat: validate ontology binding against existing BigQuery table schemas (pre-flight) #105 before Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76 — catches the more common error class (binding↔table drift) earliest in the pipeline, before extraction spends money.
• Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76 before Epic: Compile-time code generation for structured trace extractors (scoped rework) #75 — hard prerequisite per the epic.
• Epic: Compile-time code generation for structured trace extractors (scoped rework) #75 milestone-gated — Phase 1 is the unit that produces a go/no-go signal; doctor / revalidation / extra baselines are conditional follow-on, not pre-committed.
• Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58 in parallel — independent of A/B/C; remaining work is reader/runtime APIs over already-emitted tables.

[caohy1988]

Seven-finding cross-issue review folded in

Verified each against current main (HEAD 85d8525 + upstream/main #92 concept-index files) before fixing. Summary of changes by issue:

#76 — failure-code count corrected. Counted the actual codes: 4 NODE (unknown_entity, missing_node_id, duplicate_node_id, missing_key) + 3 FIELD (unknown_property, type_mismatch, unsupported_type) + 4 EDGE (unknown_relationship, unresolved_endpoint, wrong_endpoint_entity, missing_endpoint_key) = 11. Success criterion changed from "All 12+ failure codes" to "All 11 defined failure codes" with the explicit list inline so future additions update both the rules section and the AC.

#75 — three fixes:

1. Enum-fallback wording in P0.1 changed from "property type mismatch or enum miss" to "property type mismatch or unknown property," with an inline note that enum-membership miss is deferred from Feat: Ontology-aware validate_extracted_graph with fallback-scope classification (prerequisite for #75) #76's first landing because ResolvedProperty doesn't carry enum value lists today.
2. C1/C2 ontology_graph.py boundary clarified. Replaced "No impact on ontology_graph.py" with: C1 doesn't touch ontology_graph.py; C2 plugs compiled-bundle loading into the existing run_structured_extractors() hook (structured_extraction.py:198–232), reuses the prune/hint behavior at ontology_graph.py:540–571, and changes only what's passed in excluded_span_ids / partial_span_ids, not the AI SQL semantics.
3. "Measured outcomes" reframed away from per-event token attribution. Per-event extractor F1 + per-event extractor latency + fallback-rate are C1's measurement targets. Session-level prompt-size estimates and job-level BigQuery stats move to C2-only measurements, aligned with Doc: V5 migration notebook storyboard for the four-guarantee decision-lineage narrative #107's billing-honest framing. Per-event token attribution explicitly called out as not available because the AI query at ontology_graph.py:90–136 returns only (session_id, graph_json) with no per-row usage metadata.

#104 — CLI output disambiguation acceptance criterion added. Body now requires the curated CLI output dict (around cli.py:1266–1274) to gain a property_graph_status field with values "created" / "failed" / "skipped:user_requested", so JSON consumers can distinguish skipped from failed without parsing stderr. Default behavior (no flag) stays exactly as today.

#105 — strict-mode contract fully specified:

• API gains strict: bool = False parameter explicitly.
• New BindingValidationWarning type, parallel shape to BindingValidationFailure. Default-mode emits warnings to report.warnings for strict-only checks; strict=True escalates them into report.failures with the same code.
• BindingValidationReport documented with both failures and warnings fields. report.ok returns True iff failures is empty (warnings don't flip ok).
• CLI surfaces: bq-agent-sdk binding-validate --strict and ontology-build --validate-binding-strict. Default mode prints warnings to stderr but exits 0; strict mode escalates them to exit 1.
• New AC explicitly tests both default-mode (warning emitted, report.ok True) and strict-mode (failure emitted, report.ok False) behavior on the same fixture.

#96 working plan — count alignment. PR 2a description updated from "eight default-mode failure codes" to "seven default-mode failure codes plus one strict-only code (KEY_COLUMN_NULLABLE, escalates from warning to failure under strict=True)." Also tightened PR 3a from "all twelve+ failure codes" to "all 11 defined failure codes (4 NODE-scope + 3 FIELD-scope + 4 EDGE-scope per #76)."

Net

The five issues + the working plan now agree on counts, on the strict-mode contract surface, on the C1/C2 boundary inside ontology_graph.py, and on the per-event vs per-session attribution honesty (per-event for extractor metrics, per-session/job-level for token billing). Implementers should no longer make locally-reasonable but cross-issue-inconsistent choices.