feat(ontology): add --skip-property-graph for user-owned graph DDL (#104)

[caohy1988]

Implements #104 — --skip-property-graph for users with their own CREATE PROPERTY GRAPH DDL (Terraform / dbt / hand-authored). Lets the SDK populate base tables from BQ AA traces without overwriting the user's graph object on every run.

Changes

ontology_orchestrator.py

• build_ontology_graph(...) gains skip_property_graph: bool = False.
• When True, phase 5 short-circuits: no OntologyPropertyGraphCompiler is constructed, no CREATE OR REPLACE PROPERTY GRAPH runs.
• Result dict gains property_graph_status ("created" / "failed" / "skipped:user_requested") and skipped_reason ("user_requested" only when phase 5 was skipped).

cli.py (ontology-build)

• New --skip-property-graph flag, threaded through to build_ontology_graph.
• Curated CLI output dict gains property_graph_status so JSON consumers can distinguish skipped from failed without parsing stderr.
• Exit handling: skipped_reason == "user_requested" exits 0 silently; existing exit-1-with-error behavior preserved for actual graph-creation failures (property_graph_created=False for any other reason).

Docs (docs/ontology/ontology-build.md, new)

• Documents the SDK orchestrator end-to-end with a dedicated --skip-property-graph section.
• Status-field reference table mapping property_graph_status → property_graph_created + exit code, so JSON consumers have a stable contract.
• "When to use this" guidance covering Terraform/dbt-managed DDL, custom graph options, and decoupled refresh cadence.
• Python API example with the expected result-dict shape.
• Linked from docs/README.md under Ontology Reference.

Tests

Unit (tests/test_ontology_orchestrator.py):

• test_skip_property_graph_does_not_construct_compiler — asserts OntologyPropertyGraphCompiler is never called when the flag is set.
• test_property_graph_status_created_on_success — default flow with successful creation reports "created", no skipped_reason.
• test_property_graph_status_failed_on_compiler_false — default flow with create_property_graph()=False reports "failed", distinct from "skipped:user_requested".

CLI (tests/test_cli.py::TestOntologyBuild):

• test_skip_property_graph_exits_zero_with_status — exit 0, status "skipped:user_requested", no "Property Graph creation failed" message, flag threaded through.
• test_default_invocation_omits_skip_flag — default invocation passes skip_property_graph=False.
• test_skip_property_graph_status_visible_in_text_format — pins the contract that property_graph_status appears in --format=text output, not just JSON.
• test_property_graph_failure_status_failed — exit 1 with status "failed" and the existing error message preserved.

Live integration (tests/test_integration_ontology_binding.py::TestSkipPropertyGraph, gated on RUN_LIVE_BIGQUERY_TESTS=1):

• test_skip_property_graph_issues_no_create_graph_job — full end-to-end test:
  1. Create authored CREATE OR REPLACE PROPERTY GRAPH directly via SQL (simulating Terraform/dbt-managed DDL).
  2. Capture CURRENT_TIMESTAMP() after the authored DDL completes (closes the false-positive trap from Doc: V5 migration notebook storyboard for the four-guarantee decision-lineage narrative #107 cell 1.3 where the JOBS_BY_PROJECT filter would otherwise catch the test's own setup job).
  3. Run build_ontology_graph(..., dataset_id=_DATASET, table_id=_TABLE, skip_property_graph=True) — extraction reads from real agent_events, materializer writes to scratch_dataset per spec, no DDL job for phase 5.
  4. Assert property_graph_status == "skipped:user_requested" AND sum(rows_materialized.values()) > 0 (catches the silent-empty-graph trap).
  5. Query INFORMATION_SCHEMA.JOBS_BY_PROJECT WHERE creation_time > @before AND UPPER(query) LIKE '%CREATE OR REPLACE PROPERTY GRAPH%' AND query LIKE '%{_PROJECT}.{scratch_dataset}.{spec.name}%' — assert zero rows. The graph-ref filter prevents false-fail from unrelated DDL jobs running concurrently in the same project.
  6. Re-run the showcase GQL query — confirms the user's graph object survived intact.

This is exactly the contract #107 cells 1.3–1.5 will demonstrate in the V5 migration notebook when the storyboard moves to execution.

7/7 ontology-build CLI tests + 4/4 orchestrator tests pass; all 135 mocked tests in test_ontology_orchestrator.py and test_cli.py pass.

Next

After this lands, #105 (binding pre-flight) can start independently per the working plan. #76 follows; #75 C1 is gated on #76.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[caohy1988]

Follow-up: closed the two #104 AC gaps from review

Docs added — docs/ontology/ontology-build.md documents the SDK orchestrator end-to-end with a dedicated --skip-property-graph section. Includes:

• Per-flag behavior table mapping property_graph_status (created / failed / skipped:user_requested) to property_graph_created and CLI exit code, so JSON consumers have a stable reference for status disambiguation.
• "When to use this" section covering Terraform/dbt-managed DDL, custom graph options, and decoupled refresh cadence.
• Python API example showing skip_property_graph=True with the expected result-dict keys.

Live integration test added — new TestSkipPropertyGraph::test_skip_property_graph_issues_no_create_graph_job in tests/test_integration_ontology_binding.py. Gated on RUN_LIVE_BIGQUERY_TESTS=1 matching the existing live-test pattern. Sequence:

1. Create authored CREATE OR REPLACE PROPERTY GRAPH directly (simulating user-managed DDL).
2. Capture CURRENT_TIMESTAMP() after the authored DDL completes (avoids the false-positive trap from Doc: V5 migration notebook storyboard for the four-guarantee decision-lineage narrative #107 cell 1.3 where the JOBS_BY_PROJECT filter would catch the user's own setup job).
3. Run build_ontology_graph(..., skip_property_graph=True) — assert property_graph_status == "skipped:user_requested".
4. Query INFORMATION_SCHEMA.JOBS_BY_PROJECT WHERE creation_time > @before AND UPPER(query) LIKE '%CREATE OR REPLACE PROPERTY GRAPH%' — assert zero rows.
5. Re-run the showcase GQL query against the user's graph — confirms the graph object survived the SDK run intact.

This is the exact contract #107 cells 1.3–1.5 will demonstrate in the V5 migration notebook once the storyboard moves to execution.

The unit-test coverage from the original commit is unchanged and still passes (135/135).

[caohy1988]

Review fixes folded in (commit 8d45671)

(1) Live test now exercises real extraction/materialization. Was passing dataset_id=scratch_dataset so extraction queried scratch_dataset.agent_events (which doesn't exist). The AI extraction path silently returns an empty graph when the source table is missing (ontology_graph.py:683), so the test could pass while phases 1-4 did nothing. Fixed: now passes dataset_id=_DATASET, table_id=_TABLE so extraction reads the real agent_events table; materializer still writes to scratch_dataset because spec entity sources arrive 3-part-qualified to binding.target.dataset via _qualify_source (resolved_spec.py:141), so dataset_id doesn't influence output table location. Added assert sum(rows_materialized.values()) > 0 to catch the silent-empty trap.

(2) JOBS_BY_PROJECT assertion narrowed. Previously filtered every project-level job by LIKE '%CREATE OR REPLACE PROPERTY GRAPH%', which could false-fail on an unrelated graph-DDL job from another developer or test in the same project. Now also requires the query to contain the test's fully-qualified graph reference ({_PROJECT}.{scratch_dataset}.{spec.name}), scoping the assertion to the test's own scratch dataset.

(3) docs/README.md gains a row for ontology/ontology-build.md under Ontology Reference, so the new doc is discoverable.

(4) New CLI test test_skip_property_graph_status_visible_in_text_format asserts property_graph_status appears in --format=text output. Pins the contract that the status field is not JSON-only.

(5) PR body updated to reflect the docs + live test from commit 054e141 and the hardening from 8d45671. Tests section now lists unit / CLI / live coverage explicitly.

(6) CLA check — that's an admin/account-side gate that needs your action; I can't resolve it via PR commits.

Verified locally:

• pytest tests/test_ontology_orchestrator.py tests/test_cli.py → 135 passed.
• pytest tests/test_cli.py::TestOntologyBuild → 7/7 pass (was 6 before adding the text-format check).
• Live test (TestSkipPropertyGraph::test_skip_property_graph_issues_no_create_graph_job) collects clean; runs only with RUN_LIVE_BIGQUERY_TESTS=1.

[caohy1988]

Review fixes folded in (commit e98f1c2)

(1) DDL-detection blind spot fixed. The previous filter required a regressed CREATE OR REPLACE PROPERTY GRAPH to target _PROJECT.<scratch_dataset>.<spec.name>. But the orchestrator's compiler builds the DDL with dataset_id=_DATASET in this test (since the test passes _DATASET for extraction), so a regression would fire DDL targeting _PROJECT._DATASET.<spec.name> — and the old filter would miss it.

Replaced with two narrower constraints that catch a regression in either dataset:

• Graph name (spec.name) — appears in the DDL string regardless of which dataset the compiler targets.
• sdk_feature='ontology-gql' label — only SDK-issued property-graph jobs carry this label per ontology_property_graph.py:465. The test's setup CREATE PROPERTY GRAPH (issued via plain SQL, not via the SDK) does not get this label, so it doesn't trip the assertion. Cleaner than dataset-scoped filtering.

This catches the regression case the test was actually meant to detect.

(2) graph_ref source/target limitation documented. Added a "Known limitation" section to docs/ontology/ontology-build.md noting that result["graph_ref"] reports the extraction dataset, not the binding's target dataset, in split source/target setups. The materialized base tables themselves still go to the binding's target dataset per the resolved spec; only the reported string is affected. Worth a follow-up issue to either rename the field or compute it from binding.target — flagged here as out-of-scope for #108.

(3) Docs wording softened. "additional indexes, dialect-specific options" was overreaching for BigQuery property graphs; tightened to "custom labels or other DDL details the SDK's compiler doesn't generate."

(4) CLI live smoke test (finding #3 — non-blocker). Not added in this commit. The unit tests cover CLI wiring (property_graph_status in JSON + text output, exit codes for skip / fail / success, flag threading); the live test exercises the orchestrator path end-to-end. A literal CLI live smoke test (running subprocess.run(['bq-agent-sdk', 'ontology-build', '--skip-property-graph', ...])) is a reasonable follow-up but doesn't add coverage beyond what's already there.

Verified locally:

• pytest tests/test_ontology_orchestrator.py tests/test_cli.py → 136 passed.
• Live test (TestSkipPropertyGraph::test_skip_property_graph_issues_no_create_graph_job) collects clean; runs only with RUN_LIVE_BIGQUERY_TESTS=1.