feat(cli): add bq-agent-sdk binding-validate + ontology-build --validate-binding flags (#105 PR 2b)#112
Conversation
Review fixes folded in (commit 1520ee2)(1) Failure-code reference now uses lowercase JSON values. The CLI emits (2) CHANGELOG entry added under (3) Warning-only path now tested. New
This covers (4)
Verified locally: 256/256 tests pass ( Optional nit (deferred): |
…oogleCloudPlatform#105 PR 2b) Closes the user-facing surface of issue GoogleCloudPlatform#105. Builds on PR 2a's validate_binding_against_bigquery core (shipped in PR GoogleCloudPlatform#109). CLI changes (src/bigquery_agent_analytics/cli.py) Standalone command: bq-agent-sdk binding-validate --project-id ... --ontology X.yaml --binding Y.yaml [--location US] [--strict] [--format json|text|table] Loads the ontology + binding via upstream loaders, builds a google.cloud.bigquery.Client, calls validate_binding_against_bigquery(...), prints a structured JSON report on stdout. Warnings always print to stderr (one line each) so CI logs surface advisory drift even when stdout is consumed by a JSON-aware tool. Exit codes: 0 ok, 1 failure(s), 2 unexpected error (load failure, missing flags, etc). ontology-build flags: --validate-binding (default-mode pre-flight) --validate-binding-strict (strict-mode pre-flight) When set, run the validator before phase 2 (extraction). On any failure, the build short-circuits BEFORE any AI.GENERATE call fires, so authoring drift never costs tokens. Default-mode warnings print to stderr but do not block. The two flags are mutually exclusive; both are incompatible with the deprecated --spec-path form because the validator needs the unresolved Ontology + Binding pair, not a combined GraphSpec. Helper _run_binding_preflight() centralizes the load + validate + print + exit logic so it can be invoked from ontology-build without duplicating the flow. Also: --location flag added to ontology-build, threaded through to build_ontology_graph() (which has supported it on the Python side since PR GoogleCloudPlatform#108 / commit 292320b). The CLI was previously building without forwarding it. Tests (tests/test_cli.py) TestBindingValidate (6 tests): - test_clean_validation_exits_zero - test_failures_exit_one_with_payload - test_warnings_print_to_stderr_but_do_not_flip_exit - test_strict_flag_threaded_through - test_missing_required_flags_exit_2 - test_load_failure_exits_two TestOntologyBuildValidateBindingFlag (5 tests): - test_validate_binding_short_circuits_on_failure_before_build (asserts build_ontology_graph is NEVER called when validation fails — extraction does not start) - test_validate_binding_strict_short_circuits_on_nullable_keys - test_validate_binding_clean_proceeds_to_build - test_validate_binding_with_spec_path_rejected - test_both_flags_rejected (mutual exclusion) All tests use patched validators (no live BQ); the live integration test for the full flow lives in PR 2a (GoogleCloudPlatform#109). 253/253 tests pass across the touched test files. Docs New: docs/ontology/binding-validation.md - When to run (binding authoring, CI gating, ontology-build). - Standalone CLI examples (default + strict). - ontology-build integration examples. - Failure-code reference table covering all 7 default codes plus the strict-only KEY_COLUMN_NULLABLE. - CI usage pattern (GitHub Actions). - Python API example. - Cross-links to ontology-build.md, binding.md, and GoogleCloudPlatform#76's planned post-extraction validator. Updated: docs/README.md - Added binding-validation.md to the Ontology Reference table. What's NOT in this PR - Live integration test for the CLI surface (the validator itself has live coverage from PR 2a's TestBindingValidationLive). A literal subprocess-style CLI smoke test could land as a follow-up if someone wants it, but the unit tests cover wiring + exit codes + flag threading comprehensively.
…ly paths (GoogleCloudPlatform#105 PR 2b) Four review findings folded in. (1) Failure-code reference now shows lowercase JSON values The CLI emits f.code.value (e.g. 'missing_table'), not the enum attribute name ('MISSING_TABLE'). Docs previously listed the uppercase form, so users copying the table into jq filters would silently match nothing. The reference table now has both columns (Python FailureCode + JSON value), with a sample jq filter using the lowercase form. (2) CHANGELOG entry under [Unreleased] This PR adds a public CLI command (binding-validate), two public ontology-build flags (--validate-binding[-strict]), --location on ontology-build, and the validate_binding_against_bigquery Python API. All four are now called out under [Unreleased] / Added. (3) Test the warning-only path through ontology-build The new test test_validate_binding_warnings_only_proceeds_to_build patches the validator to return a BindingValidationReport with only warnings (no failures), runs ontology-build --validate-binding, and asserts: - exit code 0 - build_ontology_graph IS called (warnings don't short-circuit) - WARN: key_column_nullable appears in the CLI output This covers _run_binding_preflight()'s default-mode advisory branch, complementing the existing failure-short-circuit and clean-success tests. (4) --location now has tests and a doc note Two new tests: - TestBindingValidate::test_location_threaded_to_bigquery_client confirms binding-validate --location=EU constructs a BQ client with location='EU'. - TestOntologyBuildValidateBindingFlag:: test_location_threaded_through_orchestrator confirms ontology-build --location=EU forwards to build_ontology_graph so the orchestrator's BQ client targets the EU multi-region. docs/ontology/ontology-build.md: new 'BigQuery location' section covering --location plus a 'Pre-flight binding validation' section referencing binding-validation.md for the full --validate-binding flag reference. 256/256 tests pass (test_binding_validation.py + test_cli.py + test_ontology_orchestrator.py + test_ontology_materializer.py + test_resolved_spec.py). Autoformat clean. Optional nit (deferred, not blocking): _run_binding_preflight loads ontology + binding once for validation, _load_spec_from_args reloads them for resolve(). Worth a small helper that returns the loaded objects once, but acceptable as-is for PR 2b scope.
…loudPlatform#105 PR 2b) Two wording fixes; no behavior change. The validator uses bq_client.get_table(...) metadata calls (binding_validation.py:309), not direct INFORMATION_SCHEMA queries. Two places said the latter: (1) docs/ontology/ontology-build.md: 'pre-flight validator queries INFORMATION_SCHEMA in the same region' -> 'uses the BigQuery client with the requested location to fetch each bound table's metadata'. (2) tests/test_cli.py docstring on test_location_threaded_to_bigquery_client: same correction. The test asserts the location threads through; the wording for *why* location matters now matches the actual mechanism. 256/256 tests still pass. Autoformat clean.
0ae07e4 to
828378d
Compare
Rebased onto fresh main (PR 2a merged)#109 (PR 2a) merged at upstream Force-pushed via Verified locally: 175/175 tests pass across PR body opening paragraph removed ("stacked on #109" note no longer applies). |
Implements #105 PR 2b per the working plan on #96: the user-facing CLI surface and docs. PR 2a (#109, merged) shipped the validator core + unit tests + a live integration test.
What's new in this PR
Standalone CLI:
bq-agent-sdk binding-validatebigquery.Client, calls the validator, prints a structured JSON report on stdout."missing_table"); the docs reference table shows both the PythonFailureCodeenum name and the JSON value.0ok,1failure(s),2unexpected error (load failure, missing flags).ontology-buildintegration flagsAI.GENERATEcall fires — authoring drift never costs tokens.--spec-path(validator needs the unresolvedOntology + Bindingpair)._run_binding_preflight()centralizes the load + validate + print + exit logic.Plus:
--locationonontology-buildbuild_ontology_graph()has supportedlocationsince PR #108. The CLI was building without forwarding it. This PR threads it through.Tests
14 new CLI tests, all pass (using patched validators — no live BQ; live coverage is in PR 2a's
TestBindingValidationLive):TestBindingValidate(7 tests) — clean validation, failure payload, warnings to stderr (default mode does not flip exit), strict flag threading, missing-flag exit-2, load-failure exit-2,--locationthreaded to the BQ client.TestOntologyBuildValidateBindingFlag(7 tests):test_validate_binding_short_circuits_on_failure_before_build— assertsbuild_ontology_graphis never called when validation fails. Extraction never starts; noAI.GENERATEtokens spent.test_validate_binding_strict_short_circuits_on_nullable_keystest_validate_binding_clean_proceeds_to_buildtest_validate_binding_warnings_only_proceeds_to_build— assertsbuild_ontology_graphIS called when only warnings are present (default-mode advisory branch).test_validate_binding_with_spec_path_rejectedtest_both_flags_rejected(mutual exclusion)test_location_threaded_through_orchestrator—--location=EUreachesbuild_ontology_graph.256/256 tests pass across
test_binding_validation.py,test_cli.py,test_ontology_orchestrator.py,test_ontology_materializer.py,test_resolved_spec.py(6 live tests skip withoutRUN_LIVE_BIGQUERY_TESTS=1).Docs
New:
docs/ontology/binding-validation.md— covers:ontology-buildintegration examples.FailureCodeenum names and JSONcodevalues, plus a samplejqfilter.Updated:
docs/ontology/ontology-build.md— new "BigQuery location" section covering--locationand a "Pre-flight binding validation" section linking out tobinding-validation.md.Updated:
docs/README.md— linksbinding-validation.mdunder "Ontology Reference."Updated:
CHANGELOG.md—[Unreleased] / Addedentries for the standalone CLI, the twoontology-buildflags,--location, and the Python API.What's NOT in this PR
TestBindingValidationLive); the CLI tests here cover wiring + exit codes + flag threading +AI.GENERATE-not-called-on-failure comprehensively. A literalsubprocess.run(['bq-agent-sdk', 'binding-validate', ...])smoke test could land as a follow-up if anyone wants it._run_binding_preflightand_load_spec_from_args. Both currently load independently; small TOCTOU window. Worth a follow-up helper but not blocking 2b's scope.Working plan position
After this lands, #76 (extracted-graph validator) is the next prereq for #75 C1 per the working plan.