Skip to content

feat(cli): add bq-agent-sdk binding-validate + ontology-build --validate-binding flags (#105 PR 2b)#112

Merged
haiyuan-eng-google merged 3 commits intoGoogleCloudPlatform:mainfrom
caohy1988:feat/binding-validate-cli
May 5, 2026
Merged

feat(cli): add bq-agent-sdk binding-validate + ontology-build --validate-binding flags (#105 PR 2b)#112
haiyuan-eng-google merged 3 commits intoGoogleCloudPlatform:mainfrom
caohy1988:feat/binding-validate-cli

Conversation

@caohy1988
Copy link
Copy Markdown
Contributor

@caohy1988 caohy1988 commented May 4, 2026

Implements #105 PR 2b per the working plan on #96: the user-facing CLI surface and docs. PR 2a (#109, merged) shipped the validator core + unit tests + a live integration test.

What's new in this PR

Standalone CLI: bq-agent-sdk binding-validate

bq-agent-sdk binding-validate \
  --project-id my-project \
  --ontology my.ontology.yaml \
  --binding my-bq-prod.binding.yaml \
  --location US \
  [--strict] \
  [--format json|text|table]
  • Loads ontology + binding, builds a bigquery.Client, calls the validator, prints a structured JSON report on stdout.
  • Failure-code values are emitted as lowercase strings (e.g. "missing_table"); the docs reference table shows both the Python FailureCode enum name and the JSON value.
  • Warnings always print to stderr (one line each) so CI logs surface advisory drift even when stdout is consumed by a JSON-aware tool.
  • Exit codes: 0 ok, 1 failure(s), 2 unexpected error (load failure, missing flags).

ontology-build integration flags

bq-agent-sdk ontology-build ... --validate-binding         # default mode
bq-agent-sdk ontology-build ... --validate-binding-strict  # strict mode
  • When set, run the validator before phase 2 (extraction). On any failure, the build short-circuits before any AI.GENERATE call fires — authoring drift never costs tokens.
  • Default-mode warnings print to stderr but don't block.
  • Mutually exclusive; both incompatible with deprecated --spec-path (validator needs the unresolved Ontology + Binding pair).
  • Helper _run_binding_preflight() centralizes the load + validate + print + exit logic.

Plus: --location on ontology-build

build_ontology_graph() has supported location since PR #108. The CLI was building without forwarding it. This PR threads it through.

Tests

14 new CLI tests, all pass (using patched validators — no live BQ; live coverage is in PR 2a's TestBindingValidationLive):

  • TestBindingValidate (7 tests) — clean validation, failure payload, warnings to stderr (default mode does not flip exit), strict flag threading, missing-flag exit-2, load-failure exit-2, --location threaded to the BQ client.
  • TestOntologyBuildValidateBindingFlag (7 tests):
    • test_validate_binding_short_circuits_on_failure_before_buildasserts build_ontology_graph is never called when validation fails. Extraction never starts; no AI.GENERATE tokens spent.
    • test_validate_binding_strict_short_circuits_on_nullable_keys
    • test_validate_binding_clean_proceeds_to_build
    • test_validate_binding_warnings_only_proceeds_to_buildasserts build_ontology_graph IS called when only warnings are present (default-mode advisory branch).
    • test_validate_binding_with_spec_path_rejected
    • test_both_flags_rejected (mutual exclusion)
    • test_location_threaded_through_orchestrator--location=EU reaches build_ontology_graph.

256/256 tests pass across test_binding_validation.py, test_cli.py, test_ontology_orchestrator.py, test_ontology_materializer.py, test_resolved_spec.py (6 live tests skip without RUN_LIVE_BIGQUERY_TESTS=1).

Docs

New: docs/ontology/binding-validation.md — covers:

  • When to run (binding authoring / CI gating / ontology-build).
  • Standalone CLI examples (default + strict).
  • ontology-build integration examples.
  • Failure-code reference with both Python FailureCode enum names and JSON code values, plus a sample jq filter.
  • CI usage pattern (GitHub Actions example).
  • Python API example.

Updated: docs/ontology/ontology-build.md — new "BigQuery location" section covering --location and a "Pre-flight binding validation" section linking out to binding-validation.md.

Updated: docs/README.md — links binding-validation.md under "Ontology Reference."

Updated: CHANGELOG.md[Unreleased] / Added entries for the standalone CLI, the two ontology-build flags, --location, and the Python API.

What's NOT in this PR

  • Live integration test for the CLI subprocess flow. The validator's live coverage is in PR 2a (TestBindingValidationLive); the CLI tests here cover wiring + exit codes + flag threading + AI.GENERATE-not-called-on-failure comprehensively. A literal subprocess.run(['bq-agent-sdk', 'binding-validate', ...]) smoke test could land as a follow-up if anyone wants it.
  • Refactor to share ontology+binding loading between _run_binding_preflight and _load_spec_from_args. Both currently load independently; small TOCTOU window. Worth a follow-up helper but not blocking 2b's scope.

Working plan position

After this lands, #76 (extracted-graph validator) is the next prereq for #75 C1 per the working plan.

@caohy1988
Copy link
Copy Markdown
Contributor Author

Review fixes folded in (commit 1520ee2)

(1) Failure-code reference now uses lowercase JSON values. The CLI emits f.code.value (e.g. "missing_table"), not the enum attribute name ("MISSING_TABLE"). Docs previously listed the uppercase form, so users copying the table into jq filters would silently match nothing. The reference table now has two columns (Python FailureCode enum + JSON code value) with a sample jq filter using the lowercase form. Also added a one-paragraph note at the top of the failure-code section pointing this out.

(2) CHANGELOG entry added under [Unreleased]. Calls out the four public additions: bq-agent-sdk binding-validate, ontology-build --validate-binding[-strict], ontology-build --location, and the validate_binding_against_bigquery Python API.

(3) Warning-only path now tested. New test_validate_binding_warnings_only_proceeds_to_build patches the validator to return a BindingValidationReport(warnings=(...)) (no failures), runs ontology-build --validate-binding, and asserts:

  • exit code 0
  • build_ontology_graph is called (warnings don't short-circuit)
  • WARN: key_column_nullable appears in the CLI output

This covers _run_binding_preflight()'s default-mode advisory branch, complementing the existing failure-short-circuit and clean-success tests.

(4) --location tests + docs.

  • TestBindingValidate::test_location_threaded_to_bigquery_client — confirms binding-validate --location=EU constructs the BQ client with location='EU'.
  • TestOntologyBuildValidateBindingFlag::test_location_threaded_through_orchestrator — confirms ontology-build --location=EU forwards to build_ontology_graph.
  • docs/ontology/ontology-build.md gained a "BigQuery location" section covering --location, plus a "Pre-flight binding validation" section referencing binding-validation.md.

Verified locally: 256/256 tests pass (test_binding_validation.py, test_cli.py, test_ontology_orchestrator.py, test_ontology_materializer.py, test_resolved_spec.py). Autoformat clean.

Optional nit (deferred): _run_binding_preflight loads ontology + binding once for validation; _load_spec_from_args reloads them for resolve(). A small helper returning the pair once would close the small TOCTOU window, but acceptable as-is for PR 2b scope. Worth a follow-up if anyone wants it.

caohy1988 added 3 commits May 4, 2026 23:34
…oogleCloudPlatform#105 PR 2b)

Closes the user-facing surface of issue GoogleCloudPlatform#105. Builds on PR 2a's
validate_binding_against_bigquery core (shipped in PR GoogleCloudPlatform#109).

CLI changes (src/bigquery_agent_analytics/cli.py)

  Standalone command:
    bq-agent-sdk binding-validate
      --project-id ...
      --ontology X.yaml
      --binding Y.yaml
      [--location US]
      [--strict]
      [--format json|text|table]

  Loads the ontology + binding via upstream loaders, builds a
  google.cloud.bigquery.Client, calls
  validate_binding_against_bigquery(...), prints a structured JSON
  report on stdout. Warnings always print to stderr (one line each)
  so CI logs surface advisory drift even when stdout is consumed by
  a JSON-aware tool.

  Exit codes: 0 ok, 1 failure(s), 2 unexpected error (load failure,
  missing flags, etc).

  ontology-build flags:
    --validate-binding         (default-mode pre-flight)
    --validate-binding-strict  (strict-mode pre-flight)

  When set, run the validator before phase 2 (extraction). On any
  failure, the build short-circuits BEFORE any AI.GENERATE call
  fires, so authoring drift never costs tokens. Default-mode
  warnings print to stderr but do not block.

  The two flags are mutually exclusive; both are incompatible with
  the deprecated --spec-path form because the validator needs the
  unresolved Ontology + Binding pair, not a combined GraphSpec.

  Helper _run_binding_preflight() centralizes the load + validate +
  print + exit logic so it can be invoked from ontology-build
  without duplicating the flow.

  Also: --location flag added to ontology-build, threaded through
  to build_ontology_graph() (which has supported it on the Python
  side since PR GoogleCloudPlatform#108 / commit 292320b). The CLI was previously
  building without forwarding it.

Tests (tests/test_cli.py)

  TestBindingValidate (6 tests):
    - test_clean_validation_exits_zero
    - test_failures_exit_one_with_payload
    - test_warnings_print_to_stderr_but_do_not_flip_exit
    - test_strict_flag_threaded_through
    - test_missing_required_flags_exit_2
    - test_load_failure_exits_two

  TestOntologyBuildValidateBindingFlag (5 tests):
    - test_validate_binding_short_circuits_on_failure_before_build
      (asserts build_ontology_graph is NEVER called when validation
      fails — extraction does not start)
    - test_validate_binding_strict_short_circuits_on_nullable_keys
    - test_validate_binding_clean_proceeds_to_build
    - test_validate_binding_with_spec_path_rejected
    - test_both_flags_rejected (mutual exclusion)

  All tests use patched validators (no live BQ); the live
  integration test for the full flow lives in PR 2a (GoogleCloudPlatform#109).

  253/253 tests pass across the touched test files.

Docs

  New: docs/ontology/binding-validation.md
    - When to run (binding authoring, CI gating, ontology-build).
    - Standalone CLI examples (default + strict).
    - ontology-build integration examples.
    - Failure-code reference table covering all 7 default codes
      plus the strict-only KEY_COLUMN_NULLABLE.
    - CI usage pattern (GitHub Actions).
    - Python API example.
    - Cross-links to ontology-build.md, binding.md, and GoogleCloudPlatform#76's
      planned post-extraction validator.

  Updated: docs/README.md
    - Added binding-validation.md to the Ontology Reference table.

What's NOT in this PR

  - Live integration test for the CLI surface (the validator
    itself has live coverage from PR 2a's
    TestBindingValidationLive). A literal subprocess-style CLI
    smoke test could land as a follow-up if someone wants it, but
    the unit tests cover wiring + exit codes + flag threading
    comprehensively.
…ly paths (GoogleCloudPlatform#105 PR 2b)

Four review findings folded in.

(1) Failure-code reference now shows lowercase JSON values

The CLI emits f.code.value (e.g. 'missing_table'), not the enum
attribute name ('MISSING_TABLE'). Docs previously listed the
uppercase form, so users copying the table into jq filters would
silently match nothing. The reference table now has both columns
(Python FailureCode + JSON value), with a sample jq filter using
the lowercase form.

(2) CHANGELOG entry under [Unreleased]

This PR adds a public CLI command (binding-validate), two public
ontology-build flags (--validate-binding[-strict]), --location on
ontology-build, and the validate_binding_against_bigquery Python
API. All four are now called out under [Unreleased] / Added.

(3) Test the warning-only path through ontology-build

The new test test_validate_binding_warnings_only_proceeds_to_build
patches the validator to return a BindingValidationReport with
only warnings (no failures), runs ontology-build --validate-binding,
and asserts:
  - exit code 0
  - build_ontology_graph IS called (warnings don't short-circuit)
  - WARN: key_column_nullable appears in the CLI output

This covers _run_binding_preflight()'s default-mode advisory
branch, complementing the existing failure-short-circuit and
clean-success tests.

(4) --location now has tests and a doc note

Two new tests:
  - TestBindingValidate::test_location_threaded_to_bigquery_client
    confirms binding-validate --location=EU constructs a BQ client
    with location='EU'.
  - TestOntologyBuildValidateBindingFlag::
    test_location_threaded_through_orchestrator confirms
    ontology-build --location=EU forwards to build_ontology_graph
    so the orchestrator's BQ client targets the EU multi-region.

docs/ontology/ontology-build.md: new 'BigQuery location' section
covering --location plus a 'Pre-flight binding validation' section
referencing binding-validation.md for the full --validate-binding
flag reference.

256/256 tests pass (test_binding_validation.py + test_cli.py +
test_ontology_orchestrator.py + test_ontology_materializer.py +
test_resolved_spec.py). Autoformat clean.

Optional nit (deferred, not blocking): _run_binding_preflight
loads ontology + binding once for validation, _load_spec_from_args
reloads them for resolve(). Worth a small helper that returns the
loaded objects once, but acceptable as-is for PR 2b scope.
…loudPlatform#105 PR 2b)

Two wording fixes; no behavior change.

The validator uses bq_client.get_table(...) metadata calls
(binding_validation.py:309), not direct INFORMATION_SCHEMA queries.
Two places said the latter:

(1) docs/ontology/ontology-build.md: 'pre-flight validator queries
INFORMATION_SCHEMA in the same region' -> 'uses the BigQuery client
with the requested location to fetch each bound table's metadata'.

(2) tests/test_cli.py docstring on
test_location_threaded_to_bigquery_client: same correction. The
test asserts the location threads through; the wording for *why*
location matters now matches the actual mechanism.

256/256 tests still pass. Autoformat clean.
@caohy1988 caohy1988 force-pushed the feat/binding-validate-cli branch from 0ae07e4 to 828378d Compare May 5, 2026 06:34
@caohy1988
Copy link
Copy Markdown
Contributor Author

Rebased onto fresh main (PR 2a merged)

#109 (PR 2a) merged at upstream 8f8a36f. Rebased this branch with git rebase --onto upstream/main 6a14aa2 to drop the 6 PR 2a commits that are now on main, leaving only the 3 PR 2b commits:

828378d docs: avoid naming INFORMATION_SCHEMA as preflight mechanism (#105 PR 2b)
d4b1740 fix+test+docs: lowercase JSON codes, changelog, location + warning-only paths (#105 PR 2b)
7b032dd feat(cli): add bq-agent-sdk binding-validate + ontology-build flags (#105 PR 2b)

Force-pushed via --force-with-lease. The PR diff now shows only the CLI + docs additions.

Verified locally: 175/175 tests pass across test_binding_validation.py, test_cli.py, test_ontology_orchestrator.py (live integration tests skip without RUN_LIVE_BIGQUERY_TESTS=1).

PR body opening paragraph removed ("stacked on #109" note no longer applies).

@haiyuan-eng-google haiyuan-eng-google merged commit 4c3a293 into GoogleCloudPlatform:main May 5, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants