Skip to content

feat: Onboard bigquery-data-analytics to Evalbench CI pipeline#123

Merged
omkargaikwad23 merged 5 commits intomainfrom
release-please--evalbench-ci-onboarding
Apr 28, 2026
Merged

feat: Onboard bigquery-data-analytics to Evalbench CI pipeline#123
omkargaikwad23 merged 5 commits intomainfrom
release-please--evalbench-ci-onboarding

Conversation

@omkargaikwad23
Copy link
Copy Markdown
Contributor

This PR adds the necessary configuration files to enable automated evaluations for the bigquery-data-analytics extension repository using the evalbench framework whenever release is triggered. This setup is modeled after the existing pipeline in the cloud-sql-postgresql repository.

Key Changes

cloudbuild.yaml:

Added to the root directory to handle the evaluation workflow on release branches. It fetches PR data and triggers the standalone evaluation.

evals/ directory:

Created with all necessary configurations:

  1. ci_metadata.yaml: Tracks the extension ID and release version.
  2. dataset.json: Contains sample BigQuery-specific evaluation scenarios (bq-search-catalog and bq-ask-insights).
  3. gemini_2.5_pro_model.yaml: Configures the Gemini 2.5 Pro model for scoring and simulation.
  4. model_config.yaml: Configures the environment and setup for the BigQuery extension.
  5. run_config.yaml: The main experiment configuration file for Evalbench.
  6. substitute_env.py: A helper script to substitute environment variables in the YAML configs.

…stom dataset, model configurations, and CI integration.
@omkargaikwad23 omkargaikwad23 requested review from a team as code owners April 28, 2026 06:17
@omkargaikwad23 omkargaikwad23 added the autorelease: triggered Release please has triggered a release for this. label Apr 28, 2026
@github-actions github-actions Bot requested a review from jiaxunwu April 28, 2026 06:18
@omkargaikwad23 omkargaikwad23 added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Apr 28, 2026
@omkargaikwad23 omkargaikwad23 removed autorelease: triggered Release please has triggered a release for this. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Apr 28, 2026
@omkargaikwad23 omkargaikwad23 changed the title feat: Onboard bigquery-data-analytics to Evalbench CI pipeline release 0.0.0 feat: Onboard bigquery-data-analytics to Evalbench CI pipeline Apr 28, 2026
@prernakakkar-google
Copy link
Copy Markdown

Fix lint checks

Comment thread evals/run_config.yaml
@omkargaikwad23 omkargaikwad23 merged commit db1f480 into main Apr 28, 2026
12 of 13 checks passed
@omkargaikwad23 omkargaikwad23 deleted the release-please--evalbench-ci-onboarding branch April 28, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants