v0.14.0

Highlights

This release adds new extractors for incident management and infrastructure platforms, new cloud storage sinks, an OpenAPI/protobuf extractor, and significant CLI improvements for developer experience.

New Features

• PagerDuty extractor — Extract service and incident metadata with owned_by and belongs_to edges. Configurable incident lookback window.
• OpsGenie extractor — Extract service and incident metadata. Supports EU instances via base_url config.
• Kubernetes extractor — Extract cluster infrastructure topology (namespaces, deployments, services, jobs) with belongs_to edges. Uses raw HTTP with kubeconfig or in-cluster auth — no client-go dependency.
• OpenAPI extractor — Parse OpenAPI v2/v3 specs and protobuf .proto files, emitting API entities with endpoints, schemas, and services metadata. Supports local files (with globs), URLs, and auto-detection.
• S3 sink — Write NDJSON to Amazon S3 or S3-compatible storage (MinIO, DigitalOcean Spaces). Supports explicit credentials or default AWS credential chain.
• Azure Blob sink — Write NDJSON to Azure Blob Storage with shared key or connection string authentication.

Improvements

• CLI dry-run and limit — --dry-run and --limit flags on the run command for testing recipes without sending to sinks.
• CLI log-level override — --log-level flag on run and lint commands.
• JSON output for CLI — --format json flag on plugins list/info, entities, and edges commands for scriptable output.
• Richer run summary — Entity type breakdown and inline errors in run output.
• Better error messages — Human-readable validation errors, "did you mean?" suggestions for misspelled plugin names, actionable hints.

Bug Fixes

• Fix Node.js version pin in Vercel config for docs deployment.
• Fix HTTP extractor error tests for human-readable validation messages.

Documentation

• Fix CLI command references across all docs (list → plugins list, new recipe → recipe init, gen → recipe gen).
• Add complete flag documentation for all commands.
• Fix broken contribution links in 41 plugin READMEs.
• Add missing extractors and sinks to reference docs.
• Remove non-existent config fields and fix incorrect examples.

Dependencies

• Upgrade chronicle version.

Full Changelog: v0.13.0...v0.14.0

v0.13.0

Highlights

This release introduces structured entity/edge metadata, a restructured CLI, and significant improvements to config validation and data consistency.

New Features

• Structured entity/edge metadata — All 31 extractors now declare their entity types, edge types, and URN patterns via plugins.Info.
• Restructured CLI — New command structure: plugins list/info, recipe init/gen, top-level entities and edges commands with filtering support (--entity-type, --edge-type, --tag).
• Notion extractor — Extract page and database metadata from Notion.
• Confluence extractor — Extract page metadata and relationships from Confluence spaces.
• dbt extractor — Extract model and source metadata from dbt manifests.
• GitHub collaborator permissions — Extract collaborator access levels.
• BigQuery label-based exclusion — Exclude tables by label key-value pairs.

Improvements

• Config validation tightened — Extract fields validated with oneof, URL format validation on host fields, file existence checks, port ranges, conditional auth field requirements (Kafka TLS/SASL).
• Edge serialization fixed — Kafka and HTTP sinks now include edges in output (previously silently dropped).
• Tags standardized — All plugins use consistent platform/domain taxonomy (e.g. gcp/database, oss/bi, saas/collaboration).
• Descriptions standardized — Consistent format across all plugins.
• Exclude patterns standardized — Postgres, CouchDB, Redshift, and Presto migrated from comma-separated strings to structured exclude configs.
• pkg/errors migrated to stdlib — All direct usage replaced with fmt.Errorf and %w wrapping.

Bug Fixes

• Redshift exclude — The exclude config field was defined but never actually used to filter databases. Now wired up correctly.

Breaking Changes

• CLI commands renamed — list extractors → plugins list --type extractor, info extractor <name> → plugins info <name>, new recipe → recipe init, gen → recipe gen.
• Exclude config format — Postgres, CouchDB, Redshift, and Presto now use structured exclude (e.g. exclude: { databases: [db1] }) instead of comma-separated strings.
• Kafka sink output format — Messages are now JSON (entity + edges) instead of protobuf entity-only.

Full Changelog: v0.12.0...v0.13.0

v0.12.0

Highlights

Adds MaxCompute support for Tableau and Optimus, concurrent Compass sink processing, and major dependency upgrades.

New Features

• MaxCompute for Tableau — Add MaxCompute support for Tableau extractor.
• MaxCompute for Optimus — Add MaxCompute support for Optimus extractor and add project_id.
• Compass concurrency — Process Compass sink concurrently and improve lineage handling.
• HTTP in Tengo — Add HTTP support for Tengo script processor.

Improvements

• Salt upgrade — Upgraded raystack/salt from v0.3.2 to v0.7.0.

Bug Fixes

• Fixed test failures from dependency upgrades.
• Fixed presto-go-client EOF handling as successful row iteration.
• Fixed run command to return correct exit code on recipe failure.

Documentation

• Comprehensive documentation improvements and fixes.
• Added roadmap and fixed incorrect recipe commands.

Dependencies

• Upgraded dependencies to latest versions.

Full Changelog: v0.11.0...v0.12.0

v0.11.0

Highlights

Major improvements to Kafka extractor, BigQuery concurrency, HTTP sink templating, and Merlin authentication.

New Features

• Kafka auth — Add authentication and authorization support for Kafka extractor.
• Sink batching — Add batch size config for sink concurrency.
• HTTP before_script — Add before_script hook before making HTTP requests.
• BigQuery concurrency — Configurable concurrency for extracting assets from BigQuery.
• HTTP sink templating — Support templating for HTTP sinks.

Improvements

• Merlin — Replaced access token with ID token in Merlin extractor.
• BigQuery — Do not send preview field and rows if max_preview_rows is -1.

Bug Fixes

• Fixed multiple Kafka extractor issues.

Full Changelog: v0.10.1...v0.11.0

v0.10.1

Highlights

Build fix to include Darwin (macOS) in release artifacts.

Dependencies

• Added Darwin platform to release builds.

Full Changelog: v0.10.0...v0.10.1

v0.10.0

Highlights

BigQuery enhancements for profiling, taxonomy, and SQL query capture, plus CLI command grouping.

New Features

• BigQuery profiling — Store total rows when profiling tables.
• BigQuery taxonomy — Add taxonomy name along with policy tag name for columns.
• BigQuery SQL — Add SQL query for VIEW and MATERIALIZED VIEW in asset data.
• CLI groups — Group CLI commands for better discoverability.

Bug Fixes

• Fixed unhandled parse error for BigQuery table FQN.

Dependencies

• Upgraded goreleaser to v2.

Full Changelog: v0.9.2...v0.10.0

v0.9.2

Highlights

Adds BigQuery view lineage and OpenTelemetry instrumentation for the BigQuery extractor.

New Features

• BigQuery lineage — Add upstream lineage dependencies for VIEW and MATERIALIZED VIEW in BigQuery extractor.
• BigQuery telemetry — Instrument BigQuery extractor with OpenTelemetry.

Full Changelog: v0.9.1...v0.9.2

v0.9.1

Highlights

Comprehensive OpenTelemetry instrumentation, extractor retry support, and Frontier rename.

New Features

• OpenTelemetry — Instrument gRPC interceptor, HTTP clients, BigTable extractor, and SQL-based extractors with OpenTelemetry.
• Extractor retries — Add support for retries on extractor errors.
• BigQuery paging — Add explicit paging to BigQuery extractor with global seeded random generator.
• Service account auth — Authentication with base64-encoded service account.

Improvements

• Frontier — Renamed Shield to Frontier and updated dependencies.
• BigQuery — Random mixing values for extraction ordering.

Bug Fixes

• Fixed empty destination handling in tasks.
• Fixed Optimus emitting date as nil when value is empty string.
• Fixed duration not being correctly set in Run instance.
• Fixed Merlin model URN construction to use name instead of ID.
• Fixed unpopulated fields not being emitted for asset data.

Full Changelog: v0.9.0...v0.9.1

v0.9.0

Highlights

Rebrands to Raystack and adds crawler specifications to extractor plugins.

New Features

• Crawler specifications — Add crawler specifications to extractor plugins.

Improvements

• Raystack — Renamed project and packages from ODPF to Raystack.

Full Changelog: v0.8.0...v0.9.0

v0.8.0

Highlights

Adds support for nested field schema capture in BigQuery.

New Features

• BigQuery nested fields — Add support to capture schema of nested fields.

Bug Fixes

• Fixed asset labels not being sent to Compass sink.

Full Changelog: v0.7.4...v0.8.0