Releases: GoogleCloudPlatform/DataflowTemplates
Releases · GoogleCloudPlatform/DataflowTemplates
Dataflow Templates 2026-04-21-00_RC00
[DataStreamToSpanner] Increase condition wait timeout by 5 minutes to…
Dataflow Templates 2026-04-14-00_RC00
What's Changed
Additional New Features
- Adding schema loader transform to cdc data generator by @shreyakhajanchi in #3483
- BQ Anomaly detection: Reshuffle fanout in cdc by @claudevdm in #3621
- Make anomaly message configurable by @claudevdm in #3623
- Handle sliding window anomaly detection by @claudevdm in #3622
- Added support for PostgreSQL dialect in reverse replication by @shreyakhajanchi in #3619
- add cdc embedded connector it by @derrickaw in #3639
- Adding support for PostgreSQL as source in reverse replication by @shreyakhajanchi in #3638
- Added logic for dag creation and weighted table selection by @shreyakhajanchi in #3643
Bug fixes
- Update WriteDataChangeRecordsToAvro to support array type. by @jiangzzhu in #3558
- Fix DLQ infinite retry loop via hybrid consumption model by @aasthabharill in #3535
- Remove the default settings.xml to address the credentials decryption failure by @chamikaramj in #3583
- Add unknown exceptions as retryable to DatastreamResourceManager by @aasthabharill in #3602
- Fix machine size in flaky load tests by @aasthabharill in #3585
- Add IT for csv import template - for case when spanner write fails by @aasthabharill in #3606
- [GCS_Avro_to_Cloud_Spanner] Fix GSQL secondary index clause ordering (WHERE before INTERLEAVE IN) by @atask-g in #3595
- [kafka] - remove kafka from v1 pom and add check to kafka workflow by @derrickaw in #3628
- Add IT to test bulk migration using transformation with dropped column in Spanner by @aasthabharill in #3607
- Allowlist
score_versionfor export/import. by @ZhitongZhang in #3620 - Allow empty string to be used for the source and destination database by @le-michael in #3645
- [spanner-to-sourcedb] Add Integration Tests for retryDLQ and retryAllDLQ mode for sharded and non-sharded setup by @aasthabharill in #3564
Improvements
- Error handling for accidentally providing a Pub/Sub Topic rather than MQTT topic by @svetakvsundhar in #3555
- Added write to GCS incase of failed mutations - Import template by @shreyakhajanchi in #3519
- Add README for validation template by @manitgupta in #3584
- Optimize Kafka sink producer configs for improved batching efficiency by @kir-titievsky-google in #3481
- Upgrade to Beam 2.72.0 by @Abacn in #3557
- remove explicit google ads version by @derrickaw in #3634
- Add pipeline option for Spanner emulator support in SpannerChangeStreamsToPubSub by @hunsche in #3545
- Add parameter to set maxNumRows for dataflow jobs by @swatilodha-dev in #3632
- Update Apache Beam version in upload-jar workflow by @ahmedabu98 in #3659
- upgrade Debezium to 3.3.2 by @derrickaw in #3336
- fix neo4j upgrade issues and removed redundant property by @derrickaw in #3665
- Add relative change detector by @claudevdm in #3624
New Templates
- PubSub to ClickHouse Flex Template by @BentsiLeviav in #3458
New Contributors
- @lszinv made their first contribution in #3265
- @kir-titievsky-google made their first contribution in #3481
- @hunsche made their first contribution in #3545
- @ZhitongZhang made their first contribution in #3620
- @swatilodha-dev made their first contribution in #3632
- @le-michael made their first contribution in #3645
Full Changelog: 2026-03-24-00_RC00...2026-04-14-00_RC00
Contributors
lszinv, derrickaw, and 16 other contributors
Assets 2
Dataflow Templates 2026-04-07-00_RC03
What's Changed
Additional New Features
- Adding schema loader transform to cdc data generator by @shreyakhajanchi in #3483
- BQ Anomaly detection: Reshuffle fanout in cdc by @claudevdm in #3621
- Make anomaly message configurable by @claudevdm in #3623
- Handle sliding window anomaly detection by @claudevdm in #3622
- Added support for PostgreSQL dialect in reverse replication by @shreyakhajanchi in #3619
Bug fixes
- Update WriteDataChangeRecordsToAvro to support array type. by @jiangzzhu in #3558
- Fix DLQ infinite retry loop via hybrid consumption model by @aasthabharill in #3535
- Remove the default settings.xml to address the credentials decryption failure by @chamikaramj in #3583
- Add unknown exceptions as retryable to DatastreamResourceManager by @aasthabharill in #3602
- Fix machine size in flaky load tests by @aasthabharill in #3585
- Add IT for csv import template - for case when spanner write fails by @aasthabharill in #3606
- [GCS_Avro_to_Cloud_Spanner] Fix GSQL secondary index clause ordering (WHERE before INTERLEAVE IN) by @atask-g in #3595
- [kafka] - remove kafka from v1 pom and add check to kafka workflow by @derrickaw in #3628
- Add IT to test bulk migration using transformation with dropped column in Spanner by @aasthabharill in #3607
Improvements
- Error handling for accidentally providing a Pub/Sub Topic rather than MQTT topic by @svetakvsundhar in #3555
- Added write to GCS incase of failed mutations - Import template by @shreyakhajanchi in #3519
- Add README for validation template by @manitgupta in #3584
- Optimize Kafka sink producer configs for improved batching efficiency by @kir-titievsky-google in #3481
- Upgrade to Beam 2.72.0 by @Abacn in #3557
- remove explicit google ads version by @derrickaw in #3634
- Add pipeline option for Spanner emulator support in SpannerChangeStreamsToPubSub by @hunsche in #3545
New Templates
- PubSub to ClickHouse Flex Template by @BentsiLeviav in #3458
New Contributors
- @pabloqc made their first contribution in #3396
- @lszinv made their first contribution in #3265
- @kir-titievsky-google made their first contribution in #3481
- @hunsche made their first contribution in #3545
Full Changelog: 2026-03-24-00_RC00...2026-04-07-00_RC02
What's Changed
Additional New Features
- Adding schema loader transform to cdc data generator by @shreyakhajanchi in #3483
- BQ Anomaly detection: Reshuffle fanout in cdc by @claudevdm in #3621
- Make anomaly message configurable by @claudevdm in #3623
- Handle sliding window anomaly detection by @claudevdm in #3622
- Added support for PostgreSQL dialect in reverse replication by @shreyakhajanchi in #3619
Bug fixes
- Update WriteDataChangeRecordsToAvro to support array type. by @jiangzzhu in #3558
- Fix DLQ infinite retry loop via hybrid consumption model by @aasthabharill in #3535
- Remove the default settings.xml to address the credentials decryption failure by @chamikaramj in #3583
- Add unknown exceptions as retryable to DatastreamResourceManager by @aasthabharill in #3602
- Fix machine size in flaky load tests by @aasthabharill in #3585
- Add IT for csv import template - for case when spanner write fails by @aasthabharill in #3606
- [GCS_Avro_to_Cloud_Spanner] Fix GSQL secondary index clause ordering (WHERE before INTERLEAVE IN) by @atask-g in #3595
- [kafka] - remove kafka from v1 pom and add check to kafka workflow by @derrickaw in #3628
- Add IT to test bulk migration using transformation with dropped column in Spanner by @aasthabharill in #3607
Improvements
- Error handling for accidentally providing a Pub/Sub Topic rather than MQTT topic by @svetakvsundhar in #3555
- Added write to GCS incase of failed mutations - Import template by @shreyakhajanchi in #3519
- Add README for validation template by @manitgupta in #3584
- Optimize Kafka sink producer configs for improved batching efficiency by @kir-titievsky-google in #3481
- Upgrade to Beam 2.72.0 by @Abacn in #3557
- remove explicit google ads version by @derrickaw in #3634
- Add pipeline option for Spanner emulator support in SpannerChangeStreamsToPubSub by @hunsche in #3545
New Templates
- PubSub to ClickHouse Flex Template by @BentsiLeviav in #3458
New Contributors
- @pabloqc made their first contribution in #3396
- @lszinv made their first contribution in #3265
- @kir-titievsky-google made their first contribution in #3481
- @hunsche made their first contribution in #3545
Full Changelog: 2026-03-24-00_RC00...2026-04-07-00_RC03
Contributors
pabloqc, lszinv, and 13 other contributors
Assets 2
Dataflow Templates 2026-03-24-00_RC00
What's Changed
Additional New Features
Bug fixes
- Pin zookeeper. by @claudevdm in #3498
- [yaml] - fix out of sync template by @derrickaw in #3527
- update SQL usage by @derrickaw in #3528
Improvements
- [yaml] - update yaml-pr workflow to check for diff in generated templates by @derrickaw in #3520
New Templates
- Add BigQuery Anomaly Detection pipeline by @claudevdm in #3532
- Add bqmonitor extras: CI, IT, precombine, validate topic path by @claudevdm in #3546
New Contributors
Full Changelog: 2026-03-17-00_RC00...2026-03-24-00_RC00
What's Changed
Additional New Features
Bug fixes
- Pin zookeeper. by @claudevdm in #3498
- [yaml] - fix out of sync template by @derrickaw in #3527
- update SQL usage by @derrickaw in #3528
Improvements
- [yaml] - update yaml-pr workflow to check for diff in generated templates by @derrickaw in #3520
New Templates
- Add BigQuery Anomaly Detection pipeline by @claudevdm in #3532
- Add bqmonitor extras: CI, IT, precombine, validate topic path by @claudevdm in #3546
New Contributors
Full Changelog: 2026-03-17-00_RC00...2026-03-24-00_RC00
Contributors
derrickaw, ihebimad, and 2 other contributors
Assets 2
Dataflow Templates 2026-03-17-00_RC00
What's Changed
Additional New Features
- Adding model classes for data generator by @shreyakhajanchi in #3442
Bug fixes
- Fix Transformation Failed for Dropped Columns by @aasthabharill in #3466
- fix pipeline-yaml flex container name by @derrickaw in #3480
Improvements
- more pom properties refactoring by @derrickaw in #3434
- add current yaml templates to main readme by @derrickaw in #3475
- feat(dv): main template implementation by @manitgupta in #3473
- [yaml] - remove extra parameters not used in Postgres transforms by @derrickaw in #3492
- Fix DLQWriteTransform: DLQ file overwrites by adding window token to shard template by @darshan-sj in #3465
- [yaml] - change Sql -> SQL by @derrickaw in #3506
Full Changelog: 2026-03-10-00_RC00...2026-03-17-00_RC00
Contributors
derrickaw, manitgupta, and 3 other contributors
Assets 2
Dataflow Templates 2026-03-10-00_RC00
What's Changed
Additional New Features
- Added foreign key and indexes in spanner-common to be consumed by data generator pipeline by @shreyakhajanchi in #3438
- Dataflow Template for Firestore to Firestore Documents Copy (Clone of PR 3098) by @jingqizz in #3392
Bug fixes
- fix spotless error by @derrickaw in #3451
Improvements
- feat(dv): source and spanner reading PTransforms by @manitgupta in #3439
- dedup of surefire version by @derrickaw in #3456
- add gcs-spanner-dv path to spanner-pr.yaml workflow by @derrickaw in #3455
- feat(dv): records matching and reporting PTransforms by @manitgupta in #3440
- [yaml] - minor change to Yaml display names by @derrickaw in #3460
Package Upgrades
- Pin dependency io.airlift:aircompres due to a vulnerability by @chamikaramj in #3414
New Contributors
Full Changelog: 2026-03-03-00_RC00...2026-03-10-00_RC00
Contributors
derrickaw, manitgupta, and 3 other contributors
Assets 2
Dataflow Templates 2026-03-03-00_RC00
What's Changed
Additional New Features
- Add support for UDF function transformations in PubSubToRedis template by @wladabutenko in #3338
Bug fixes
- yaml/python workflow update - add missing pom file change detection by @derrickaw in #3374
- Add metadata for sharding in Bulk Migration and relax change event column verification in Live Migration by @aasthabharill in #3402
- Investigate counters in reverse replication job by @aasthabharill in #3134
- upgrade go to 1.25 to fix cicd issue by @derrickaw in #3413
- Fix TemplatesReleaseMojo yaml folder structure and upload paths by @derrickaw in #3368
Improvements
- DatastreamResourceManager - improving resilience by adding retries by @darshan-sj in #3387
- [StreamingDataGenerator] Fix IllegalStateException: zip file closed in StreamingDataGenerator by @darshan-sj in #3422
- refactor more pom properties by @derrickaw in #3388
- feat(Spanner): Add support for MySQL generated columns in Spanner reverse replication template. by @pratickchokhani in #3306
New Templates
- feat(dv): Reporting & Read Dofns by @manitgupta in #3400
- feat(dv): hashing and schema processing by @manitgupta in #3401
Full Changelog: 2026-02-23-00_RC00...2026-03-03-00_RC00
Contributors
derrickaw, manitgupta, and 4 other contributors
Assets 2
Dataflow Templates 2026-02-23-00_RC00
What's Changed
Additional New Features
Bug fixes
- fix beam snapshot workflow by @derrickaw in #3326
- SpannerToCassandraSourceLT fails with FileNotFoundException for session.json by @aasthabharill in #3342
- Fix timeout types of bigtable export job by @mutianf in #3389
Improvements
- raise avro version to 1.12.1 and aggregate property usage by @derrickaw in #3334
- Initialize GCS Spanner Validation Template by @manitgupta in #3318
- upgrade embedded cassandra to 5.0.3 by @derrickaw in #3335
- raise astra version to 2.1.4 by @derrickaw in #3333
- Refactor Github Action per b/485167538 by @google-admin in #3351
- Update Cloud_Bigtable_to_GCS_SequenceFile with new options by @mutianf in #3386
- feat(Spanner): Live Migration: Remove generated columns from change events during transformation and add tests for this behavior by @pratickchokhani in #3255
New Templates
- feat(dv): visitor pattern hashing utilities by @manitgupta in #3377
- feat(dv): schema mappers and validation combiners by @manitgupta in #3378
New Contributors
- @google-admin made their first contribution in #3351
Full Changelog: 2026-02-17-00_RC00...2026-02-23-00_RC00
Contributors
google-admin, derrickaw, and 4 other contributors
Assets 2
Dataflow Templates 2026-02-17-00_RC00
What's Changed
Bug fixes
- fix: fix file descriptor leak due to unclosed driver by @fbiville in #3320
- AstraDBToBigQueryIT fix - different region and dataset issues by @derrickaw in #3323
- [sourcedb-to-spanner, datastream-to-spanner] Bulk Migration DLQ uses Spanner Mutations but Live Retry consumes it as Source Row by @aasthabharill in #3259
Improvements
- fix links to other workflows by @derrickaw in #3327
- Update GCS Sink writer to include additional metadata by @manitgupta in #3317
- feat(Spanner): Add workflow dispatch options to run specific test suites or individual tests within Spanner PR by @pratickchokhani in #3337
Package Upgrades
Full Changelog: 2026-02-10-00_RC02...2026-02-17-00_RC00
Contributors
fbiville, derrickaw, and 3 other contributors
Assets 2
Dataflow Templates 2026-02-10-00_RC02
What's Changed
Bug fixes
- [Datastream] - Update gcs subscriber and file pattern as optional by @yair-harel in #3307
Improvements
- Fix CVE-2024-25710 by @tarun-google in #3297
- feat(Spanner): Bulk template generated column support by @pratickchokhani in #3254
Full Changelog: 2026-02-03-00_RC01...2026-02-10-00_RC00
Contributors
pratickchokhani, yair-harel, and tarun-google