41 Pull requests merged by 21 people

• fix the version string checks in KafkaIO

  #35703 merged Jul 27, 2025

• Upgrade base image for dev-support docker image(Ubuntu 20.04 -> Ubuntu 24.04)

  #35694 merged Jul 26, 2025

• sdks/python/scripts: support pytest user markers

  #35655 merged Jul 25, 2025

• Cloudsql pg colab

  #35690 merged Jul 25, 2025

• Fix flink runner test

  #34913 merged Jul 25, 2025

• Cherrypick: Remove tox dependency

  #35687 merged Jul 25, 2025

• Remove tox dependency

  #35679 merged Jul 25, 2025

• Update README.md

  #35689 merged Jul 25, 2025

• Bump github.com/go-sql-driver/mysql from 1.9.2 to 1.9.3 in /sdks

  #35365 merged Jul 25, 2025

• Bump golang.org/x/oauth2 from 0.12.0 to 0.27.0 in /.test-infra/mock-apis

  #35630 merged Jul 25, 2025

• docs: update known issues in CHANGES.md with YAML Flatten bug (#35678)

  #35681 merged Jul 25, 2025

• docs: update known issues in CHANGES.md with YAML Flatten bug

  #35678 merged Jul 24, 2025

• [release-2.67] Cherrypick #35669 onto the release branch

  #35674 merged Jul 24, 2025

• Update Vertex AI embedding handlers to use RemoteModelHandler

  #35670 merged Jul 24, 2025

• Allow setting BatchSize and MaxBufferingDuration in JDBCIO's WriteWithResults

  #35669 merged Jul 24, 2025

• Fix vendor Calcite 1.40

  #35671 merged Jul 23, 2025

• vendor calcite 1.40

  #35661 merged Jul 23, 2025

• fix dicomio tag mismatch (#30760)

  #35658 merged Jul 23, 2025

• Bump google.golang.org/api from 0.241.0 to 0.243.0 in /sdks

  #35663 merged Jul 23, 2025

• Dont fail pipeline on failed stat

  #35640 merged Jul 23, 2025

• Fix jdbc logical type issues when running a yaml pipeline.

  #35659 merged Jul 23, 2025

• [YAML] A Streaming Inference Pipeline - YouTube Comments Sentiment Analysis

  #35375 merged Jul 22, 2025

• Fix failed rows conversion missing 'as_dict' error

  #35533 merged Jul 22, 2025

• Bump github.com/testcontainers/testcontainers-go from 0.37.0 to 0.38.0 in /sdks

  #35615 merged Jul 22, 2025

• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.84.0 to 1.84.1 in /sdks

  #35650 merged Jul 22, 2025

• Bump form-data from 4.0.3 to 4.0.4 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel

  #35654 merged Jul 22, 2025

• [Python] Implement combiner deferred side inputs

  #35601 merged Jul 22, 2025

• [Beam SQL] Support DATABASE concept in Beam SQL, with implementation for Iceberg

  #35641 merged Jul 22, 2025

• [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247)

  #35558 merged Jul 22, 2025

• Fix PostCommit Python Xlang IO Dataflow job

  #35638 merged Jul 22, 2025

• Fix IcebergIO Integration Tests job

  #35653 merged Jul 22, 2025

• Distinguishing bigquery logging failures severities

  #35373 merged Jul 22, 2025

• Include WaitOn in wildcard imports

  #35645 merged Jul 21, 2025

• Fix typo

  #35639 merged Jul 21, 2025

• Extend Schema Registry Support on Managed Kafka I/O to Google's Managed Schema Registry Solution

  #35085 merged Jul 21, 2025

• [IcebergIO] robust handling for filtering time types; expose new features to YAML SDK

  #35515 merged Jul 21, 2025

• [BQ]: Update error message for too big of a BQ tablerow

  #35567 merged Jul 21, 2025

• Adding clustering support for python storage write api

  #35526 merged Jul 21, 2025

• Bump github.com/aws/aws-sdk-go-v2/config from 1.29.17 to 1.29.18 in /sdks

  #35634 merged Jul 21, 2025

• build: update gradle wrapper from 8.4 to 8.14.3

  #35624 merged Jul 21, 2025

• Fix out of range

  #35051 merged Jul 20, 2025

32 Pull requests opened by 21 people

• Use unvendored calcite

  #35642 opened Jul 21, 2025

• Exclude META-INF/maven in expansion services

  #35643 opened Jul 21, 2025

• Introduce Schema Registry Functionality to Managed KafkaIO Write.

  #35644 opened Jul 21, 2025

• Add documentation for Wait.On and WaitOn transforms

  #35647 opened Jul 21, 2025

• Add Example for Iceberg REST Catalog CDC

  #35649 opened Jul 21, 2025

• Bump com.diffplug.spotless:spotless-plugin-gradle from 5.6.1 to 7.2.1

  #35651 opened Jul 22, 2025

• Bump com.diffplug.spotless from 5.6.1 to 7.2.1

  #35652 opened Jul 22, 2025

• Add support for lambda name pickling.

  #35656 opened Jul 22, 2025

• Avoid unreasonably long stage names for @ptransform_fn.

  #35660 opened Jul 22, 2025

• Remove ZetaSQL

  #35662 opened Jul 23, 2025

• Bump google.golang.org/grpc from 1.73.0 to 1.74.2 in /sdks

  #35664 opened Jul 23, 2025

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.17.84 to 1.17.85 in /sdks

  #35665 opened Jul 23, 2025

• [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

  #35667 opened Jul 23, 2025

• feat(yaml): add schema unification for Flatten transform

  #35672 opened Jul 23, 2025

• [Java] Further simplify MoreFutures usage in case it is cause of issues with ForkJoin pool stuckness

  #35673 opened Jul 23, 2025

• Reflect that my previous PR was cherrypicked into 2.67

  #35676 opened Jul 24, 2025

• [DO NOT MERGE] Prototype Vertex MultiModal embedding handler

  #35677 opened Jul 24, 2025

• Bump github.com/aws/smithy-go from 1.22.4 to 1.22.5 in /sdks

  #35682 opened Jul 25, 2025

• Bump cloud.google.com/go/storage from 1.55.0 to 1.56.0 in /sdks

  #35683 opened Jul 25, 2025

• Upgrade Errorprone to 2.31.0

  #35684 opened Jul 25, 2025

• feat(WIP): Qdrant Search Handler

  #35686 opened Jul 25, 2025

• JUnit5 support

  #35688 opened Jul 25, 2025

• [yaml] - add nullable field test for readFromBigQuery and update create core logic

  #35692 opened Jul 25, 2025

• Cloudsql MySQL embeddings colab.

  #35695 opened Jul 25, 2025

• Add BigTableRead connector and new feature implemented

  #35696 opened Jul 25, 2025

• workflows: run ML tests requiring docker-in-docker environment on `ubuntu-latest`

  #35698 opened Jul 25, 2025

• Update beam_PostCommit_Python_Portable_Flink.yml

  #35699 opened Jul 25, 2025

• Revert "Refactor: separate SplittableTruncateSizedRestrictions"

  #35700 opened Jul 25, 2025

• Add Terraform configuration and IAM management scripts for GCP project

  #35701 opened Jul 25, 2025

• Cachiman README.md

  #35704 opened Jul 27, 2025

• Update Python Dependencies

  #35705 opened Jul 27, 2025

• sdks/python: sink data with Milvus Search I/O connector

  #35708 opened Jul 27, 2025

25 Issues closed by 11 people

• The PostCommit Java ValidatesRunner Flink Java8 job is flaky

  #32949 closed Jul 27, 2025

• [Bug]: Missing Type Implementations in parseDefaultExpression for ClickHouseIO

  #33692 closed Jul 27, 2025

• [Bug]: SparkRunner reads input data twice when using FileIO

  #33771 closed Jul 27, 2025

• [Bug]: Avro GenericRecord to Row conversion does not support some logical types and any custom conversions

  #34009 closed Jul 25, 2025

• [Bug]: BigqueryIO - can't insert data to bigquery table which's field name start with number

  #35625 closed Jul 25, 2025

• [Task]: Support non-ascii BigQuery field name for BigQueryIO STORAGE_WRITE_API

  #33991 closed Jul 24, 2025

• The PostCommit Python Arm job is flaky

  #30760 closed Jul 24, 2025

• Performance Regression or Improvement: sideinpts_python_batch_1gb_1kb_10workers_1000window_99key_percent_dict:runtime

  #35623 closed Jul 23, 2025

• Performance Regression or Improvement: test_cloudml_benchmark_criteo_10GB-runtime_sec:runtime_sec

  #35648 closed Jul 23, 2025

• [Feature Request]: Managed IO connector for Apache Iceberg to offer dynamically handling schemas

  #33724 closed Jul 23, 2025

• [Task]: Investigate Failures in Go Flink Load Tests with original load

  #33753 closed Jul 23, 2025

• [Bug]: WriteToBigQuery doesn't do WRITE_TRUNCATE properly with identical table names but in different datasets

  #34247 closed Jul 23, 2025

• [Bug]: JDBC javasdk_date:v1 decode error

  #33442 closed Jul 23, 2025

• Side inputs not working in CombineGlobally

  #19851 closed Jul 23, 2025

• The PostCommit Python Xlang IO Dataflow job is flaky

  #33253 closed Jul 22, 2025

• The IcebergIO Integration Tests job is flaky

  #31931 closed Jul 22, 2025

• [Feature Request]: {BigQuery IO Iceberg} - Allow users to run streaming reads

  #33725 closed Jul 22, 2025

• [Feature Request]: Managed IO connector for Apache Iceberg to allow users to define file size for writes

  #33727 closed Jul 22, 2025

• [Bug]: SpannerIO ExecuteStreamingRead timeout not being set

  #33738 closed Jul 22, 2025

• [Bug]: BigQuery failures logged as warning even when the rows won't be retried

  #35356 closed Jul 22, 2025

• [Bug]: BigTable client "One or more TimeSeries could not be written" Java 2.65.0

  #35326 closed Jul 22, 2025

• The Python ValidatesContainer Dataflow ARM job is flaky

  #33065 closed Jul 21, 2025

• The Publish Beam SDK Snapshots job is flaky

  #32161 closed Jul 21, 2025

• [Feature Request]: Update Python JDBC write transforms to support configuring the batch size

  #32891 closed Jul 21, 2025

• [Bug]: BigQuery streaming throwing OutOfRangeException

  #30177 closed Jul 20, 2025

11 Issues opened by 8 people

• [Task]: Stablize and improve Prism runner

  #35707 opened Jul 27, 2025

• [Task]: Add Anomaly Detection PTransform to Beam

  #35706 opened Jul 27, 2025

• [Failing Test]: :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixPROCESS falis

  #35702 opened Jul 26, 2025

• [Bug]: YAML MLTransformer is not registering SentenceTransformerEmbeddings

  #35693 opened Jul 25, 2025

• [Failing Test]: beam_PreCommit_Python_Dataframes (Run Python_Dataframes PreCommit 3.12)

  #35691 opened Jul 25, 2025

• Log Spam - Auto Commit has been disabled

  #35685 opened Jul 25, 2025

• [Bug]: Yaml writeToBigQuery - java.lang.IllegalArgumentException: Unexpected type_info: TYPEINFO_NOT_SET

  #35668 opened Jul 23, 2025

• [Bug]: YAML Flatten incorrectly drops fields when input PCollections' schema are different

  #35666 opened Jul 23, 2025

• [Task]: Document WaitOn / WaitOn transforms in Beam Transform catalog

  #35646 opened Jul 21, 2025

• [Task]: Polish Beam SQL user experience

  #35637 opened Jul 21, 2025

• > # :wave: Welcome to GitHub Learning Lab's "Introduction to GitHub"

  #35636 opened Jul 21, 2025

50 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• WIP Add support to AbstractWindmillStream to transition between physical streams within the same logical stream

  #35523 commented on Jul 23, 2025 • 3 new comments

• Add dataflow option

  #35120 commented on Jul 23, 2025 • 3 new comments

• [1/2] sdks/python: enrich data with CloudSQL [PostgreSQL, MySQL, SQLServer]

  #34398 commented on Jul 22, 2025 • 2 new comments

• GCP Access Control

  #35107 commented on Jul 25, 2025 • 2 new comments

• [Java] Add parsedData to Hl7v2Message and Update HL7v2IO Docs

  #34213 commented on Jul 25, 2025 • 0 new comments

• [Java] Add Gauge Metric Extraction to DataflowMetrics

  #34307 commented on Jul 27, 2025 • 0 new comments

• add graceful restart mechanism for GetWorkStream to prevent DEADLINE_…

  #34367 commented on Jul 27, 2025 • 0 new comments

• Concat protos in BQStorageWriteAPI - solve edge cases during mering of nested repeated fields

  #34436 commented on Jul 24, 2025 • 0 new comments

• use WindmillChannelFactory to control what types of channels to generate

  #34653 commented on Jul 23, 2025 • 0 new comments

• Fail Fast if Resources Do Not Exist in Kafka Cluster.

  #34658 commented on Jul 22, 2025 • 0 new comments

• Introduce OutputBuilder in Java SDK

  #34902 commented on Jul 21, 2025 • 0 new comments

• [Gradle] Clean up DataflowRunner's Java distroless container image build tasks

  #34999 commented on Jul 21, 2025 • 0 new comments

• Update Beam Protobuf Schema (Java)

  #35150 commented on Jul 25, 2025 • 0 new comments

• Iobase streaming support in FileBasedSink, exposed in TextIO

  #35253 commented on Jul 25, 2025 • 0 new comments

• Adding GCP Spanner Change Stream support for Python

  #35453 commented on Jul 25, 2025 • 0 new comments

• fixed the large row errors fof BigQuery IO storage write

  #35478 commented on Jul 23, 2025 • 0 new comments

• Secret management service

  #35524 commented on Jul 25, 2025 • 0 new comments

• [GrowableOffsetRangeTracker] Use UnsignedLong instead of BigDecimal to calculate progress

  #35561 commented on Jul 25, 2025 • 0 new comments

• adds pre-commit hook to standardize whitespaces, adds EditorConfig to set the indents

  #35564 commented on Jul 25, 2025 • 0 new comments

• [YAML] A Streaming Inference Pipeline - Taxi Fare Estimation

  #35568 commented on Jul 25, 2025 • 0 new comments

• try synchronized when calling APPEND_CLIENTS

  #35576 commented on Jul 24, 2025 • 0 new comments

• [Test only] Vendor Calcite 1.40

  #35588 commented on Jul 23, 2025 • 0 new comments

• Added Filter and Projection Pushdown support to ParquetIO and Beam SQL's ParquetTable

  #35589 commented on Jul 25, 2025 • 0 new comments

• Reenable prism as default

  #35621 commented on Jul 25, 2025 • 0 new comments

• [Java Harness] Improve caching performance and measure overhead for cache internals.

  #35632 commented on Jul 23, 2025 • 0 new comments

• [GSOC 25] Enhanced Interactive Pipeline Development Environment for JupyterLab

  #35128 commented on Jul 21, 2025 • 0 new comments

• [Bug]: BQ TableRow does not accept column named 'f'

  #33531 commented on Jul 21, 2025 • 0 new comments

• [Bug]: Got error 'ValueError: Schema with id <some uuid> has encoding_positions_set=True, but not all fields have encoding_position set' for no apparent reason

  #35318 commented on Jul 22, 2025 • 0 new comments

• [Task]: Remove powermock dependency

  #34056 commented on Jul 23, 2025 • 0 new comments

• The PostRelease Nightly Snapshot job is flaky

  #30505 commented on Jul 23, 2025 • 0 new comments

• [Bug]: gprcio limitation to < 1.66 in Python is problematic

  #34081 commented on Jul 23, 2025 • 0 new comments

• [Bug]: Incorrect $partition Metadata in Trino for Iceberg Tables Written via IcebergIO.writeRows with Timestamp Partitioning

  #35417 commented on Jul 24, 2025 • 0 new comments

• The LoadTests Java PubsubIO job is flaky

  #35194 commented on Jul 24, 2025 • 0 new comments

• [Task]: Add Prism to runner capability matrix

  #34660 commented on Jul 25, 2025 • 0 new comments

• [Task]: Support more Beam portable schema types as Python types

  #25946 commented on Jul 25, 2025 • 0 new comments

• [Task]: Manage Infra privileges via Infra-as-code

  #33756 commented on Jul 25, 2025 • 0 new comments

• [Bug]: BigqueryIO is very slow if using storage api and dynamic destination to write data to over thousand different tables with high data skew

  #32508 commented on Jul 25, 2025 • 0 new comments

• [Feature Request]: Support Java 25

  #35627 commented on Jul 25, 2025 • 0 new comments

• [Task]: Migrate Python SDK away from the google-apitools library

  #35611 commented on Jul 25, 2025 • 0 new comments

• [Bug]: package google-apitools missing when using PDM package manager

  #35593 commented on Jul 25, 2025 • 0 new comments

• [Feature Request]: Resilient Fallback log and Auto-Creation of Default Storage Bucket.

  #35584 commented on Jul 25, 2025 • 0 new comments

• [Bug]: DebeziumIO and RequestResponseIO in different package than all other IOs

  #35557 commented on Jul 25, 2025 • 0 new comments

• JUnit5 support

  #18733 commented on Jul 25, 2025 • 0 new comments

• [Bug]: Re-enable GoUsingJava xlang suites

  #32492 commented on Jul 26, 2025 • 0 new comments

• [GSoC 2025] Beam ML Integration Tracking: Vector DB, Feature Store, and Embedding Generator

  #35046 commented on Jul 27, 2025 • 0 new comments

• init dummy file

  #33767 commented on Jul 27, 2025 • 0 new comments

• Enable timeout setting for Python TestPipeline (#29646)

  #33866 commented on Jul 24, 2025 • 0 new comments

• [BEAM-6394] Add support to write protobuf data using ProtoParquetReader

  #34063 commented on Jul 25, 2025 • 0 new comments

• Fix Docker build error by adding fallback for python3.12-distutils

  #34144 commented on Jul 25, 2025 • 0 new comments

• Fix ProtoCoder NoSuchMethodException

  #34194 commented on Jul 27, 2025 • 0 new comments