What happened?

We've discovered a data corruption problem impacting Python SDK versions 2.53.0 through 2.58.0.

Symptoms:
Beam pipelines utilizing these Python SDKs and reading data from Google Cloud Storage (GCS) via GcsIO, either directly or indirectly (e.g., through BigQueryIO), might experience data corruption. This issue can occur randomly, depending on network and GCS conditions.

Root cause:
In version 2.53.0, we switched the GcsIO implementation from apitools to the GCS Python client library google-cloud-storage (#29360). This library internally uses google-resumable-media for data downloads and uploads. The minimum google-cloud-storage version is set to 2.10.0, which relies on google-resumable-media version 2.3.2 or newer.

Unfortunately, a bug in google-resumable-media 2.3.0 is found to impact resumable downloads. When such a download starts from the middle of a GCS blob (non-zero start position), an incorrect start position is used in the HTTP call to GCS, leading to the wrong data chunk being returned.

The bug has been fixed since 2.7.2.

Mitigation:

• Upgrade google-cloud-storage to version 2.18.2 or newer in the runtime environment. This will automatically include a newer google-resumable-media version with the fix. For information on specifying pipeline runtime dependencies, see: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner