apache/beam

DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc formats.

Open

#18199 opened on Jun 3, 2022

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (7,313 stars) (4,097 forks)batch import
P3dataflowgood first issuerunnerstests

Description

Cloud Dataflow has a minor history of small bugs related to various code paths expecting there to be or not be a trailing forward-slash in these location fields. The way that Beam's integration tests are set up, we are likely to only have one of these two cases tested (there is a single set of integration test pipeline options).

We should add a dedicated DataflowRunner integration test to handle this case.

Actually, we should probably canonicalize the URLs so that we only ever produce one version.

Imported from Jira BEAM-1194. Original Jira may contain additional context. Reported by: dhalperi.

Contributor guide

Tech stack
java
Domain
backendcloud
Issue type
refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
JavaApache Beam basicsDataflowRunner knowledge
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Review the DataflowRunner source code for usage of gcpTempLocation and stagingLocation. Examine existing integration tests for DataflowRunner to understand test patterns. Consider implementing a canonicalization method to enforce consistent trailing slash handling. Consult the linked Jira issue (BEAM 1194) for additional context and potential edge cases.