apache/beam

DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc formats.

Open

#18,199 opened on Jun 3, 2022

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (4,097 forks)batch import
P3dataflowgood first issuerunnerstests

Repository metrics

Stars
 (7,313 stars)
PR merge metrics
 (Avg merge 6d) (243 merged PRs in 30d)

Description

Cloud Dataflow has a minor history of small bugs related to various code paths expecting there to be or not be a trailing forward-slash in these location fields. The way that Beam's integration tests are set up, we are likely to only have one of these two cases tested (there is a single set of integration test pipeline options).

We should add a dedicated DataflowRunner integration test to handle this case.

Actually, we should probably canonicalize the URLs so that we only ever produce one version.

Imported from Jira BEAM-1194. Original Jira may contain additional context. Reported by: dhalperi.

Contributor guide

Research direction
Inspect the DataflowRunner code to see where gcpTempLocation and stagingLocation are used and ensure trailing slashes are handled consistently. Consider adding integration tests.
Tech stack
java
Domain
backendclouddata
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
JavaGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
55