apache/beam

Python SDK should stage only up-to-date versions of pipeline dependencies defined by requirements file.

Open

#19,968 opened on Jun 4, 2022

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (4,097 forks)batch import
P3bugcoregood first issuepython

Repository metrics

Stars
 (7,313 stars)
PR merge metrics
 (Avg merge 6d) (243 merged PRs in 30d)

Description

When staging pipeline dependencies specified in requirements.txt file and transitive deps, we stage entire content of requirements-cache directory on user's machine, which may include dependencies that were cached a while ago and are no longer needed. See [1] for discussion and some ideas how we can avoid staging packages we don't need.

[1] https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E

Imported from Jira BEAM-8900. Original Jira may contain additional context. Reported by: tvalentyn.

Contributor guide

Research direction
Inspect the requirements cache directory and implement a mechanism to stage only up to date dependencies by comparing package timestamps or using pip's dependency resolution to determine which packages are needed.
Tech stack
python
Domain
backendbuild system
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Pythonpip
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75