Pin versions/ranges for spark-connect-client python dependencies using a constraints file
Maintainers usually reply within 1 day
@Maleware is already working on this.
Since Oct 12, 2026.
Assessment
This issue has not been assessed yet.
Description
The current Dockerfile breaks as not all packages are pinned.
spark-connect-client/Dockerfile:44-48 (docker-images origin/main) pins only the four top-level packages:
RUN pip install --no-cache-dir --user \
"pyspark[connect]==${PRODUCT_VERSION}" \
"jupyterlab==4.4.1" \
"scikit-learn==1.3.1" \
"matplotlib==3.10.1"
A combination of the following breaks the build:
- scikit-learn 1.3.1 declares numpy <2.0,>=1.17.3 (METADATA inside the image), so numpy ends up on e.g. 1.26.4.
- pyarrow 26.0.0 declares no dependencies at all on PyPI (
requires_dist: nullfrom https://pypi.org/pypi/pyarrow/26.0.0/json), yet it refuses NumPy 1.x at import time (seehttps://github.com/apache/arrow/pull/50769) - pip can't see that conflict
Pinning pyarrow alone would fix that, but the next undeclared runtime constraint could break a nightly build the same way. The real problem is that nothing besides those four packages is pinned.
Suggestion
Use the existing pattern in docker-images by adding a constraints directory to spark-connect-client
- Dominant language
- Rust
- Stars
- 74
- Forks
- 6
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/spark-k8s-operator
-
type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
stackabletech/spark-k8s-operator#775 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
stackabletech/spark-k8s-operator#806 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
stackabletech/spark-k8s-operator#805 ·
Maintainers usually reply within 1 day
-
Application driver Pod is not terminated when Spark is OOMKilledMay be free again @razvan claimed this 43 days ago, and no pull request is open. Opentype/bug
stackabletech/spark-k8s-operator#780 · 1 assignee ·
Maintainers usually reply within 1 day
-
Allow defining a time to live for finished SparkApplication driver podsPossibly taken @sweb claimed this 18 days ago. Opencustomer-request
Difficulty 4/5 3-5 days Newbie friendliness 55/100
stackabletech/spark-k8s-operator#774 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
All issues in stackabletech/spark-k8s-operator
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
storytold/photocraft#14822 ·
Maintainers usually reply within 1 day
-
bug css
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
dginev/latexml-oxide#853 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 2 days
-
[Bug] export_build_environment admits its output directory only after the export ran in the build computerPossibly taken @viniciusdsandrade claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day