run upstream tests
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- devops, testing-qa
Research direction
Start with Debian's debian/tests/run-all and Ruby's test/runner.rb, then compare that post-build approach with the existing Docker image test flow and the reported ruby:slim failures. Done means upstream tests run after the image is built, with failures resolved or excluded and each exclusion documented with a justification.
Written by the indexing model from the issue text.
Description
Similar to https://github.com/docker-library/golang/issues/552, there are a lot of upstream tests we could run and should run somewhere, but during build doesn't necessarily make sense.
In https://salsa.debian.org/ruby-team/ruby/-/blob/5b91c2c6b64195da7110d341daba904b9d12c9d5/debian/tests/run-all, we can see that Debian does some clever things to run a lot of the upstream tests post-build via "autopkgtests" (which is very akin to how we run tests on our images after they're built -- they typically run against the built + installed packages).
The basic gist is ruby test/runner.rb, and Debian's got a pretty hefty collection of tests they exclude which will probably inspire tests we need to exclude too (especially # FIXME for now, we are excluding the tests for C extensions; couldn't figure out how to properly build them without building everything else, since we'll have the same issue with doing this post-build).
So we'll need to recreate our "download the source code" logic to get the test files back, but that's the easy part. My own very basic testing of this idea on ruby:slim comes up with "7 failures, 6 errors, 176 skips", so there's clearly some work to do here (for each of those, fixing the image, accommodating the test somehow with runtime flags/packages/etc, or building up a list of tests to skip and writing down justifications for why we think they're valid to skip/ignore).
- Dominant language
- Shell
- Stars
- 624
- Forks
- 322
- Avg merge
- 22d 3h
- Merged PRs (30d)
- 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from docker-library/ruby
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
docker-library/ruby#507 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
docker-library/ruby#502 · 3 comments ·
-
Request
Difficulty 5/5 Over a week Newbie friendliness 30/100
docker-library/ruby#130 · 7 comments · 1 reaction ·
All issues in docker-library/ruby
Similar issues
-
Issue-Enhancement Needs-Triage
Difficulty 1/5 Under an hour Newbie friendliness 86/100
PowerShell/PowerShell#28061 · 2 reactions ·
-
Feature Request: Add ability to load custom environment variables in linux-exec-server-installer.sh Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
microsoft/vscode-remote-release#11867 ·
-
AuTest Bug Tests
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/trafficserver#13714 ·
-
Update to NCCL 2.32 Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
conda-forge/nccl-feedstock#166 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
vllm-project/agentic-api#358 · 1 comment ·