rspec-queue does not fail if there are NameErrors in specs
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- ruby
- Domain
- cli, testing-qa
Research direction
Start with ruby/lib/rspec/queue.rb around lines 387-401 and reproduce the NameError case using rspec-queue, comparing its exit status with bare rspec. Trace how NameError and syntax errors affect the reported result and determine whether the completed behavior should return a non-zero status without breaking retries; done means regression coverage and correct failure status.
Written by the indexing model from the issue text.
Description
I've just encountered the following behaviour with the rspec runner of ci-queue:
- Modify a spec to reference an uninitialized constant so as to trigger a
NameError, e.g. by referring to a non-existent class:
describe MyNonExistentClass do
...
end
- Run rspec-queue
- rspec-queue exits successfully.
I guessed wrongly that rspec-queue would fail with a non-zero exit code, but it exited with 0.
Bare rspec exits with 1 in the above situation.
I'm wondering if this is intended behaviour - perhaps its needed for retries to work?
The error message output is:
NameError: uninitialized constant MyNonExistentClass
Debugging shows that line 400 is where the 0 exit code is returned, because the NameError causes @world.non_example_failure to be true. Syntax errors appear to have the same result:
- Dominant language
- Ruby
- Stars
- 240
- Forks
- 36
- Avg merge
- 28m
- Merged PRs (30d)
- 3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Shopify/ci-queue
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
-
Difficulty 4/5 3-5 days Newbie friendliness 50/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
All issues in Shopify/ci-queue
Similar issues
-
SyncEm always forwards a dummy block, so wrapped methods lose their no-block/Enumerator behavior Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug P2
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
dependabot
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
githubnext/gh-aw-cao#13349 ·
-
user-reported
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Kong/developer.konghq.com#7316 ·