Ruby: a Rakefile's `warning = true` is never read, Cucumber features are not tests, project runner scripts (test/runner, brew tests/style/typecheck, ruby test/*.rb) are not runners

Start with the Ruby checks in ecosystems/ruby/practices.py, where RUBY_WARNING_SETTINGS reads only .rspec and the TEST_HELPERS tuple defined in ecosystems/ruby/init.py, and with docs/ecosystems/ruby.md, which says a test task's warning = true counts. Reproduce the fluentd case with a Rakefile setting t.warning = true and check the warning requirement fails. Done means the Rakefile setting is detected, and runners such as test/runner, ruby test/*.rb and .feature files are classified correctly, which needs maintainer agreement on scope first.

fluentd sets warning = true on the Rake::TestTask that rake test runs (docs/ecosystems/ruby.md says it counts) but RUBY_WARNING_SETTINGS never reads Rakefiles → FAIL "missing Ruby warnings". rspec's 140 .feature files (run by script/run_cucumber) are unrecognised. puma runs its Minitest suite through test/runner; fzf runs minitest with ruby test/…rb directly; Homebrew runs RuboCop/shellcheck/shfmt (brew style), Sorbet (brew typecheck) and RSpec (brew tests) through its own commands.

Evidence from the corpus audit

junegunn/fzf / tests-run (Ruby)

• kind: FALSE_NEGATIVE
• actual: WARN (Ruby place): tests found in 9 file(s), but no CI configuration runs rspec, rake test or rails test
• expected: PASS: both CI jobs run the minitest suite directly with ruby.
• evidence: .github/workflows/linux.yml:62: run: make install && ./install --all && tmux new-session -d && ruby test/runner.rb --verbose; .github/workflows/macos.yml:45: run: make install && ./install --all && LC_ALL=C tmux new-session -d && ruby test/test_go.rb --verbose; Gemfile:5: gem 'minitest', '5.25.4'; Makefile:98,101: itest: ... 'tmux -f /dev/null new-session -d && ruby test/runner.rb; s=$$?; tmux kill-server; exit $$s'; \
• evidence (validator): docs/ecosystems/ruby.md:7-8: tests are run "by rspec, rake test / spec or rails test, or by the Rakefile's test tasks"; ruby test/<file>.rb (minitest's native way) is not a runner.
• repro: Gemfile with minitest, test/test_x.rb, workflow run: ruby test/test_x.rb.
• confidence: medium

fluent/fluentd / compiler-warnings (Rakefile warning = true never read)

• kind: FALSE_NEGATIVE
• actual: FAIL: "Ruby: missing Ruby warning settings: Ruby warnings; found RuboCop or Standard"
• expected: PASS: docs/ecosystems/ruby.md says a test task's warning = true counts, and fluentd's Rakefile sets it on the Rake::TestTask that rake test (what CI runs) depends on. The setting is in fact never read from a Rakefile: RUBY_WARNING_SETTINGS reads only .rspec and the test helpers.
• evidence: Rakefile:11: task test: [:base_test]; Rakefile:29: Rake::TestTask.new(:base_test) do |t|; Rakefile:48: t.warning = true; .github/workflows/test.yml:56: run: bundle exec rake test TESTOPTS="-v --report-slow-tests --no-show-detail-immediately". --answers (compiler-warnings, Ruby): "found": {"RuboCop or Standard": ["CI (.github/workflows/rubocop.yml)"]}, "missing": ["Ruby warnings"]. Validator ecosystems/ruby/practices.py:33-35: RUBY_WARNING_SETTINGS = requirement_reader("Ruby warning settings", (".rspec", *TEST_HELPERS), ...) with ecosystems/ruby/__init__.py:32: TEST_HELPERS = (".simplecov", "spec/spec_helper.rb", "spec/rails_helper.rb", "test/test_helper.rb") - no Rakefile. (sinatra's "found Ruby warnings" comes from rack-protection/.rspec --warnings, not from its Rakefile's t.warning = true.)
• repro: Rakefile with Rake::TestTask.new(:test) do |t| t.warning = true end, a workflow running bundle exec rake test, Gemfile, test/test_foo.rb (scratch repros a, a2 (Minitest::TestTask), a3 (default task), a4, a5: all FAIL "no Ruby warning settings found").
• confidence: high

puma/puma / tests-run

• kind: UNSUPPORTED
• actual: FAIL: "Ruby: tests found in 62 file(s), but no CI configuration runs rspec, rake test or rails test"
• expected: PASS: every test job compiles the extension with bundle exec rake compile and then runs the suite through the repository's own Minitest runner script test/runner (a Ruby script that loads minitest); a project-specific runner, so a gap rather than a bug, but the result reads as "puma does not run its tests in CI", which is wrong.
• evidence: .github/workflows/tests.yml:132-134: - name: test ... run: test/runner --verbose (and :194-196); test/runner:1-5: #!/usr/bin/env ruby ... A simplified test runner; it runs all tests by default. It assumes that "bundle exec rake compile" has been run.; Rakefile:127: Rake::TestTask.new(:test) do |t|, :135: task :default => [:rubocop, :test].
• repro: workflow step run: test/runner, where test/runner is a Ruby script with require "minitest/autorun"; expect the runner recognised when the script's head shows minitest (or at least an UNKNOWN rather than "no CI configuration runs").
• confidence: low

Homebrew/brew / code-formatting, static-analysis, type-checking, tests-run (brew's own dev commands)

• kind: UNSUPPORTED
• actual: code-formatting FAIL "Ruby: Library/Homebrew: ... is defined but CI does not run RuboCop, Standard or syntax_tree"; static-analysis FAIL "Ruby: RuboCop is configured ... but does not run; Shell: shellcheck is configured (.shellcheckrc) but does not run"; type-checking FAIL "Ruby: Library/Homebrew: Sorbet is configured (Library/Homebrew/sorbet/config) but does not run"; tests-run FAIL "Ruby: Library/Homebrew: tests found in 11 file(s), but no CI configuration runs rspec, rake test or rails test"
• expected: Homebrew runs all of these through its own dev commands: brew style (RuboCop + shellcheck + shfmt), brew typecheck (Sorbet srb tc), brew tests (RSpec); the four FAILs are all wrong for the real project. A project-specific wrapper, so a gap, but the biggest Ruby project in the corpus ends up with its whole CI invisible.
• evidence: .github/workflows/tests.yml:62: - run: brew style; :64: - run: brew typecheck; :97: run: brew style homebrew/core homebrew/cask; the tests (...) matrix jobs (.allow.txt platform-matrix lines tests (Linux, 1/2), tests (macOS, 1/2), ...) run brew tests; Library/Homebrew/dev-cmd/typecheck.rb (named in sorbet.yml:6).
• repro: unclear (would need per-project command aliases, e.g. a --run-as "brew style=rubocop" mapping or reading Library/Homebrew/dev-cmd/*.rb)
• confidence: medium

rspec/rspec / tests-run (Cucumber)

• kind: UNSUPPORTED
• actual: unrecognised_suffixes: .feature (140 files); tests-run PASS for Ruby credits only the RSpec specs.
• expected: Cucumber features are a standard Ruby acceptance-test format and CI runs them (script/run_cucumber); they should count as tests and .feature should not be "unrecognised".
• evidence: .github/workflows/rspec.yml:53: - run: ../script/run_cucumber; rspec-core/features/*.feature etc. (140 files).
• repro: features/x.feature plus a workflow running bundle exec cucumber; expect tests found and run.
• confidence: low

Found by the regression-corpus audit (235 real repositories, one or more per ecosystem profile; corpus and expected results in corpus/). Evidence is quoted from the audited checkouts at the commits pinned in corpus/projects.toml; <checkouts>/<owner>__<repo> is such a checkout.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt