Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

wordpress.phpunit: database-type=mdi-native excludes 100% of candidates (excluded=1335, routed=0) on multisite + managed bootstrap

Closed
  • Is something wrong?
#2,500 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
mariadb, mysql, php, typescript
Domain
backend, databases, testing-qa

Research direction

Start by reproducing the wordpress.phpunit component with multisite and managed bootstrap at wp-codebox commit d4e0f713, comparing database-type=mysql with mdi-native. Trace the candidate selection and routing path after discovery, then inspect the exclusion reporting and selected=0 handling. Done means candidates route under mdi-native, exclusions expose a reason, and a zero-selection run fails fast.

Written by the indexing model from the issue text.

Description

Summary

With database-type=mdi-native, wordpress.phpunit discovers the full candidate set and then excludes 100% of it, routing zero tests. The same component on database-type=mysql runs 1,177 tests normally.

FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0

Discovery clearly works — it finds 1,335 candidates. Routing/selection is where everything is dropped.

Reproduction

  • wp-codebox: d4e0f713 (current main), so this includes #2490 (mdi-native backend), #2494 (native multisite PHPUnit state), #2497 (explicit native MDI sources), and #2498 (retain PHPUnit failure artifacts).
  • Consumer: Homeboy wordpress extension, managed install, provenance-valid (homeboy extension setup wordpress run clean; version reported 0.26.8 at the managed source path).
  • Component under test: a WordPress plugin with
    • wp_codebox_multisite: true
    • wp_codebox_phpunit_bootstrap_mode: "managed"
    • 101 test files / 1,335 PHPUnit candidates

Control vs variable, same component, same commit, only database_type changed:

database_type Result
mysql (external service) 1,177 passed, 7 skipped, 1,177 total, 577s
mdi-native 0 passed, 0 failed, 0 skipped, 0 total — excluded=1335

Behaviour detail

  • The run does not fail fast at the point of exclusion. After reporting excluded=1335 it continues until the harness budget is consumed. In one run it reported test suite consumed 1336s of its 1500s budget (89%) having executed nothing.
  • With #2498 present the run surfaces failed test command did not expose an observed failed test ID rather than silently producing nothing, which is an improvement, but there is still no diagnostic explaining why 1,335 candidates were excluded.

Why this matters

mdi-native is the difference between "tests need an external MySQL/MariaDB with provisioned credentials" and "tests need nothing." For consumers, that is the difference between a suite any contributor or agent can run locally and one gated behind secrets. Right now the backend validates, plans, and reports success at the config layer while running zero tests — so it cannot be trusted as a MySQL-compatible harness.

Credit where due: the consumer side handles this safely. Homeboy classifies zero executed tests as invalid_evidence → passed: false, so a release preflight on this backend is correctly blocked rather than shipping untested code. The danger is not a silent false pass in that toolchain, but the failure is silent enough that another consumer could plausibly treat "exit 0, no failures" as success.

Asks

  1. Emit a reason for exclusion. excluded=1335 with no per-candidate or aggregate cause is not diagnosable. Even a single aggregate reason code would have made this self-service.
  2. Fix the routing/selection path for mdi-native so candidates route the way they do under mysql. The identical candidate count (1,335) under both backends suggests discovery is shared and only selection/routing diverges.
  3. Fail fast on selected=0 when candidates>0. Consuming the full time budget after excluding everything wastes ~22 minutes per run and obscures the real error.

Possibly relevant

The combination here is multisite + managed bootstrap. #2494 specifically addressed native multisite PHPUnit state, so this may be an unhandled remainder of that work rather than a general mdi-native failure — I have not tested single-site to isolate that, and would be happy to if it helps.

Dominant language
TypeScript
Stars
17
Forks
4
Avg merge
47m
Merged PRs (30d)
67

Getting set up

We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Automattic/wp-codebox

All issues in Automattic/wp-codebox

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.