[Tests] Tracker: improve unit test quality (Farley score 7.2 → ~8.1)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 15/100
Research direction
This is a parent tracker for sub-issues #809 to #819, not a single change. Start from the audit table and pick one small sub-issue, such as #809 (SESSION_API_KEY isolation in the quota route tests) or #811 (RESEND_* env setup in test_resend_keycloak.py), then read the test files it names under tests/. Done means that sub-issue's tests pass in isolation and in the full suite.
Written by the indexing model from the issue text.
Description
Tracker for improving the unit test suite. It comes from an audit of tests/ (384 files, 6,875 test functions) against Dave Farley's 8 properties of good tests, at 79124d5. Each sub-issue below is linked as a GitHub sub-issue and grouped by the Farley property it improves.
Current Farley Score: 7.2 / 10 (Good) → about 8.1 if all sub-issues are done
| Property | Weight | Score now | Sub-issues |
|---|---|---|---|
| Repeatable | 1.25 | 7.0 | #809, #810 |
| Atomic | 1.0 | 7.5 | #811, #812 |
| Maintainable | 1.5 | 6.5 | #813, #814 |
| Understandable | 1.5 | 7.5 | #815, #816 |
| Granular | 1.0 | 7.5 | #817 |
| Fast | 0.75 | 6.5 | #818, #819 |
| Necessary | 1.0 | 8.0 | none (see below) |
| First (TDD) | 1.0 | 7.0 | none (see below) |
Work order is reliability first: 🔴 Critical (Repeatable/Atomic), then 🟠 Medium, then 🟡 Low. See the priority tables below.
Priority tables
Critical items always come first, whatever their efficiency. Within Medium and Low, rows are sorted by efficiency (weight × Δ score ÷ effort, where Low=1, Medium=2, High=3).
🔴 Critical: tests may pass or fail for the wrong reason
| Sub-issue | Improvement | Farley property | Effort | Status |
|---|---|---|---|---|
| #809 | Isolate tests from an ambient SESSION_API_KEY (17 quota route tests fail on main when it is set) |
Repeatable | Low | Confirmed failing |
| #811 | test_resend_keycloak.py sets RESEND_* env vars at import time, which leak into the whole session |
Atomic | Low | Confirmed leak, no failures yet |
| #810 | Pin the clock in the budget maintenance-processor test and widen the calendar guard to all test files | Repeatable | Low | Confirmed: breaks the guard's own rule |
| #812 | Restore module and env state after importlib.reload, sys.modules and os.environ edits |
Atomic | Medium | Confirmed leaks, no failures yet |
| #760 | Migration 186 tests fail in the full suite | Repeatable | — | ✅ Fixed by #805 |
🟠 Medium: visible quality problems
| Sub-issue | Improvement | Farley property | Δ score | Effort | Efficiency |
|---|---|---|---|---|---|
| #813 | Replace heavy patching of private names with shared fakes (677 patch targets; 103 tests with 8 or more patches) | Maintainable | +1.0 | High | 0.50 |
| #814 | Move 782 function-level imports to the top of the module | Maintainable | +0.3 | Low | 0.45 |
| #815 | Split very large test modules (up to 6.6k lines) and tests over 120 lines | Understandable | +0.5 | High | 0.25 |
| #816 | Fix the routes conftest.py docstring that still describes SQLite |
Understandable | +0.1 | Low | 0.15 |
🟡 Low: developer experience
| Sub-issue | Improvement | Farley property | Δ score | Effort | Efficiency |
|---|---|---|---|---|---|
| #818 | Remove real sleeps (one test waits 12 s; another uses a 1 s "is it ready" wait) | Fast | +0.5 | Low | 0.38 |
| #817 | Parametrize the 22 tests with 15 or more asserts (test_has_exact_mention has 40) |
Granular | +0.3 | Low | 0.30 |
| #819 | Speed up the alembic migration tests (18–30 s each; the 19 slowest tests in the suite) | Fast | +1.0 | High | 0.25 |
Quick wins (low effort): #809, #810, #811, #814, #816, #817, #818
By Farley property
🔁 Repeatable: same result for any machine, environment or date
- #809 🔴 Isolate tests from an ambient
SESSION_API_KEY. 17 quota route tests fail onmainwhen it is set. Effort: Low. - #810 🔴 Pin the clock in the budget maintenance-processor test and widen the calendar guard to all test files. Effort: Low.
- #760 🔴 Migration 186 tests fail in the full suite. Already fixed by #805 (each xdist worker had a different encryption key).
⚛️ Atomic: no state leaks between tests
- #811 🔴
test_resend_keycloak.pysetsRESEND_*env vars for the whole session at import time. Effort: Low. - #812 🔴 (latent) Restore module and env state after
importlib.reload,sys.modulesandos.environedits. Effort: Medium.
🔧 Maintainable: tests survive refactors
- #813 🟠 Replace heavy patching of private names (677 targets; 103 tests with 8 or more patches) with shared fakes. Δ +1.0, effort High, efficiency 0.50.
- #814 🟠 Move 782 function-level imports to module level. Δ +0.3, effort Low, efficiency 0.45.
📖 Understandable: easy to read and navigate
- #815 🟠 Split very large modules (up to 6.6k lines) and tests over 120 lines. Δ +0.5, effort High, efficiency 0.25.
- #816 🟠 Fix the stale SQLite reference in the routes
conftest.pydocstring. Δ +0.1, effort Low, efficiency 0.15.
🎯 Granular: one failure points to one cause
- #817 🟡 Parametrize the 22 tests with 15 or more assertions (
test_has_exact_mentionhas 40). Δ +0.3, effort Low, efficiency 0.30.
⚡ Fast: quick feedback
- #818 🟡 Remove real sleeps (12 s in
test_pull_with_progress_logging; a 1 s readiness wait). Δ +0.5, effort Low, efficiency 0.38. - #819 🟡 Speed up alembic migration tests (18–30 s each; the 19 slowest tests). Δ +1.0, effort High, efficiency 0.25.
✅ Necessary / 🧪 First: no action needed
- Necessary: the 104 apparent "no assert" tests are fixtures named
test_client/test_database, or deliberate "does not raise" tests. - First: the suite shows behaviour-focused names, regression references and guard tests. Nothing here would raise this score in a measurable way.
Findings dropped after checking
- The 5 session-scoped fixtures only hold the Postgres server and template handles, which never change; every test gets its own cloned database.
- Of the 224 tests that read
datetime.now(), only one breaks the budget calendar guard's rule. It is covered by #810.
Suggested order
- Quick reliability wins: #809, #811, #810
- #812
- Quick wins in other tiers: #816, #814, #818, #817
- Larger refactors: #813 → #815, and #819
When the sub-issues are done, re-run the audit in a fresh conversation (no target scores in the prompt) to get an unbiased before/after comparison.
This tracker was created by an AI agent (OpenHands) on behalf of the user.
- Dominant language
- Python
- Stars
- 9
- Forks
- 17
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 396
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenHands/enterprise
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
OpenHands/enterprise#1269 ·
Maintainers usually reply within 1 day
-
type: test
Difficulty 1/5 Under an hour Newbie friendliness 84/100
OpenHands/enterprise#816 ·
Maintainers usually reply within 1 day
-
type: test
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
OpenHands/enterprise#814 ·
Maintainers usually reply within 1 day
-
type: test
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
OpenHands/enterprise#810 ·
Maintainers usually reply within 1 day
-
type: test
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
OpenHands/enterprise#811 ·
Maintainers usually reply within 1 day
All issues in OpenHands/enterprise
Similar issues
-
[Bug]: reme CLI crashes when the running server used a relative config pathPossibly taken @nurlanmnn claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
agentscope-ai/ReMe#770 ·
Maintainers usually reply within 1 day
-
Pi: paused model calls are reported as "not signed in" when the model's provider comes from an extensionPossibly taken @balcsida claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
microsoft/TokenEconomizer#288 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 67/100
sickn33/agentic-awesome-skills#2437 ·
Maintainers usually reply within 1 day
-
site-bug triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ethereum/consensus-specs#5846 ·
Maintainers usually reply within 1 day