[Tests] Tracker: improve unit test quality (Farley score 7.2 → ~8.1)

Open
#820 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
15/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
testing

Research direction

This is a parent tracker for sub-issues #809 to #819, not a single change. Start from the audit table and pick one small sub-issue, such as #809 (SESSION_API_KEY isolation in the quota route tests) or #811 (RESEND_* env setup in test_resend_keycloak.py), then read the test files it names under tests/. Done means that sub-issue's tests pass in isolation and in the full suite.

Written by the indexing model from the issue text.

Description

type: test

Tracker for improving the unit test suite. It comes from an audit of tests/ (384 files, 6,875 test functions) against Dave Farley's 8 properties of good tests, at 79124d5. Each sub-issue below is linked as a GitHub sub-issue and grouped by the Farley property it improves.

Current Farley Score: 7.2 / 10 (Good) → about 8.1 if all sub-issues are done

Property Weight Score now Sub-issues
Repeatable 1.25 7.0 #809, #810
Atomic 1.0 7.5 #811, #812
Maintainable 1.5 6.5 #813, #814
Understandable 1.5 7.5 #815, #816
Granular 1.0 7.5 #817
Fast 0.75 6.5 #818, #819
Necessary 1.0 8.0 none (see below)
First (TDD) 1.0 7.0 none (see below)

Work order is reliability first: 🔴 Critical (Repeatable/Atomic), then 🟠 Medium, then 🟡 Low. See the priority tables below.

Priority tables

Critical items always come first, whatever their efficiency. Within Medium and Low, rows are sorted by efficiency (weight × Δ score ÷ effort, where Low=1, Medium=2, High=3).

🔴 Critical: tests may pass or fail for the wrong reason
Sub-issue Improvement Farley property Effort Status
#809 Isolate tests from an ambient SESSION_API_KEY (17 quota route tests fail on main when it is set) Repeatable Low Confirmed failing
#811 test_resend_keycloak.py sets RESEND_* env vars at import time, which leak into the whole session Atomic Low Confirmed leak, no failures yet
#810 Pin the clock in the budget maintenance-processor test and widen the calendar guard to all test files Repeatable Low Confirmed: breaks the guard's own rule
#812 Restore module and env state after importlib.reload, sys.modules and os.environ edits Atomic Medium Confirmed leaks, no failures yet
#760 Migration 186 tests fail in the full suite Repeatable — ✅ Fixed by #805
🟠 Medium: visible quality problems
Sub-issue Improvement Farley property Δ score Effort Efficiency
#813 Replace heavy patching of private names with shared fakes (677 patch targets; 103 tests with 8 or more patches) Maintainable +1.0 High 0.50
#814 Move 782 function-level imports to the top of the module Maintainable +0.3 Low 0.45
#815 Split very large test modules (up to 6.6k lines) and tests over 120 lines Understandable +0.5 High 0.25
#816 Fix the routes conftest.py docstring that still describes SQLite Understandable +0.1 Low 0.15
🟡 Low: developer experience
Sub-issue Improvement Farley property Δ score Effort Efficiency
#818 Remove real sleeps (one test waits 12 s; another uses a 1 s "is it ready" wait) Fast +0.5 Low 0.38
#817 Parametrize the 22 tests with 15 or more asserts (test_has_exact_mention has 40) Granular +0.3 Low 0.30
#819 Speed up the alembic migration tests (18–30 s each; the 19 slowest tests in the suite) Fast +1.0 High 0.25

Quick wins (low effort): #809, #810, #811, #814, #816, #817, #818


By Farley property

🔁 Repeatable: same result for any machine, environment or date
  • #809 🔴 Isolate tests from an ambient SESSION_API_KEY. 17 quota route tests fail on main when it is set. Effort: Low.
  • #810 🔴 Pin the clock in the budget maintenance-processor test and widen the calendar guard to all test files. Effort: Low.
  • #760 🔴 Migration 186 tests fail in the full suite. Already fixed by #805 (each xdist worker had a different encryption key).
⚛️ Atomic: no state leaks between tests
  • #811 🔴 test_resend_keycloak.py sets RESEND_* env vars for the whole session at import time. Effort: Low.
  • #812 🔴 (latent) Restore module and env state after importlib.reload, sys.modules and os.environ edits. Effort: Medium.
🔧 Maintainable: tests survive refactors
  • #813 🟠 Replace heavy patching of private names (677 targets; 103 tests with 8 or more patches) with shared fakes. Δ +1.0, effort High, efficiency 0.50.
  • #814 🟠 Move 782 function-level imports to module level. Δ +0.3, effort Low, efficiency 0.45.
📖 Understandable: easy to read and navigate
  • #815 🟠 Split very large modules (up to 6.6k lines) and tests over 120 lines. Δ +0.5, effort High, efficiency 0.25.
  • #816 🟠 Fix the stale SQLite reference in the routes conftest.py docstring. Δ +0.1, effort Low, efficiency 0.15.
🎯 Granular: one failure points to one cause
  • #817 🟡 Parametrize the 22 tests with 15 or more assertions (test_has_exact_mention has 40). Δ +0.3, effort Low, efficiency 0.30.
⚡ Fast: quick feedback
  • #818 🟡 Remove real sleeps (12 s in test_pull_with_progress_logging; a 1 s readiness wait). Δ +0.5, effort Low, efficiency 0.38.
  • #819 🟡 Speed up alembic migration tests (18–30 s each; the 19 slowest tests). Δ +1.0, effort High, efficiency 0.25.
✅ Necessary / 🧪 First: no action needed
  • Necessary: the 104 apparent "no assert" tests are fixtures named test_client/test_database, or deliberate "does not raise" tests.
  • First: the suite shows behaviour-focused names, regression references and guard tests. Nothing here would raise this score in a measurable way.
Findings dropped after checking
  • The 5 session-scoped fixtures only hold the Postgres server and template handles, which never change; every test gets its own cloned database.
  • Of the 224 tests that read datetime.now(), only one breaks the budget calendar guard's rule. It is covered by #810.
Suggested order
  1. Quick reliability wins: #809, #811, #810
  2. #812
  3. Quick wins in other tiers: #816, #814, #818, #817
  4. Larger refactors: #813 → #815, and #819

When the sub-issues are done, re-run the audit in a fresh conversation (no target scores in the prompt) to get an unbiased before/after comparison.


This tracker was created by an AI agent (OpenHands) on behalf of the user.

Dominant language
Python
Stars
9
Forks
17
Avg merge
1d 7h
Merged PRs (30d)
396

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenHands/enterprise

All issues in OpenHands/enterprise

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.