Scoped reload to measure source imported before the coverage engine starts

Aperta
  • Qualcosa non va?
#747 0 commenti 1 reazione 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
28/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
testing-qa

Direzione di ricerca

Start by reviewing the existing pth/embed machinery and the plugins documentation, then trace how --cov, source, and source_pkgs are resolved when coverage starts. Compare the snapshot-and-diff and source-discovery approaches, including the listed reload hazards; done means an agreed opt-in design with clear scope, safeguards, and documentation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Explore a scoped, opt-in mechanism for pytest-cov to measure code that was imported before the coverage engine started — the classic "plugin imports the project under test" problem — by reloading only the affected source modules once coverage is active.

This is a solution-focused follow-up to #635 (and the cluster #682, #414, #439, #397, #372, #359, #587). In #635 the conclusion was essentially "use coverage run -m pytest, or manually pre-start via the pth/env-var machinery, or scrub sys.modules yourself in the offending plugin," and @ionelmc rightly cautioned that broad sys.modules manipulation is a "big no-no" (it invites blame reports like #620). The goal here is to discuss whether a narrowly-scoped reload could sidestep that objection.

The limitation

When a plugin is imported before pytest-cov's engine starts (entry-point plugins, or anything pytest-enabler-style that imports the project under test to decide whether to enable coverage), every line executed at import time — import, class, def, decorators — is recorded as missed, because it never executes again under coverage. The plugins docs document this and state there's currently "no way to measure your pytest plugin."

Concretely: pytest-perf is itself a pytest plugin, so its own runner.py import-time lines are unmeasurable via pytest --cov; a 100% diff-coverage gate can't be met without either coverage run -m pytest or a per-project reload hack (jaraco/pytest-perf#18, jaraco/pytest-perf#23).

Approaches worth exploring

Both aim to reload only modules that (a) were imported before coverage started and (b) fall under the configured --cov/source — never stdlib, never third-party, never the whole sys.modules.

  1. Snapshot-and-diff. Capture the set of sys.modules keys as early as possible (e.g. via the pth/embed machinery that already exists for subprocess start-up). After the coverage engine is active, reload only the modules that appeared since the snapshot and that match the source configuration — i.e. exactly the modules a plugin pulled in early. This bounds the blast radius to plugin-activated source.

  2. Source-discovery. From the resolved --cov/source/source_pkgs configuration, discover the target modules and, for any already present in sys.modules when coverage starts, importlib.reload just those. Reloading is scoped strictly to the code under measurement.

Caveats to discuss

  • Reloading has real hazards: module-level singletons, registries, dataclass/enum identity, C extensions, and already-bound references to pre-reload objects. Scoping to source-under-measurement reduces but doesn't eliminate this.
  • It would need to be opt-in (a config flag), off by default, and clearly documented as "measures import-time lines at the cost of a scoped reload."
  • The existing coverage run -m pytest remains the clean answer for those who can wrap the invocation; this is for the plugin-activation workflows (pytest-enabler et al.) where wrapping is undesirable, per #635.

Question

Would a scoped, opt-in reload — bounded to the configured source and to modules imported before the engine started — be an acceptable direction, or are the reload hazards fundamental enough that documenting coverage run -m pytest should remain the only recommendation?

Lingua principale
Python
Stelle
2.1k
Fork
253
Merge medio
12g 4h
PR unite (30g)
4

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pytest-dev/pytest-cov

Tutte le issue di pytest-dev/pytest-cov

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.