test(e2e): align markers and execution entry points

Chiusa
#545 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Una pull request collegata è già stata integrata.

  • #559 di @yeongseon — integrata

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
58/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
azure, github-actions, python
Ambito
ci-cd, testing

Direzione di ricerca

Start by reading the Hatch task in pyproject.toml, the two modules under tests/e2e/, and the test command in .github/workflows/e2e-azure.yml; reproduce the reported collection failure with python -m pytest --collect-only -m e2e tests/e2e/ --no-cov. Update the marker and entry-point contract together, then run the focused regression check and confirm the selector collects 21 tests without changing default unit coverage. Check that missing E2E_BASE_URL gives a clear error for dedicated certification runs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ci needs-triage priority:medium test

Context

Review of azure-functions-validation-python at commit b38d702adb1039ed1eda2ef3799d873af2506f41.

The supported make e2e-local and make e2e-azure entry points invoke this Hatch script:

e2e-azure = "pytest -v -m e2e tests/e2e/"

However, neither tests/e2e/test_validation_e2e.py nor tests/e2e/test_pipeline_branches_e2e.py applies the e2e marker. The latter's module-level pytestmark is only a skipif for a missing base URL. The test package and shared conftest do not add an E2E collection marker.

A selector-level reproduction from the repository root, using the same selector as the Hatch task and no Azure connection, is:

python -m pytest --collect-only -m e2e tests/e2e/ --no-cov

Observed result at the reviewed commit:

no tests collected (21 deselected) in 0.07s
exit code: 5

Expected behavior: the dedicated E2E entry points select the 21 intended real-host tests.

This does not mean the release workflow skipped those tests. Its current command is pytest tests/e2e -v --no-cov ..., without -m e2e. Since the tests are unmarked, that command currently runs them despite the default marker exclusions. The latest successful v0.16.0 release, run 38485749334 / Azure job 115599053509, recorded 21 passed in 12.65s on Python 3.12.15.

A marker-only fix would break the release command: default pytest options exclude e2e, and they also enforce 95% coverage of local package code. The Hatch and workflow entry points must be updated together, selecting E2E explicitly and disabling local source coverage for HTTP-only remote tests.

Acceptance Checklist

  • Apply the e2e marker consistently to both real-host modules while preserving their intended configuration checks.
  • Update the Hatch E2E task and Azure workflow command together to use -m e2e --no-cov, or an equivalent explicit selector and coverage policy.
  • Verify that the dedicated selector collects the intended 21 tests rather than exiting with code 5.
  • Keep real-host E2E tests excluded from the default unit suite and leave the unit-test 95% coverage policy unchanged.
  • Ensure both Makefile entry points and the release workflow use the same E2E selection contract.
  • Provide a clear error for a dedicated certification invocation with missing E2E_BASE_URL, so an entirely skipped suite cannot be mistaken for a successful live verification.
  • Add a focused regression check for the marker/entry-point contract; it should not require provisioning Azure.
  • Preserve the existing body/query/path/header, async, response serialization, sanitized error, and worker context coverage in the release E2E suite.

Out of scope

  • Replacing the real Azure gate with import-only or mock-only tests.
  • Claiming that the current successful release ran zero E2E tests.
  • Altering runtime validation behavior or reducing the unit-test coverage threshold.
  • Deploying additional Azure resources merely to reproduce the collection failure.
  • Reviewing or changing E2E configuration in sibling repositories.

References

Lingua principale
Python
Stelle
13
Fork
4
Merge medio
14h 46m
PR unite (30g)
89

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.