Testing is far too fragile
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 22/100
Direzione di ricerca
The failures come from the image comparison in tests/integration/test_advanced_plot_examples.py and tests/integration/test_contour_plot_examples.py, which compare rendered plots against reference images in tests/reference-example-images. Run one failing case, such as test_example_1_basic_cylindrical, and read how the rms value is compared with the reference. The issue does not say whether the fix should be a tolerance change, regenerated references, or environment pinning, so a maintainer decision is needed first. Done means the suite passes across environments under the chosen approach.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Subject of the issue
I've just pulled the latest release, and 24 tests fail. They are all minor fails to do with image mismatch, but it's just too fragile.
================================================================= short test summary info ==================================================================
FAILED tests/integration/test_advanced_plot_examples.py::test_example_14_vector_with_colour_contour - Failed: Image mismatch for example 14: {'rms': np.float64(0.07727821087173495), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_advanced_plot_examples.py::test_example_15_polar_vector_plot - Failed: Image mismatch for example 15: {'rms': np.float64(1.2350842133667295), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference...
FAILED tests/integration/test_advanced_plot_examples.py::test_example_16b_basic_stream_plot - Failed: Image mismatch for example 16b: {'rms': np.float64(19.617677264399287), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_advanced_plot_examples.py::test_example_16c_enhanced_stream_plot - Failed: Image mismatch for example 16c: {'rms': np.float64(21.34025810212168), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference...
FAILED tests/integration/test_advanced_plot_examples.py::test_example_17_basic_stipple_plot - Failed: Image mismatch for example 17: {'rms': np.float64(0.21548615766477328), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_advanced_plot_examples.py::test_example_18_polar_stipple_plot - Failed: Image mismatch for example 18: {'rms': np.float64(0.14397116764560633), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_advanced_plot_examples.py::test_canari_1 - Failed: Image mismatch for example canari_1: {'rms': np.float64(0.02305860896416126), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/re...
FAILED tests/integration/test_advanced_plot_examples.py::test_example_40_tracks_polar_stereographic - Failed: Image mismatch for example 40: {'rms': np.float64(0.01662284425174423), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_contour_plot_examples.py::test_example_1_basic_cylindrical - Failed: Image mismatch for example 1: {'rms': np.float64(0.18352397386241887), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference...
FAILED tests/integration/test_contour_plot_examples.py::test_example_3_map_limits_and_levels - Failed: Image mismatch for example 3: {'rms': np.float64(0.08961256746282698), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference...
FAILED tests/integration/test_contour_plot_examples.py::test_example_4_north_pole_stereographic - Failed: Image mismatch for example 4: {'rms': np.float64(0.5533641260245302), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference-...
FAILED tests/integration/test_contour_plot_examples.py::test_example_5_south_pole_with_boundary - Failed: Image mismatch for example 5: {'rms': np.float64(0.5826523379800569), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference-...
FAILED tests/integration/test_contour_plot_examples.py::test_regression_npstere_da193_image - Failed: Image mismatch: {'rms': np.float64(0.011906453533254112), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference-example-imag...
FAILED tests/integration/test_contour_plot_examples.py::test_example_7_lat_pressure_zonal_mean - Failed: Image mismatch for example 7: {'rms': np.float64(0.18799208017491612), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference...
FAILED tests/integration/test_contour_plot_examples.py::test_example_8_log_scale_pressure - Failed: Image mismatch for example 8: {'rms': np.float64(0.11046718985438769), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference...
FAILED tests/integration/test_contour_plot_examples.py::test_example_19_multiple_subplots - Failed: Image mismatch for example 19: {'rms': np.float64(4.15330387349586), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference-e...
FAILED tests/integration/test_contour_plot_examples.py::test_example_21other_rgp_plasma - Failed: Image mismatch for example 21other: {'rms': np.float64(2.4241767024258962), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/refe...
FAILED tests/integration/test_contour_plot_examples.py::test_example_22_rgp_rotated_projection - Failed: Image mismatch for example 22: {'rms': np.float64(0.8955457536170187), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/reference...
FAILED tests/integration/test_contour_plot_examples.py::test_example_23_rotated_grid_axes_overlay - Failed: Image mismatch for example 23: {'rms': np.float64(0.28493929640571825), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_contour_plot_examples.py::test_example_34_lambert_conformal - Failed: Image mismatch for example 34: {'rms': np.float64(0.07193669150389208), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_contour_plot_examples.py::test_example_35_mollweide_projection - Failed: Image mismatch for example 35: {'rms': np.float64(0.21729055554358212), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_contour_plot_examples.py::test_example_36_mercator_projection - Failed: Image mismatch for example 36: {'rms': np.float64(0.22801506410925787), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_contour_plot_examples.py::test_example_37_orthographic_projection - Failed: Image mismatch for example 37: {'rms': np.float64(0.19407875641259678), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
FAILED tests/integration/test_contour_plot_examples.py::test_example_38_robinson_projection - Failed: Image mismatch for example 38: {'rms': np.float64(0.22239464014539573), 'expected': '/Users/bnl28/syncthing/Repositories/cf-plot/tests/referenc...
======================================================= 24 failed, 72 passed, 3 deselected in 46.75s =======================================================
Steps to reproduce
Attempt to run tests in a different environment than those in which it was built.
Your environment
The version of cf-plot and its dependencies in which you are encountering an issue.
Platform: macOS-15.7.7-arm64-arm-64bit
Python: 3.12.13
packaging: 26.0
numpy: 2.4.3
cfdm.core: 1.13.0.0
udunits2 library: /Users/bnl28/miniforge3/envs/work26/bin/../lib/libudunits2.dylib
HDF5 library: 1.14.6
netcdf library: 4.9.3
netCDF4: 1.7.3
h5netcdf: 1.8.1
h5py: 3.16.0
pyfive: 1.2.0.dev79+g34ae13ead
zarr: 3.1.5
fsspec: 2026.2.0
scipy: 1.17.1
dask: 2026.3.0
distributed: 2026.3.0
cftime: 1.6.5
cfunits: 3.3.7
cfdm: 1.13.0.0
esmpy/ESMF: not available
psutil: 7.2.2
matplotlib: 3.10.8
activestorage: 0.4.0
cfplot: 3.5.0
cf: 3.19.0
Expected behaviour
Tests should pass
Actual behaviour
Too many fail, but when you investigate they're all trivial:
As anb eaxmple:
- Lingua principale
- Python
- Stelle
- 21
- Fork
- 11
- Merge medio
- 3h 36m
- PR unite (30g)
- 2
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di NCAS-CMS/cf-plot
-
testing
Difficoltà 3/5 1-2 giorni Idoneità per principianti 58/100
-
Add `AGENTS.md` fileApertadocs enhancement
Difficoltà 3/5 1-2 giorni Idoneità per principianti 64/100
-
Add animation hooks into cf-plotForse di nuovo libera @bnlawrence l’ha presa 102 giorni fa e non c’è nessuna pull request aperta. Apertaenhancement
-
Ugly nemo plotting issueApertabug v35plus
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
Tutte le issue di NCAS-CMS/cf-plot
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 63/100
GAM-team/got-your-back#547 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
UKGovernmentBEIS/inspect_ai#5821 ·
I maintainer di solito rispondono entro 1 giorno
-
docs help wanted
Difficoltà 1/5 1-3 ore Idoneità per principianti 75/100
-
evidence wanted good first issue help wanted
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
open-coder-ai/agentseam#195 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno