Tests that run on CRAN, possibly missing changes in other packages

Aperta
  • Qualcosa non va?
#415 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
r
Ambito
testing-qa

Direzione di ricerca

Start by reviewing the snapshot-only test blocks and the five cores = 2 blocks named in the issue, including test_psis.R, test_gpdfit.R, test_relative_eff.R, test_E_loo.R, and test_loo_and_waic.R. Check testthat’s CRAN snapshot behavior and the CRAN macOS restriction described in the issue; decide which tests should run on CRAN and add fixed-value checks or narrower skips accordingly. Done means the relevant posterior numeric behavior and parallel paths are exercised on CRAN without exceeding its stated limits.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

projpred encountered a problem where some slow tests depending on other packages were not run on CRAN, and lme4 made a change which was then not caught by reverse dependency checks. I checked loo tests and currently there are also some things that are not tested in CRAN. The good thing is that those other packages are rstan, rstanarm, and posterior, so it is likely we would know about the changes.

Snapshot tests do not run on CRAN.

In testthat 3.3.2, expect_snapshot() and expect_snapshot_value() default to cran = FALSE, and loo never sets cran = TRUE. This is the biggest gap, larger than the skip_on_cran() blocks. Thirteen test_that() blocks contain only snapshot expectations, so on CRAN they check nothing. These blocks guard the numbers that loo gets from posterior:

loo code posterior function fixed-value check that runs on CRAN
psis() main path gpdfit(sort_x = FALSE) none (test_psis.R:13 is a snapshot)
psis() smoothing qgeneralized_pareto() none
loo::gpdfit() gpdfit(wip, min_grid_pts) none (test_gpdfit.R is snapshots only)
psislw() (old) gpdfit(wip = FALSE), qgp yes, test_psislw.R:30-33
relative_eff() ess_mean() none (test_relative_eff.R:7); line 14 only compares against ess_mean
E_loo() diagnostic pareto_khat(tail, ndraws_tail) none (test_E_loo.R:136,144)
loo()/waic() all the above plus matrixStats none (test_loo_and_waic.R:23)
  • Interface changes are already caught on CRAN. These functions run in tests that CRAN does run, so a removed or renamed posterior argument would cause an error.
  • Numeric changes are not caught. Examples are a new default grid, prior or tail length in gpdfit, ess_mean or pareto_khat. Only the old psislw() path has hard-coded values.

Should we

  • enable some snapshot tests in CRAN?
  • add new guard tests with small fixed non-random inputs that are run on CRAN?
  • trust that when something changes in posterior, we remember to check things in loo?
parallel tests do not run on CRAN

None of the cores = 2 code runs on CRAN. That covers fork (mclapply) and PSOCK (parLapply, plus clusterExport in relative_eff). The stated reason applies only to macOS.

Should we

  • in the 5 cores = 2 blocks, skip only on CRAN macOS, instead of on all of CRAN? This uses at most 2 cores, which CRAN allows.
Lingua principale
R
Stelle
159
Fork
37
Merge medio
18h 1m
PR unite (30g)
8

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di stan-dev/loo

Tutte le issue di stan-dev/loo

Issue simili

Altre issue su R

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.