Tests that run on CRAN, possibly missing changes in other packages

Aberta
  • Algo errado?
#415 2 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
48/100
Tipo de issue
Refatoração
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
r
Domínio
testing-qa

Direção de pesquisa

Start by reviewing the snapshot-only test blocks and the five cores = 2 blocks named in the issue, including test_psis.R, test_gpdfit.R, test_relative_eff.R, test_E_loo.R, and test_loo_and_waic.R. Check testthat’s CRAN snapshot behavior and the CRAN macOS restriction described in the issue; decide which tests should run on CRAN and add fixed-value checks or narrower skips accordingly. Done means the relevant posterior numeric behavior and parallel paths are exercised on CRAN without exceeding its stated limits.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

projpred encountered a problem where some slow tests depending on other packages were not run on CRAN, and lme4 made a change which was then not caught by reverse dependency checks. I checked loo tests and currently there are also some things that are not tested in CRAN. The good thing is that those other packages are rstan, rstanarm, and posterior, so it is likely we would know about the changes.

Snapshot tests do not run on CRAN.

In testthat 3.3.2, expect_snapshot() and expect_snapshot_value() default to cran = FALSE, and loo never sets cran = TRUE. This is the biggest gap, larger than the skip_on_cran() blocks. Thirteen test_that() blocks contain only snapshot expectations, so on CRAN they check nothing. These blocks guard the numbers that loo gets from posterior:

loo code posterior function fixed-value check that runs on CRAN
psis() main path gpdfit(sort_x = FALSE) none (test_psis.R:13 is a snapshot)
psis() smoothing qgeneralized_pareto() none
loo::gpdfit() gpdfit(wip, min_grid_pts) none (test_gpdfit.R is snapshots only)
psislw() (old) gpdfit(wip = FALSE), qgp yes, test_psislw.R:30-33
relative_eff() ess_mean() none (test_relative_eff.R:7); line 14 only compares against ess_mean
E_loo() diagnostic pareto_khat(tail, ndraws_tail) none (test_E_loo.R:136,144)
loo()/waic() all the above plus matrixStats none (test_loo_and_waic.R:23)
  • Interface changes are already caught on CRAN. These functions run in tests that CRAN does run, so a removed or renamed posterior argument would cause an error.
  • Numeric changes are not caught. Examples are a new default grid, prior or tail length in gpdfit, ess_mean or pareto_khat. Only the old psislw() path has hard-coded values.

Should we

  • enable some snapshot tests in CRAN?
  • add new guard tests with small fixed non-random inputs that are run on CRAN?
  • trust that when something changes in posterior, we remember to check things in loo?
parallel tests do not run on CRAN

None of the cores = 2 code runs on CRAN. That covers fork (mclapply) and PSOCK (parLapply, plus clusterExport in relative_eff). The stated reason applies only to macOS.

Should we

  • in the 5 cores = 2 blocks, skip only on CRAN macOS, instead of on all of CRAN? This uses at most 2 cores, which CRAN allows.
Linguagem predominante
R
Estrelas
159
Forks
37
Merge médio
18h 1min
PRs com merge (30d)
8

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de stan-dev/loo

Todas as issues de stan-dev/loo

Issues semelhantes

Mais issues de R

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.