Pre-ranking for multivariate calibration assessment

Abierto
  • ¿Algo no está bien?
#1,064 2 comentarios 1 reacción 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
38/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
r
Área
analytics, data

Línea de trabajo

Empiece leyendo las secciones 2.1–2.2 de Gneiting et al. (2008) e inspeccione la nueva clase forecast_multivariate_sample. Compare integrar coverage en score.forecast_multivariate_sample() con añadir get_multivariate_ranks(), y aclare después la API prevista y el tratamiento de los rangos. Se considera terminado cuando exista una implementación documentada del pre-ranking con una salida de calibración que pueda visualizarse como histogramas de rangos.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

Implement pre-ranking methods from Gneiting et al. (2008) to assess calibration of multivariate forecasts via univariate rank histograms.

Motivation

When forecasting multiple correlated quantities (e.g., multiple locations, age groups, or forecast horizons), standard univariate calibration checks miss dependency structure. A forecast could be perfectly calibrated marginally (each location's PIT is uniform) but miss the correlation structure entirely. Gneiting et al. (2008) propose "pre-ranking" approaches that reduce multivariate calibration assessment to univariate ranks, which can then be visualised via rank histograms. This could complement the new forecast_multivariate_sample class by providing calibration diagnostics.

Pre-ranking methods

The paper describes two main pre-ranking approaches:

  1. Multivariate rank (Section 2.1): Based on pre-ranks using an orthant semi-ordering. For each point, count how many other points lie "to the lower left" in all dimensions. The observation's rank among samples is then interpreted as a calibration metric. Ties are resolved randomly.

  2. Minimum spanning tree (MST) rank (Section 2.2): Remove each point in turn and compute the MST length of the remaining points. Rank observation by where its MST length falls among sample MST lengths.

Design options

This could be implemented in two ways

Option A: Include in score()

Add coverage columns to score.forecast_multivariate_sample() output:

  • multivariate_coverage_50, multivariate_coverage_90, etc.
  • Each is 0/1 per forecast unit (like univariate interval_coverage_*)
  • Aggregation via summarise_scores() gives coverage proportions
  • Existing visualisation tools can display coverage deviations

This parallels how univariate interval coverage works and feels most natural.

Option B: Standalone function
get_multivariate_ranks(forecast_multivariate, method = "average_rank")

Returns the observation's rank among samples (1 to M+1). This can be passed to get_pit_histogram() for visualisation. Useful for detailed calibration diagnostics beyond coverage.

Reference

Gneiting, T., Stanberry, L. I., Grimit, E. P., Held, L., & Johnson, N. A. (2008). Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. Test, 17(2), 211-235. https://doi.org/10.1007/s11749-008-0114-x -- available at https://stat.uw.edu/sites/default/files/files/reports/2008/tr537.pdf

Lenguaje dominante
R
Estrellas
65
Forks
24
Merge medio
1 h 24 min
PR fusionados (30 d)
2

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de epiforecasts/scoringutils

Todos los issues de epiforecasts/scoringutils

Issues similares

Más issues de R

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.