Support allocation scoring via forecast_allocation class

Aperta
  • Qualcosa non va?
#1,113 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
r
Ambito
analytics, data

Direzione di ricerca

Inizia leggendo la discussione sullo scoring delle allocazioni in #1069 e la relativa implementazione del variogramma in #1111, quindi confronta il costruttore e le metriche proposti con le convenzioni di scoringutils. Risolvi le questioni di progettazione aperte prima di implementare as_forecast_allocation(), gpl_loss() e gpl_regret(); il lavoro è completo quando le allocazioni precalcolate possono essere valutate e riepilogate tramite il workflow indicato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Summary

Add a forecast_allocation class to support scoring of resource allocation forecasts using generalised piecewise linear (GPL) loss functions, complementing the allocation scoring framework in the alloscore package (Gerding et al.).

Motivation

Allocation scoring (Gerding et al., 2023) evaluates forecasts through the lens of constrained resource allocation.
The central question is whether a forecast leads to good resource allocation decisions when a decision-maker must distribute a limited budget K across targets (e.g. locations).

This is directly relevant to public health forecasting (distributing vaccines, antivirals, hospital beds across regions) and was discussed in #1069 alongside the variogram score.

The alloscore package implements both the optimisation step (finding optimal allocations from forecast distributions) and the scoring step (evaluating allocations against outcomes).
This proposal covers only the scoring step in scoringutils, with alloscore handling the optimisation.

Reference: Gerding, A., Reich, N.G., Rogers, B. and Ray, E.L. (2023). Evaluating infectious disease forecasts with allocation scoring rules. arXiv:2312.16201

Proposed design

New class: forecast_allocation

An allocation forecast stores, per target within an allocation group:

  • The allocation amount (predicted)
  • The observed demand (observed)
  • GPL loss parameters for scoring
Column Type Description
observed numeric Observed demand at this target
predicted numeric Allocated amount at this target
.alloc_group_id any Groups targets sharing a budget constraint
K numeric Total resource budget for this group
kappa numeric GPL scale parameter (default: 1)
alpha numeric GPL asymmetry parameter (default: 0.5)
w numeric Per-unit cost weight (default: 1)

The .alloc_group_id column plays the same structural role as .mv_group_id in forecast_multivariate_sample.

Constructor
as_forecast_allocation(
  data,
  forecast_unit = NULL,
  alloc_across = NULL,  # columns defining what is allocated across
  K = NULL,             # column name (required)
  kappa = NULL,         # column name or default value
  alpha = NULL          # column name or default value
)
Scoring metrics
  • gpl_loss(): realised GPL loss summed within each allocation group
  • gpl_regret(): excess loss relative to oracle allocation with perfect foresight
User workflow
# User computes allocations via alloscore or their own method
# Then evaluates them in scoringutils:
forecast <- as_forecast_allocation(
  allocation_data,
  alloc_across = "location",
  K = "K", alpha = "alpha"
)

scores <- score(forecast)
scores |> summarise_scores(by = "model")
Relationship to alloscore

This design complements alloscore rather than replacing it:

Concern alloscore scoringutils
Compute optimal allocations from forecast distributions Yes No
Score pre-computed allocations against outcomes Yes Yes
Evaluation pipeline (summarise, compare, visualise) Limited Yes
Generality

The design works for any constrained vector prediction problem with asymmetric loss, not just epidemic resource allocation.

Open questions

  1. Should as_forecast_allocation() strictly require sum(w * predicted) == K or allow under-utilisation?
  2. How to handle the g (increment) function in GPL loss: support identity and log as built-in string options?
  3. Should oracle computation live in scoringutils, or should users provide oracle scores from alloscore?
  4. Should GPL parameters be data columns, metric arguments via purrr::partial(), or both?

Related issues

  • #1069 (original variogram/allocation score discussion)
  • #1111 (variogram score implementation)

This was opened by a bot. Please ping @seabbs for any questions.

Lingua principale
R
Stelle
65
Fork
24
Merge medio
1h 24m
PR unite (30g)
2

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di epiforecasts/scoringutils

Tutte le issue di epiforecasts/scoringutils

Issue simili

Altre issue su R

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.