Probabilistic measures with Monte Carlo draws

Open
#36 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
julia

Research direction

Start by reading the existing probabilistic-measure entry points and the CRPS implementation, then compare them with the proposal's array-of-draws API and weighting requirements. Done means supported measures accept draw arrays with same-sized weights where possible, while unsupported cases such as LogScore remain defined.

Written by the indexing model from the issue text.

Description

Sometimes we don't have a density function for a predictive distribution but instead have IID draws from that distribution. e.g. in Bayesian inference, we often only have access to the posterior predictive distribution through Monte Carlo draws. It would be useful if this package supported statistical estimation of the measure in such cases.

Proposal

I propose the API for probabilistic measures be expanded so that the first argument to a probabilistic measure can be an array of predictions representing these draws instead of just a distribution, where possible (e.g. this would not be possible for LogScore). When the first argument is an array of draws, then weights must be an array with the same size.

Example: CRPS

For example, the continuously ranked probability score (CRPS) can be defined for a predictive distribution q as
CRPS(q, y) = E[|ŷ-y|] - E[|ŷ-ŷ'|], where ŷ' and ŷ are independent random variables drawn from q.
For many named distributions, this can be evaluated exactly. But it can also be estimated from draws.

The naive estimator of CRPS for an IID sample of size N is.
mean(abs.(ŷ .- y)) - mean(abs.(ŷ .- ŷ')) * (N // (2 * (N-1))). For continuous predictions, this can be reformulated using order statistics to reduce the cost to O(Nlog(N)) (see https://hal.science/hal-02976423/document)

Dominant language
Julia
Stars
23
Forks
9
Avg merge
4d 2h
Merged PRs (30d)
1

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JuliaAI/StatisticalMeasures.jl

All issues in JuliaAI/StatisticalMeasures.jl

Similar issues

More Julia issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.