Probabilistic measures with Monte Carlo draws
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- julia
- Domain
- data, machine-learning
Research direction
Start by reading the existing probabilistic-measure entry points and the CRPS implementation, then compare them with the proposal's array-of-draws API and weighting requirements. Done means supported measures accept draw arrays with same-sized weights where possible, while unsupported cases such as LogScore remain defined.
Written by the indexing model from the issue text.
Description
Sometimes we don't have a density function for a predictive distribution but instead have IID draws from that distribution. e.g. in Bayesian inference, we often only have access to the posterior predictive distribution through Monte Carlo draws. It would be useful if this package supported statistical estimation of the measure in such cases.
Proposal
I propose the API for probabilistic measures be expanded so that the first argument to a probabilistic measure can be an array of predictions representing these draws instead of just a distribution, where possible (e.g. this would not be possible for LogScore). When the first argument is an array of draws, then weights must be an array with the same size.
Example: CRPS
For example, the continuously ranked probability score (CRPS) can be defined for a predictive distribution q as
CRPS(q, y) = E[|ŷ-y|] - E[|ŷ-ŷ'|], where ŷ' and ŷ are independent random variables drawn from q.
For many named distributions, this can be evaluated exactly. But it can also be estimated from draws.
The naive estimator of CRPS for an IID sample of size N is.
mean(abs.(ŷ .- y)) - mean(abs.(ŷ .- ŷ')) * (N // (2 * (N-1))). For continuous predictions, this can be reformulated using order statistics to reduce the cost to O(Nlog(N)) (see https://hal.science/hal-02976423/document)
- Dominant language
- Julia
- Stars
- 23
- Forks
- 9
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 1
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from JuliaAI/StatisticalMeasures.jl
-
Difficulty 5/5 Over a week Newbie friendliness 45/100
JuliaAI/StatisticalMeasures.jl#107 · 5 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
JuliaAI/StatisticalMeasures.jl#62 · 5 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
JuliaAI/StatisticalMeasures.jl#27 · 1 comment ·
All issues in JuliaAI/StatisticalMeasures.jl
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
Sienna-Platform/PowerSimulations.jl#1737 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
FluxML/Optimisers.jl#254 ·
-
Minor: floodplain warning percentage, scalar output error message and a never-true identity checkOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
JuliaAI/MLJTransforms.jl#101 ·
Maintainers usually reply within 1 day
-
@contents blocks fail to render on Julia 1.14Possibly taken @sbfnk claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LuxDL/DocumenterVitepress.jl#410 ·