Measures with bespoke aggregation methods
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- backend-api-design, machine-learning
Research direction
Read the evaluation and aggregation interfaces, especially measurements_plus, finish, and external_aggregation_mode, which the issue identifies as the relevant hooks. The example defines CurveAggregation and RocCurveMeasure and shows the desired ROC curve output across folds. Done means evaluate supports aggregating non-scalar measures across folds and can infer when per-observation measurements do not apply.
Written by the indexing model from the issue text.
Description
Hi,
It would be great if the MLJ evaluate function could also report measures that do not report a single scalar per observation. I am thinking in particular about ROC/PR curves.
My intuition is that it does make sense (but please comment otherwise) to evaluate such measures across folds and aggregate them in some way. I have managed to "hack" into the internals to do this and thought there might be interest to make this more readily available for users. If so, I am happy to help, otherwise equally happy to keep this on my side.
Aggregation Mechanism
ROC curves are defined by FPR/TPR vectors which may not be aligned. To do so I perform linear interpolation based on a common grid. Currently, the measure also reports AUC for plotting purposes but this does not need to be the case and could be reported independently by measures=[auc, RocCurveMeasure()].
Some difficulties I have encountered
- These measures are not defined at the observation level, in particular I had to use
per_observation=falseinevaluate. Could this be inferred from the measure? - Ideally the grid would be defined within
RocCurveMeasure, notCurveAggregationwhich is not exposed to the user. The problem is that the aggregation functionsmeasurements_plusandfinishare defined by the aggregation and don't know anything about the measure.
The code
using StatisticalMeasures
using StatisticalMeasuresBase
using Printf
using CairoMakie
using MLJLinearModels
using MLJBase
####################################################################
### Define an Aggregation function and
### - `measurements_plus`
### - `finish`
####################################################################
function interpolate_roc(fpr, tpr; grid=range(0.0, 1.0; length=1001))
length(fpr) == length(tpr) ||
throw(ArgumentError("fpr and tpr must have equal lengths"))
isempty(fpr) && throw(ArgumentError("ROC arrays must not be empty"))
issorted(fpr) || throw(ArgumentError("fpr must be increasing"))
map(grid) do u
i = searchsortedlast(fpr, u)
if i == 0
return Float64(first(tpr))
elseif i == length(fpr)
return Float64(last(tpr))
end
# Linear Interpolation
α = (u - fpr[i]) / (fpr[i+1] - fpr[i])
return (1 - α) * tpr[i] + α * tpr[i+1]
end
end
struct CurveAggregation{T} <: StatisticalMeasuresBase.AggregationMode
grid::T
end
CurveAggregation(;grid=range(0.0, 1.0; length=1001)) = CurveAggregation(grid)
function StatisticalMeasuresBase.measurements_plus(::CurveAggregation, folds_measurements, weights)
return ([
(auc=fold_measurements.auc*fold_weight, curve=fold_measurements.curve)
for (fold_measurements, fold_weight) in zip(folds_measurements, weights)
], length(folds_measurements))
end
StatisticalMeasuresBase.measurements_plus(::CurveAggregation, folds_measurements, ::Nothing) =
(folds_measurements, length(folds_measurements))
function StatisticalMeasuresBase.finish(measurements, normalizer, agg::CurveAggregation, skipnan, ::Val{true})
(
auc = sum(m.auc for m in measurements) / normalizer,
fpr = agg.grid,
tprs = stack([interpolate_roc(m.curve[1], m.curve[2]; grid=agg.grid) for m in measurements])
)
end
####################################################################
### Define a RocCurveMeasure measure
####################################################################
struct RocCurveMeasure end
function (m::RocCurveMeasure)(ŷ, y)
return (auc=auc(ŷ, y), curve=roc_curve(ŷ, y))
end
StatisticalMeasuresBase.external_aggregation_mode(::RocCurveMeasure) = CurveAggregation()
####################################################################
### Example Usage
####################################################################
# Run evaluation
X, y = MLJBase.make_moons(1_000)
e = evaluate(LogisticClassifier(), X, y;
measure= RocCurveMeasure(),
resampling=StratifiedCV(),
per_observation=false
)
# Extract Data
roc_curve_results = e.measurement[1]
grid = CurveAggregation().grid
μ_tpr = vec(mean(roc_curve_results.tprs, dims=2))
stder = vec(std(roc_curve_results.tprs, dims=2)) ./ sqrt(size(roc_curve_results.tprs, 2))
# Plot
fig = Figure()
ax = Axis(fig[1, 1],
title=@sprintf("AUC: %.2f", roc_curve_results.auc),
xlabel="False positive rate",
ylabel="True positive rate",
limits=((0, 1), (0, 1))
)
band!(ax, grid, μ_tpr .- 1.96stder, μ_tpr .+ 1.96stder, alpha=0.5)
lines!(ax, grid, μ_tpr, color=:blue)
ablines!(ax, 0, 1, color=:black)
fig
the
- Dominant language
- Julia
- Stars
- 23
- Forks
- 9
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 1
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from JuliaAI/StatisticalMeasures.jl
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
JuliaAI/StatisticalMeasures.jl#62 · 5 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
JuliaAI/StatisticalMeasures.jl#36 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
JuliaAI/StatisticalMeasures.jl#27 · 1 comment ·
All issues in JuliaAI/StatisticalMeasures.jl
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
Sienna-Platform/PowerSimulations.jl#1737 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
FluxML/Optimisers.jl#254 ·
-
Minor: floodplain warning percentage, scalar output error message and a never-true identity checkOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
JuliaAI/MLJTransforms.jl#101 ·
Maintainers usually reply within 1 day
-
@contents blocks fail to render on Julia 1.14Possibly taken @sbfnk claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LuxDL/DocumenterVitepress.jl#410 ·