Measures with bespoke aggregation methods

Open
#107 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active

Research direction

Read the evaluation and aggregation interfaces, especially measurements_plus, finish, and external_aggregation_mode, which the issue identifies as the relevant hooks. The example defines CurveAggregation and RocCurveMeasure and shows the desired ROC curve output across folds. Done means evaluate supports aggregating non-scalar measures across folds and can infer when per-observation measurements do not apply.

Written by the indexing model from the issue text.

Description

Hi,

It would be great if the MLJ evaluate function could also report measures that do not report a single scalar per observation. I am thinking in particular about ROC/PR curves.

My intuition is that it does make sense (but please comment otherwise) to evaluate such measures across folds and aggregate them in some way. I have managed to "hack" into the internals to do this and thought there might be interest to make this more readily available for users. If so, I am happy to help, otherwise equally happy to keep this on my side.

Aggregation Mechanism

ROC curves are defined by FPR/TPR vectors which may not be aligned. To do so I perform linear interpolation based on a common grid. Currently, the measure also reports AUC for plotting purposes but this does not need to be the case and could be reported independently by measures=[auc, RocCurveMeasure()].

Some difficulties I have encountered

  • These measures are not defined at the observation level, in particular I had to use per_observation=false in evaluate. Could this be inferred from the measure?
  • Ideally the grid would be defined within RocCurveMeasure, not CurveAggregation which is not exposed to the user. The problem is that the aggregation functions measurements_plus and finish are defined by the aggregation and don't know anything about the measure.

The code

using StatisticalMeasures
using StatisticalMeasuresBase
using Printf
using CairoMakie
using MLJLinearModels
using MLJBase

####################################################################
### Define an Aggregation function and
### - `measurements_plus`
### - `finish`
####################################################################

function interpolate_roc(fpr, tpr; grid=range(0.0, 1.0; length=1001))
    length(fpr) == length(tpr) ||
        throw(ArgumentError("fpr and tpr must have equal lengths"))
    isempty(fpr) && throw(ArgumentError("ROC arrays must not be empty"))
    issorted(fpr) || throw(ArgumentError("fpr must be increasing"))

    map(grid) do u
        i = searchsortedlast(fpr, u)

        if i == 0
            return Float64(first(tpr))
        elseif i == length(fpr)
            return Float64(last(tpr))
        end

        # Linear Interpolation
        α = (u - fpr[i]) / (fpr[i+1] - fpr[i])
        return (1 - α) * tpr[i] + α * tpr[i+1]
    end
end

struct CurveAggregation{T} <: StatisticalMeasuresBase.AggregationMode 
    grid::T
end

CurveAggregation(;grid=range(0.0, 1.0; length=1001)) = CurveAggregation(grid)

function StatisticalMeasuresBase.measurements_plus(::CurveAggregation, folds_measurements, weights)
    return ([
        (auc=fold_measurements.auc*fold_weight, curve=fold_measurements.curve) 
        for (fold_measurements, fold_weight) in zip(folds_measurements, weights)
    ], length(folds_measurements))
end

StatisticalMeasuresBase.measurements_plus(::CurveAggregation, folds_measurements, ::Nothing) = 
    (folds_measurements, length(folds_measurements))

function StatisticalMeasuresBase.finish(measurements, normalizer, agg::CurveAggregation, skipnan, ::Val{true})
    (
        auc = sum(m.auc for m in measurements) / normalizer,
        fpr = agg.grid,
        tprs = stack([interpolate_roc(m.curve[1], m.curve[2]; grid=agg.grid) for m in measurements])
    )
end

####################################################################
### Define a RocCurveMeasure measure
####################################################################

struct RocCurveMeasure end

function (m::RocCurveMeasure)(ŷ, y)
    return (auc=auc(ŷ, y), curve=roc_curve(ŷ, y))
end

StatisticalMeasuresBase.external_aggregation_mode(::RocCurveMeasure) = CurveAggregation()

####################################################################
### Example Usage
####################################################################

# Run evaluation
X, y  = MLJBase.make_moons(1_000)
e = evaluate(LogisticClassifier(), X, y; 
    measure= RocCurveMeasure(), 
    resampling=StratifiedCV(), 
    per_observation=false
)

# Extract Data
roc_curve_results = e.measurement[1]
grid = CurveAggregation().grid
μ_tpr = vec(mean(roc_curve_results.tprs, dims=2))
stder = vec(std(roc_curve_results.tprs, dims=2)) ./ sqrt(size(roc_curve_results.tprs, 2))

# Plot
fig = Figure()
ax = Axis(fig[1, 1],
    title=@sprintf("AUC: %.2f", roc_curve_results.auc),
    xlabel="False positive rate",
    ylabel="True positive rate",
    limits=((0, 1), (0, 1))
)
band!(ax, grid, μ_tpr .- 1.96stder, μ_tpr .+ 1.96stder, alpha=0.5)
lines!(ax, grid, μ_tpr, color=:blue)
ablines!(ax, 0, 1, color=:black)
fig

Imagethe

Dominant language
Julia
Stars
23
Forks
9
Avg merge
4d 2h
Merged PRs (30d)
1

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JuliaAI/StatisticalMeasures.jl

All issues in JuliaAI/StatisticalMeasures.jl

Similar issues

More Julia issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.