Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[SKEEP-005]: Ground-truth validation for stateful metrics

Open
  • Is something wrong?
#1,223 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
kotlin, python, pytorch, scikit-learn
Domain
machine-learning, testing-qa

Research direction

Start by reading skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt and the existing ground-truth fixture pattern; the proposal file docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc has not yet been written. Draft the proposal to answer the fixture format, batch-sequence, tolerance, and metadata questions. Done means the proposal is registered in docs/modules/skeep/nav.adoc and the “Current Proposals” table, with its status awaiting maintainer review.

Written by the indexing model from the issue text.

Description

research size:m skeep skill:design tracking

Proposal document: docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc — not yet written; writing it is this issue's Lane 0 task
Status: Draft (proposal to be filed)
Branch: feature/skeep-005-ground-truth-stateful-metrics

Trigger

Compiler, graph-export, or runtime integration — specifically the test-integration
pattern every future Metric inherits. skainet-test-groundtruth's OperationExecutor
(skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt)
maps a test case's operation name to one stateless TensorOps call and returns one
Tensor<FP32, Float>. A Metric (sk.ainet.lang.nn.metrics.Metric) is stateful —
update() across many batches, compute() returning a scalar Double, reset() — and
there is no MetricExecutor or equivalent anywhere in the harness. That shape mismatch
is why the Precision/Recall/F1Score feature (#1222) ships with unit tests only and
never earns the "✅ ground-truth validated" badge tensor ops get.

Summary

Add a parallel MetricExecutor and GroundTruthMetricCase (predictions tensor, targets
tensor, expected scalar, metric parameters such as averaging mode) so metrics can be
cross-checked against sklearn / torchmetrics references the same way matmul is
cross-checked against PyTorch today, in the same CI job. Additive: OperationExecutor
and the existing GGUF op-fixture format are unchanged.

Improvising this inside whichever metric PR gets there first — instead of deciding its
shape once, durably — is exactly the drift SKEEP exists to prevent; hence a proposal
rather than a sub-issue of #1222.

Questions the proposal must answer (don't leave them to the implementer):

  • Python-side fixture format: a new @ExecutableMetric decorator in skainet-ground-truth,
    or reuse of @Executable with a scalar-tensor convention?
  • Does a metric case carry one batch or a sequence of batches (to exercise
    accumulation across update() calls)?
  • Scalar tolerance semantics vs. tensor element-wise tolerance.
  • Does this warrant its own GGUF metadata convention, or a lighter fixture format since
    metrics don't need the op-parameter machinery conv2d/pooling need?

Related DARC features

  • #1222 — Precision, Recall, F1Score (blocked for ground-truth coverage only; ships without it)
  • Any future metric (AUC, MeanSquaredError, …) inherits whatever this decides
  • #984 — Ground truth validation Phase 2 roadmap (this is a sibling concern to #985–#988)

Sub-issues

To be filed once the proposal is Accepted. Suggested phases: (0) spike the fixture
format against Accuracy, which already ships; (1) MetricExecutor +
GroundTruthMetricCase; (2) wire Precision/Recall/F1Score.

Status upkeep

  • Proposal registered in docs/modules/skeep/nav.adoc and the "Current Proposals" table
  • Maintainer moved status to Accepted
  • Implementation PR(s) linked here and flipped the proposal's Status: to Implemented
Dominant language
Kotlin
Stars
52
Forks
15
Avg merge
1d 15h
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SKaiNET-developers/SKaiNET

All issues in SKaiNET-developers/SKaiNET

Similar issues

More Kotlin issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.