Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[SKEEP-005]: Ground-truth validation for stateful metrics

Aperta
  • Qualcosa non va?
#1,223 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
kotlin, python, pytorch, scikit-learn
Ambito
machine-learning, testing-qa

Direzione di ricerca

Start by reading skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt and the existing ground-truth fixture pattern; the proposal file docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc has not yet been written. Draft the proposal to answer the fixture format, batch-sequence, tolerance, and metadata questions. Done means the proposal is registered in docs/modules/skeep/nav.adoc and the “Current Proposals” table, with its status awaiting maintainer review.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

research size:m skeep skill:design tracking

Proposal document: docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc — not yet written; writing it is this issue's Lane 0 task
Status: Draft (proposal to be filed)
Branch: feature/skeep-005-ground-truth-stateful-metrics

Trigger

Compiler, graph-export, or runtime integration — specifically the test-integration
pattern every future Metric inherits. skainet-test-groundtruth's OperationExecutor
(skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt)
maps a test case's operation name to one stateless TensorOps call and returns one
Tensor<FP32, Float>. A Metric (sk.ainet.lang.nn.metrics.Metric) is stateful —
update() across many batches, compute() returning a scalar Double, reset() — and
there is no MetricExecutor or equivalent anywhere in the harness. That shape mismatch
is why the Precision/Recall/F1Score feature (#1222) ships with unit tests only and
never earns the "✅ ground-truth validated" badge tensor ops get.

Summary

Add a parallel MetricExecutor and GroundTruthMetricCase (predictions tensor, targets
tensor, expected scalar, metric parameters such as averaging mode) so metrics can be
cross-checked against sklearn / torchmetrics references the same way matmul is
cross-checked against PyTorch today, in the same CI job. Additive: OperationExecutor
and the existing GGUF op-fixture format are unchanged.

Improvising this inside whichever metric PR gets there first — instead of deciding its
shape once, durably — is exactly the drift SKEEP exists to prevent; hence a proposal
rather than a sub-issue of #1222.

Questions the proposal must answer (don't leave them to the implementer):

  • Python-side fixture format: a new @ExecutableMetric decorator in skainet-ground-truth,
    or reuse of @Executable with a scalar-tensor convention?
  • Does a metric case carry one batch or a sequence of batches (to exercise
    accumulation across update() calls)?
  • Scalar tolerance semantics vs. tensor element-wise tolerance.
  • Does this warrant its own GGUF metadata convention, or a lighter fixture format since
    metrics don't need the op-parameter machinery conv2d/pooling need?

Related DARC features

  • #1222 — Precision, Recall, F1Score (blocked for ground-truth coverage only; ships without it)
  • Any future metric (AUC, MeanSquaredError, …) inherits whatever this decides
  • #984 — Ground truth validation Phase 2 roadmap (this is a sibling concern to #985–#988)

Sub-issues

To be filed once the proposal is Accepted. Suggested phases: (0) spike the fixture
format against Accuracy, which already ships; (1) MetricExecutor +
GroundTruthMetricCase; (2) wire Precision/Recall/F1Score.

Status upkeep

  • Proposal registered in docs/modules/skeep/nav.adoc and the "Current Proposals" table
  • Maintainer moved status to Accepted
  • Implementation PR(s) linked here and flipped the proposal's Status: to Implemented
Lingua principale
Kotlin
Stelle
52
Fork
15
Merge medio
1g 1h
PR unite (30g)
41

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET

Tutte le issue di SKaiNET-developers/SKaiNET

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.