Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[SKEEP-005]: Ground-truth validation for stateful metrics

Abierto
  • ¿Algo no está bien?
#1,223 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
kotlin, python, pytorch, scikit-learn
Área
machine-learning, testing-qa

Línea de trabajo

Start by reading skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt and the existing ground-truth fixture pattern; the proposal file docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc has not yet been written. Draft the proposal to answer the fixture format, batch-sequence, tolerance, and metadata questions. Done means the proposal is registered in docs/modules/skeep/nav.adoc and the “Current Proposals” table, with its status awaiting maintainer review.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

research size:m skeep skill:design tracking

Proposal document: docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc — not yet written; writing it is this issue's Lane 0 task
Status: Draft (proposal to be filed)
Branch: feature/skeep-005-ground-truth-stateful-metrics

Trigger

Compiler, graph-export, or runtime integration — specifically the test-integration
pattern every future Metric inherits. skainet-test-groundtruth's OperationExecutor
(skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt)
maps a test case's operation name to one stateless TensorOps call and returns one
Tensor<FP32, Float>. A Metric (sk.ainet.lang.nn.metrics.Metric) is stateful —
update() across many batches, compute() returning a scalar Double, reset() — and
there is no MetricExecutor or equivalent anywhere in the harness. That shape mismatch
is why the Precision/Recall/F1Score feature (#1222) ships with unit tests only and
never earns the "✅ ground-truth validated" badge tensor ops get.

Summary

Add a parallel MetricExecutor and GroundTruthMetricCase (predictions tensor, targets
tensor, expected scalar, metric parameters such as averaging mode) so metrics can be
cross-checked against sklearn / torchmetrics references the same way matmul is
cross-checked against PyTorch today, in the same CI job. Additive: OperationExecutor
and the existing GGUF op-fixture format are unchanged.

Improvising this inside whichever metric PR gets there first — instead of deciding its
shape once, durably — is exactly the drift SKEEP exists to prevent; hence a proposal
rather than a sub-issue of #1222.

Questions the proposal must answer (don't leave them to the implementer):

  • Python-side fixture format: a new @ExecutableMetric decorator in skainet-ground-truth,
    or reuse of @Executable with a scalar-tensor convention?
  • Does a metric case carry one batch or a sequence of batches (to exercise
    accumulation across update() calls)?
  • Scalar tolerance semantics vs. tensor element-wise tolerance.
  • Does this warrant its own GGUF metadata convention, or a lighter fixture format since
    metrics don't need the op-parameter machinery conv2d/pooling need?

Related DARC features

  • #1222 — Precision, Recall, F1Score (blocked for ground-truth coverage only; ships without it)
  • Any future metric (AUC, MeanSquaredError, …) inherits whatever this decides
  • #984 — Ground truth validation Phase 2 roadmap (this is a sibling concern to #985–#988)

Sub-issues

To be filed once the proposal is Accepted. Suggested phases: (0) spike the fixture
format against Accuracy, which already ships; (1) MetricExecutor +
GroundTruthMetricCase; (2) wire Precision/Recall/F1Score.

Status upkeep

  • Proposal registered in docs/modules/skeep/nav.adoc and the "Current Proposals" table
  • Maintainer moved status to Accepted
  • Implementation PR(s) linked here and flipped the proposal's Status: to Implemented
Lenguaje dominante
Kotlin
Estrellas
52
Forks
15
Merge medio
1 d 15 h
PR fusionados (30 d)
36

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SKaiNET-developers/SKaiNET

Todos los issues de SKaiNET-developers/SKaiNET

Issues similares

Más issues de Kotlin

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.