[SKEEP-005]: Ground-truth validation for stateful metrics
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 25/100
- issue の種類
- 機能追加
- 明瞭さ
- おおむね明確
- 活発さ
- 静か
調査の方向性
Start by reading skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt and the existing ground-truth fixture pattern; the proposal file docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc has not yet been written. Draft the proposal to answer the fixture format, batch-sequence, tolerance, and metadata questions. Done means the proposal is registered in docs/modules/skeep/nav.adoc and the “Current Proposals” table, with its status awaiting maintainer review.
索引モデルが issue の本文から書いたものです。
説明
Proposal document: docs/modules/skeep/pages/005-ground-truth-stateful-metrics.adoc — not yet written; writing it is this issue's Lane 0 task
Status: Draft (proposal to be filed)
Branch: feature/skeep-005-ground-truth-stateful-metrics
Trigger
Compiler, graph-export, or runtime integration — specifically the test-integration
pattern every future Metric inherits. skainet-test-groundtruth's OperationExecutor
(skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/OperationExecutor.kt)
maps a test case's operation name to one stateless TensorOps call and returns one
Tensor<FP32, Float>. A Metric (sk.ainet.lang.nn.metrics.Metric) is stateful —
update() across many batches, compute() returning a scalar Double, reset() — and
there is no MetricExecutor or equivalent anywhere in the harness. That shape mismatch
is why the Precision/Recall/F1Score feature (#1222) ships with unit tests only and
never earns the "✅ ground-truth validated" badge tensor ops get.
Summary
Add a parallel MetricExecutor and GroundTruthMetricCase (predictions tensor, targets
tensor, expected scalar, metric parameters such as averaging mode) so metrics can be
cross-checked against sklearn / torchmetrics references the same way matmul is
cross-checked against PyTorch today, in the same CI job. Additive: OperationExecutor
and the existing GGUF op-fixture format are unchanged.
Improvising this inside whichever metric PR gets there first — instead of deciding its
shape once, durably — is exactly the drift SKEEP exists to prevent; hence a proposal
rather than a sub-issue of #1222.
Questions the proposal must answer (don't leave them to the implementer):
- Python-side fixture format: a new
@ExecutableMetricdecorator inskainet-ground-truth,
or reuse of@Executablewith a scalar-tensor convention? - Does a metric case carry one batch or a sequence of batches (to exercise
accumulation acrossupdate()calls)? - Scalar tolerance semantics vs. tensor element-wise tolerance.
- Does this warrant its own GGUF metadata convention, or a lighter fixture format since
metrics don't need the op-parameter machinery conv2d/pooling need?
Related DARC features
- #1222 — Precision, Recall, F1Score (blocked for ground-truth coverage only; ships without it)
- Any future metric (
AUC,MeanSquaredError, …) inherits whatever this decides - #984 — Ground truth validation Phase 2 roadmap (this is a sibling concern to #985–#988)
Sub-issues
To be filed once the proposal is Accepted. Suggested phases: (0) spike the fixture
format against Accuracy, which already ships; (1) MetricExecutor +
GroundTruthMetricCase; (2) wire Precision/Recall/F1Score.
Status upkeep
- Proposal registered in
docs/modules/skeep/nav.adocand the "Current Proposals" table - Maintainer moved status to
Accepted - Implementation PR(s) linked here and flipped the proposal's
Status:toImplemented
- 主要言語
- Kotlin
- スター
- 52
- フォーク
- 15
- 平均マージ
- 1日 15時間
- マージ済み PR（30日）
- 36
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
SKaiNET-developers/SKaiNET のほかの issue
-
coding good first issue size:xs skill:kotlin-core sub-issue
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
SKaiNET-developers/SKaiNET#1323 ·
メンテナーはふだん 1 日以内に返信
-
coding good first issue platform size:xs skill:js sub-issue
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
SKaiNET-developers/SKaiNET#1232 ·
メンテナーはふだん 1 日以内に返信
-
skeep tracking
難易度 5/5 1週間以上 初心者へのやさしさ 15/100
SKaiNET-developers/SKaiNET#1331 ·
メンテナーはふだん 1 日以内に返信
-
assessment size:s skill:review sub-issue
難易度 4/5 1〜2日 初心者へのやさしさ 18/100
SKaiNET-developers/SKaiNET#1330 ·
メンテナーはふだん 1 日以内に返信
-
documentation good first issue size:s skill:docs sub-issue
難易度 3/5 1〜2日 初心者へのやさしさ 78/100
SKaiNET-developers/SKaiNET#1329 ·
メンテナーはふだん 1 日以内に返信
SKaiNET-developers/SKaiNET の issue をすべて見る
似ている issue
-
bugbane/qf good first issue
難易度 2/5 1〜3時間 初心者へのやさしさ 64/100
osservatorionessuno/bugbane#173 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
-
Bayern feature:Application ready web
難易度 2/5 1〜3時間 初心者へのやさしさ 67/100
digitalfabrik/entitlementcard#3176 ·
メンテナーはふだん 7 日以内に返信
-
bug need-device
難易度 2/5 1〜3時間 初心者へのやさしさ 64/100
ramesh130/superplayer#495 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
maxrave-dev/SimpMusic#2649 ·
メンテナーはふだん 2 日以内に返信