Dimensional verdict scoring for multi-model judge (Meta-Ralph pattern)
まだ誰も着手していません。
評価
- 難易度
- 3/5
- 見積もり時間
- 半日
- 初心者へのやさしさ
- 42/100
調査の方向性
Start with automation/judge/verdict-schema.json, then read automation/judge/pre-push-judge.sh, the judge prompt templates, and plans/terraphim-skills-judge.md. Confirm how the six dimensions, thresholds, escalation fields, and weighted aggregation fit together. Done means the schema and prompts describe dimensional scoring and the judge plan reflects the proposed flow; feedback-loop work remains dependent on #597.
索引モデルが issue の本文から書いたものです。
説明
Summary
Replace binary pass/fail verdicts in the multi-model judge system with 6-dimensional continuous scoring. Each judge tier produces a score vector instead of a single verdict, enabling nuanced escalation decisions and feedback loops.
Motivation
Inspired by vibeship-spark-intelligence Meta-Ralph quality gating pattern. Current judge system uses 3-tier binary verdicts (pass/fail). Meta-Ralph scores across 6 dimensions with configurable thresholds.
Current State
Tier 1 (quick): pass/fail
|
v (if fail)
Tier 2 (deep): pass/fail
|
v (if disagree)
Tier 3 (tiebreaker): pass/fail
Proposed State
Tier 1 (quick): [reliability: 0.7, context_fit: 0.9, freshness: 0.8, ...]
|
v (if any dimension below threshold)
Tier 2 (deep): [reliability: 0.85, context_fit: 0.6, freshness: 0.9, ...]
|
v (if dimensions conflict across tiers)
Tier 3 (tiebreaker): [reliability: 0.9, context_fit: 0.8, freshness: 0.95, ...]
|
v
Final verdict: weighted aggregate across all tiers that responded
Six Scoring Dimensions
- Reliability -- How often has this type of finding been validated in past verdicts?
- Context fit -- How relevant is the finding to the specific code being reviewed?
- Freshness -- Is the finding based on current patterns or stale heuristics?
- Authority -- How credible is the source model for this type of finding?
- Signal strength -- Confidence in the signal (strong evidence vs. weak heuristic)
- Utility -- How actionable is the finding? (specific fix vs. vague warning)
Escalation Logic
Replace binary escalation with dimensional:
- Escalate when any dimension drops below tier-specific threshold (not just overall pass/fail)
- Skip escalation when all dimensions are above threshold even if one tier "failed" on a specific finding
- Aggregate final verdict as weighted average across tiers, with later tiers weighted higher on dimensions where earlier tiers scored low
Feedback Loop (Future)
Track whether findings were acted on (follow-rate):
- Finding surfaced -> developer fixed issue -> positive signal
- Finding surfaced -> developer dismissed -> negative signal
- Over time, adjust per-dimension weights based on follow-rate
This requires #597 (event sourcing) for tracking.
Schema Extension
Extend automation/judge/verdict-schema.json:
{
"dimensions": {
"reliability": { "score": 0.85, "threshold": 0.7 },
"context_fit": { "score": 0.9, "threshold": 0.6 },
"freshness": { "score": 0.8, "threshold": 0.5 },
"authority": { "score": 0.7, "threshold": 0.6 },
"signal_strength": { "score": 0.6, "threshold": 0.5 },
"utility": { "score": 0.9, "threshold": 0.7 }
},
"escalated": false,
"escalation_reason": null
}
Affected Components
automation/judge/verdict-schema.json(extend schema)automation/judge/pre-push-judge.sh(update prompt to request dimensional scores)- Judge prompt templates (instruct models to score per dimension)
- terraphim-skills judge plan (
plans/terraphim-skills-judge.md)
Dependencies
- #597 Event sourcing (for feedback loop tracking)
- #598 Budget-aware routing (dimensional scoring needs cost-aware model selection)
Estimated Effort
~4 hours for schema + prompt changes. Feedback loop integration is a follow-up after #597.
- 主要言語
- Rust
- スター
- 62
- フォーク
- 5
- 平均マージ
- 2時間 27分
- マージ済み PR（30日）
- 1
環境構築
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
terraphim/terraphim-ai のほかの issue
-
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
terraphim/terraphim-ai#885 ·
-
難易度 4/5 3〜5日 初心者へのやさしさ 55/100
terraphim/terraphim-ai#871 ·
-
enhancement
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
terraphim/terraphim-ai#810 · コメント 2 件 ·
-
enhancement
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
terraphim/terraphim-ai#729 ·
-
enhancement
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
terraphim/terraphim-ai#728 ·
terraphim/terraphim-ai の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
solana-foundation/pay-kit#341 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
anthropics/buffa#487 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
arkworks-rs/algebra#1161 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
lbjlaq/Antigravity-Manager#3525 · コメント 2 件 · リアクション 1 件 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
rustdesk/rustdesk-server#708 ·