Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

feat(insights): use Jev to check that insight claims attach numbers to the right metric, population and period

オープン
  • 何か問題がありますか？
#968 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
typescript
領域
ai, analytics

調査の方向性

Start with apps/insights/src/agent.ts and evals/quality.ts, then trace the existing evaluateWithJev calls and the evals README failure example. Review how results.json and the finish tool handle evaluation output. Done means the staged eval, log-only runtime check, and eventual enforcement path are defined and covered by the known-bad calibration case.

索引モデルが issue の本文から書いたものです。

説明

@izadoesdev

Problem

validateNumericGrounding (apps/insights/src/agent.ts) checks that every number in a published claim appears somewhere in the claim's cited sources. It doesn't check that the number belongs to the metric, population, period, or direction the claim attaches it to. The evals README records the failure this leaves open:

a brief assigned transaction count 100 to gross revenue 10000; the number-presence guard accepted it because 100 existed elsewhere

Revenue now avoids this through structured field selections, but per the README every other source still relies on the numeric guard. Catching misattribution in those cases is manual today: evals/quality.ts writes the semantic rubric in plain English as reviewRequired (for example "verify every final count against its exact date range and population … The 200 entrants are eligible website visitors, not login attempts"), and each of those cases prints REVIEW REQUIRED for a person to check.

Proposal

Use Jev, the evaluation model insights already calls through evaluateWithJev, as a semantic judge, rolled out so nothing blocks production until it has proven itself:

  1. Evals only. In evals/quality.ts:
    • each reviewRequired rubric becomes a Jev boolean question over the final outcome and the tool results it cites
    • each string evidence claim gets boolean questions for metric, population, period, and direction attribution against its resolved sources
    • probabilities are recorded in results.json, and the summary sorts REVIEW REQUIRED cases by the judge's uncertainty, so review starts with the ambiguous ones instead of all of them
    • a small frozen set of known-bad outcomes (starting with the transactions-as-revenue case above) acts as a calibration check. If Jev can't separate these from correct outcomes, we stop here with a cheap negative result.
  2. Runtime, log-only. Run the per-claim check in the finish tool for string claims, record the verdicts as log fields, and don't change behaviour.
  3. Enforce. When a claim is confidently misattributed, return a tool error naming the claim and the mismatch (metric, population, period, or direction), so the agent corrects it within its existing finish attempts, the same way numeric-grounding errors work today.

The questions for one finish attempt go in a single batched request through the existing evaluateWithJev, which bills input tokens only.

Questions

  • Is steps 1 → 3 the rollout you'd want, or should this stay an eval-only tool?

  • Should a confident mismatch block publication, or only demote the outcome to private?

  • Beyond insights, where else would you want Jev in analytics? Some I considered:

    • uptime content checks: a 200 response that is actually an error, challenge, maintenance, or empty page, since monitors only look at status codes today
    • long-tail error cause triage per fingerprint: deploy/chunk-load skew, third-party script, network, or app bug
    • session outcome classification: intent and whether the visitor finished or got stuck, from sampled event trails
    • referrer spam detection on distinct referrer domains

    I'd like to hear which of these you'd prioritise, or whether there's a use case you already have in mind.

Disclosure: I used Claude Code to trace the grounding validation and eval runner and to draft this write-up; I verified the file references and README quotes by hand.

主要言語
TypeScript
スター
1.2k
フォーク
227
平均マージ
21時間 11分
マージ済み PR（30日）
238

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

databuddy-analytics/Databuddy のほかの issue

databuddy-analytics/Databuddy の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。