Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

feat(insights): use Jev to check that insight claims attach numbers to the right metric, population and period

Abierto
  • ¿Algo no está bien?
#968 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
typescript
Área
ai, analytics

Línea de trabajo

Start with apps/insights/src/agent.ts and evals/quality.ts, then trace the existing evaluateWithJev calls and the evals README failure example. Review how results.json and the finish tool handle evaluation output. Done means the staged eval, log-only runtime check, and eventual enforcement path are defined and covered by the known-bad calibration case.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

@izadoesdev

Problem

validateNumericGrounding (apps/insights/src/agent.ts) checks that every number in a published claim appears somewhere in the claim's cited sources. It doesn't check that the number belongs to the metric, population, period, or direction the claim attaches it to. The evals README records the failure this leaves open:

a brief assigned transaction count 100 to gross revenue 10000; the number-presence guard accepted it because 100 existed elsewhere

Revenue now avoids this through structured field selections, but per the README every other source still relies on the numeric guard. Catching misattribution in those cases is manual today: evals/quality.ts writes the semantic rubric in plain English as reviewRequired (for example "verify every final count against its exact date range and population … The 200 entrants are eligible website visitors, not login attempts"), and each of those cases prints REVIEW REQUIRED for a person to check.

Proposal

Use Jev, the evaluation model insights already calls through evaluateWithJev, as a semantic judge, rolled out so nothing blocks production until it has proven itself:

  1. Evals only. In evals/quality.ts:
    • each reviewRequired rubric becomes a Jev boolean question over the final outcome and the tool results it cites
    • each string evidence claim gets boolean questions for metric, population, period, and direction attribution against its resolved sources
    • probabilities are recorded in results.json, and the summary sorts REVIEW REQUIRED cases by the judge's uncertainty, so review starts with the ambiguous ones instead of all of them
    • a small frozen set of known-bad outcomes (starting with the transactions-as-revenue case above) acts as a calibration check. If Jev can't separate these from correct outcomes, we stop here with a cheap negative result.
  2. Runtime, log-only. Run the per-claim check in the finish tool for string claims, record the verdicts as log fields, and don't change behaviour.
  3. Enforce. When a claim is confidently misattributed, return a tool error naming the claim and the mismatch (metric, population, period, or direction), so the agent corrects it within its existing finish attempts, the same way numeric-grounding errors work today.

The questions for one finish attempt go in a single batched request through the existing evaluateWithJev, which bills input tokens only.

Questions

  • Is steps 1 → 3 the rollout you'd want, or should this stay an eval-only tool?

  • Should a confident mismatch block publication, or only demote the outcome to private?

  • Beyond insights, where else would you want Jev in analytics? Some I considered:

    • uptime content checks: a 200 response that is actually an error, challenge, maintenance, or empty page, since monitors only look at status codes today
    • long-tail error cause triage per fingerprint: deploy/chunk-load skew, third-party script, network, or app bug
    • session outcome classification: intent and whether the visitor finished or got stuck, from sampled event trails
    • referrer spam detection on distinct referrer domains

    I'd like to hear which of these you'd prioritise, or whether there's a use case you already have in mind.

Disclosure: I used Claude Code to trace the grounding validation and eval runner and to draft this write-up; I verified the file references and README quotes by hand.

Lenguaje dominante
TypeScript
Estrellas
1.2k
Forks
227
Merge medio
21 h 11 min
PR fusionados (30 d)
238

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de databuddy-analytics/Databuddy

Todos los issues de databuddy-analytics/Databuddy

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.