Corpus-scale scan gaps (catalog scan, novelty accuracy)

Abierto
  • ¿Algo no está bien?
#413 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
20/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
testing-qa

Línea de trabajo

Read the listed issues and pull requests (#219, #311, #358, #403, #406, #408, #414, #417, and #370) to understand which gaps are already covered and the remaining scope. The body groups corpus-scan and evaluation problems into P0, P1, and P2, but does not define one change or its acceptance criteria; establish a bounded task and how to verify it before implementation.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Gaps found running dokimi over a ~2K-skill corpus sample (GitSkills study). P0 = corrupts reported numbers, P1 = blocks/distorts corpus runs, P2 = ergonomics.

P0 (existing issues)

  • B1 #219 bad LLM replies cached or misreported
  • B2 #311 (fixed by #358) extraction.reached counts spans of a failed batch
  • B3 #407 phase 0: 1 valid panel vote → agreement 1.0; ground_truth not forwarded; identity hash ignores role→model

P1

  • A1 YAML parse error reported as "missing a 'description'" (44.7K of 224K load errors) — covered by #403
  • A2 _scan_skill_files crashes on ENAMETOOLONG for a sibling path; should be an error row — PR #417
  • A3 cross-catalog checks (name-conflict, mirror-drift, near-duplicate) fire between unrelated skills of a corpus root; need per-skill mode — PR #414
  • A4 SKILL.md nested in another skill's bundle counted as separate skill (2,105 rows / 1,979 dirs) — PR #414
  • B4 partial result not written at --max-usd stop — #406
  • B5 judge output ceiling 4096 tokens; no cap-partial for analyze/atoms/probe/shorten; absent adjudication silent; no triage per-run estimate
  • B6 no built-in test-retest (--repeats 1 noise null)

P2

  • A5 catalog eval selects 1,741 skills where scan finds 2,105 on same root; rule not stated
  • A6 no public per-skill review API (review_skill_dir(path)) — PR #414
  • A7 no --jsonl / --jobs / resume for 1M+ skills — PR #414
  • A8 no rule filter (--ignore name-mismatch)
  • A9 no warning when judge is same model family as measured model (extend #408)
  • B7 routing output estimate low — #403
  • B8 replay provenance per answer — #370
Lenguaje dominante
Python
Estrellas
11
Forks
9
Merge medio
1 d 11 h
PR fusionados (30 d)
335

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de intel/dokimi

Todos los issues de intel/dokimi

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.