V2 (maintainers): suggest new CREs when no good match (post-pipeline)

Aperta
  • Qualcosa non va?
#1,110 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend-api-design

Direzione di ricerca

This is a v2 feature for maintainers only, to be implemented after the current pipeline work (PR #1088) is stable. The issue describes detecting 'no good CRE' cases and proposing new CRE candidates. Start by understanding the existing OIE/librarian pipeline and the eval harness from PR #1088. The output schema for proposed CREs and suggested graph edges needs to be designed. The work is clearly marked as not for newcomers.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement existing-maintainers-only not ready for development

Status

Maintainers only · Version 2 · after the current OIE/librarian pipeline work finishes.

Not for GSoC or drive-by contributors. Do not start implementation until the v1 pipeline (canonical gold fixes, retrieval levers, hop-distance eval) is landed and stable.

Label: existing-maintainers-only

Context

During OIE B2 / production-accuracy work (PR #1088 eval harness), the lab stack included a local Agentic graph/hint gold expand that scored Agentic 5/5 even though prodclone had no Agentic hub Links. That was eval-only (doctored answer key), but it points at a real product capability we want for v2:

When Module C / librarian cannot find a good existing CRE (or only weak multi-hop near-misses), suggest creating a new CRE (and optional Links / Related scaffolding) instead of forcing a bad mapping.

Related findings from hop-distance analysis on canonical gold (SUMMARY winner):

  • Exact Links bar ≈ 43%
  • Distance-1 soft (exact ∪ Related/Contains neighbor) ≈ 68%
  • 0% invented CRE ids in predictions today — the model stays on real CREs; we lack a deliberate “propose new CRE” path

Goal (v2)

  1. Detect “no good CRE” cases (e.g. below margin / only multi-hop / empty shortlist).
  2. Propose a new CRE candidate (name, description, suggested parent Contains and/or Related neighbors) from section text + graph hints.
  3. Keep this clearly separated from production Links writes — review queue / draft artifacts only until mentors approve.
  4. Optionally reuse patterns from the eval-only Agentic stub + graph/hint expand, without treating that as a ship metric.

Non-goals (v1 / now)

  • Implementing this during the current pipeline accuracy push.
  • Counting Agentic stub gold expands toward production accuracy.
  • Auto-writing Links to opencre.org without review.

Acceptance criteria (draft, when v2 starts)

  • Documented trigger conditions (“no good CRE”).
  • Output schema for proposed CRE + suggested graph edges.
  • Eval harness hook: count how often we would propose vs force-map (no gold rewrite).
  • Explicit flag default off in prod.

References

  • PR #1088 B2 denominators (local Agentic expand → 58/67 lab only).
  • Companion: fix canonical LLM Top 10 gold to AI-topic CREs / AI Exchange (in progress on accuracy track).
Lingua principale
Python
Stelle
210
Fork
167
Merge medio
4g 12h
PR unite (30g)
28

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OWASP/OpenCRE

Tutte le issue di OWASP/OpenCRE

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.