V2 (maintainers): suggest new CREs when no good match (post-pipeline)

オープン
  • 何か問題がありますか？
#1,110 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
30/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
backend-api-design

調査の方向性

This is a v2 feature for maintainers only, to be implemented after the current pipeline work (PR #1088) is stable. The issue describes detecting 'no good CRE' cases and proposing new CRE candidates. Start by understanding the existing OIE/librarian pipeline and the eval harness from PR #1088. The output schema for proposed CREs and suggested graph edges needs to be designed. The work is clearly marked as not for newcomers.

索引モデルが issue の本文から書いたものです。

説明

enhancement existing-maintainers-only not ready for development

Status

Maintainers only · Version 2 · after the current OIE/librarian pipeline work finishes.

Not for GSoC or drive-by contributors. Do not start implementation until the v1 pipeline (canonical gold fixes, retrieval levers, hop-distance eval) is landed and stable.

Label: existing-maintainers-only

Context

During OIE B2 / production-accuracy work (PR #1088 eval harness), the lab stack included a local Agentic graph/hint gold expand that scored Agentic 5/5 even though prodclone had no Agentic hub Links. That was eval-only (doctored answer key), but it points at a real product capability we want for v2:

When Module C / librarian cannot find a good existing CRE (or only weak multi-hop near-misses), suggest creating a new CRE (and optional Links / Related scaffolding) instead of forcing a bad mapping.

Related findings from hop-distance analysis on canonical gold (SUMMARY winner):

  • Exact Links bar ≈ 43%
  • Distance-1 soft (exact ∪ Related/Contains neighbor) ≈ 68%
  • 0% invented CRE ids in predictions today — the model stays on real CREs; we lack a deliberate “propose new CRE” path

Goal (v2)

  1. Detect “no good CRE” cases (e.g. below margin / only multi-hop / empty shortlist).
  2. Propose a new CRE candidate (name, description, suggested parent Contains and/or Related neighbors) from section text + graph hints.
  3. Keep this clearly separated from production Links writes — review queue / draft artifacts only until mentors approve.
  4. Optionally reuse patterns from the eval-only Agentic stub + graph/hint expand, without treating that as a ship metric.

Non-goals (v1 / now)

  • Implementing this during the current pipeline accuracy push.
  • Counting Agentic stub gold expands toward production accuracy.
  • Auto-writing Links to opencre.org without review.

Acceptance criteria (draft, when v2 starts)

  • Documented trigger conditions (“no good CRE”).
  • Output schema for proposed CRE + suggested graph edges.
  • Eval harness hook: count how often we would propose vs force-map (no gold rewrite).
  • Explicit flag default off in prod.

References

  • PR #1088 B2 denominators (local Agentic expand → 58/67 lab only).
  • Companion: fix canonical LLM Top 10 gold to AI-topic CREs / AI Exchange (in progress on accuracy track).
主要言語
Python
スター
210
フォーク
167
平均マージ
4日 12時間
マージ済み PR（30日）
28

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

OWASP/OpenCRE のほかの issue

OWASP/OpenCRE の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。