Research spike: semantic prototype rules with positive and negative exemplars

Aperta
  • Qualcosa non va?
#38 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
machine-learning, python
Ambito
data, machine-learning

Direzione di ricerca

Start by inspecting the repository’s existing rule matching, benchmark, and evaluation entry points, then identify an initial classification workload and its data splits. Pin one sentence encoder and compare regex-only, prototype-only, TF-IDF, and embedding-classifier baselines using positive exemplars and hard negatives. Report risk/coverage, cold and warm latency, memory, and exemplar evidence; keep calibration and test examples out of exemplar creation and threshold selection, and document where prototypes help, fail, or duplicate simpler baselines.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

help wanted research

Question

Can a small semantic-prototype matcher handle useful regions that regex rules miss while remaining selective and inspectable?

Benchmark-local scope

  • Pin one sentence encoder; do not perform model shopping.
  • Represent each class/route with positive exemplars and hard negatives.
  • Score absolute similarity plus positive-versus-negative/class margin.
  • Choose thresholds on a selection split, not the final test.
  • Return evidence: nearest positive/negative exemplar IDs, similarities, margin, and encoder revision.
  • Compare against regex-only, prototype-only, TF-IDF, and embedding-classifier baselines.

An initial classification workload is preferable. Whole-document similarity is not sufficient for span extraction.

Acceptance

  • Reproducible configuration and split protocol.
  • Risk/coverage curve rather than one accuracy number.
  • Cold/warm latency and memory measured.
  • No calibration/test examples used to create exemplars or select thresholds.
  • A short result note documents where prototypes help, fail, or duplicate simpler baselines.

Non-goals

  • Adding a stable public RuleFormat in the first pull request.
  • Core-runtime or RuleIR migration.
  • Claiming formal future-error guarantees.

If the spike is useful, its result can inform the RuleIR RFC and a later API proposal.

Lingua principale
Python
Stelle
34
Fork
9
Merge medio
9g 13h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/rulechef

Tutte le issue di KRLabsOrg/rulechef

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.