Knowledge-Augmented Generation (KAG): evaluate schema-constrained graphs and logical-form reasoning on top of GraphRAG

Aperta
#855 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp

Direzione di ricerca

Start with the ROADMAP results for Phases 5.2–5.2.2 and 6.2.1 / #405, then inspect Rag.NET.Graph, Rag.NET.GraphRag, the graph store, and the Phase 5.2 benchmark and answer-level harness. The proposed evaluation covers schema-constrained extraction, logical-form decomposition, and chunk-to-graph indexing on MultiHop-RAG. Done means benchmark results against the existing controls and a decision on whether to pursue KAG, which needs a domain-specific corpus, package placement, and OpenSPG versus native .NET implementation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

What

Knowledge-Augmented Generation comes from Ant Group's OpenSPG framework: Liang et al., "KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation", arXiv:2409.13731, Sep 2024. It goes beyond GraphRAG in three ways:

  1. An LLM-friendly, schema-constrained knowledge representation. Entities and relations follow a domain schema, rather than whatever open extraction produces.
  2. Mutual indexing between the knowledge graph and the source chunks, so graph facts and text evidence point at each other.
  3. A logical-form-guided hybrid reasoning engine. Questions are decomposed into logical forms and answered by graph traversal and symbolic steps, with text retrieval alongside.

It targets multi-hop, fact-rigorous, professional-domain questions, such as legal, medical and compliance. The cost is upfront: schema design, a heavier extraction pipeline, and keeping the graph maintained.

What Rag.NET already has

KAG would build on existing pieces, not start from nothing.

  • GraphRAG: Rag.NET.Graph and Rag.NET.GraphRag, with entity extraction, a graph store, local search and global search.
  • The measured record (ROADMAP, Phases 5.2–5.2.2):
    • On MultiHop-RAG, GraphRAG retrieval did not help: −0.02761 against the candidate-set control.
    • Local search hurt answers, 0.210 against dense 0.350, because of context pollution.
    • Global search helped on entity questions, 0.844 against 0.772, and did not help on yes/no questions.
    • Phase 6.2.1 / #405 showed that the dropped graph endpoints are mostly common nouns and paraphrases, not entities the extractor missed.

So the open question is not "can Rag.NET do graphs". It is whether KAG's specific additions fix what the measurements showed GraphRAG getting wrong.

Proposal: measure before building

Building full KAG is a large piece of work, so this issue scopes it to an evaluation that decides whether to build it:

  1. Schema-constrained extraction. Run the existing extractor against a user-supplied domain schema of allowed entity and relation types. Then measure graph quality against today's open extraction on the same corpus. This tests whether a schema removes the noise behind the local-search pollution.
  2. Logical-form decomposition. Turn a multi-hop question into an explicit chain of graph lookups over the existing graph store, instead of a similarity walk. Measure it on the MultiHop-RAG questions that Phase 5.2 already ran.
  3. Chunk ↔ graph mutual index. Check what the existing graph store already records about source chunks, and what is missing for "fact → evidence chunk" citations.

Each step reuses the Phase 5.2 benchmark and answer-level harness, so the result is a number against the same controls, not a judgement call.

Needs a decision

  • Whether KAG is worth pursuing at all, given that GraphRAG did not beat dense retrieval on this corpus. A professional-domain corpus with a real schema may be needed before any result means anything.
  • Whether schema support goes into Rag.NET.Graph or a new package.
  • Whether to integrate with OpenSPG itself, which is a Java/Python service, or reimplement the ideas natively in .NET.

Generated by Claude Code

Lingua principale
C#
Stelle
5
Fork
2
Merge medio
14h 28m
PR unite (30g)
190

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di MarcelRoozekrans/Rag.NET

Tutte le issue di MarcelRoozekrans/Rag.NET

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.