Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

PaperFlow verbatim-quote validation rejects faithful quotes due to PDF extraction artifacts (whitespace, ligatures, hyphenation, page numbers, chunk boundaries)

Chiusa
  • Qualcosa non va?
#157 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Inizia in quantmind/flows/_paper_summary.py, in _validate_research_draft, e traccia il modo in cui PaperFlow.build() fornisce chunks e provenienza. Riproduci il problema con arXiv 1706.03762v7, quindi copri gli artefatti di estrazione elencati e i casi di confine tra chunks, mantenendo il rifiuto delle citazioni fabbricate. Il lavoro è completato quando le citazioni fedeli passano, le citazioni fabbricate continuano a essere rifiutate e l’esecuzione end-to-end di PaperFlow termina.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

stale

Summary

_validate_research_draft in quantmind/flows/_paper_summary.py requires a research finding's quote to be an exact substring of the cited chunk's text (finding.quote not in chunk.text). With PDF-sourced papers, pymupdf's extracted text contains layout artifacts inside sentences, so a model that quotes the words faithfully — but normalizes layout — fails the check, and PaperFlow.build() raises ValueError: research finding quote is not present in its chunk.

We hit this repeatedly running PaperFlow on the repo's own example paper (arXiv 1706.03762v7) with a non-OpenAI model (kimi-k3 via the litellm path), at temperature 0 and 1. Every rejected quote we inspected was word-for-word present in the paper; none was fabricated.

Artifact classes observed (all real cases from 1706.03762v7)

  1. Line breaks / irregular indentation inside sentences — extracted text: "permission to\nreproduce the tables and figures in this paper solely for use in journalistic or\n scholarly works."; the model quotes the sentence with single spaces.
  2. Ligature glyphs — pymupdf emits ff/fi (e.g. effective); models write effective. NFKC normalization folds these.
  3. End-of-line hyphenation — "English-\nto-German" (hyphen must be kept when joining) vs. cases where the hyphen should be dropped. Neither plain whitespace collapsing nor unconditional dehyphenation handles both.
  4. Page numbers interleaved mid-sentence at page breaks — "...faster than recurrent layers when the sequence\n6\n\nlength n is smaller than..." (the 6 is the page number). A faithful quote naturally omits it.
  5. Chunk-boundary spans and near-miss chunk attribution — with chunk_size=512, chunk_overlap=64, a faithful quote can cross the cited chunk's boundary, and models sometimes cite a neighboring chunk index within their group. The quote is verbatim in the material the agent was shown, but not within the single cited chunk.

Suggested direction

Keep the anti-fabrication guarantee (every quoted fragment must appear verbatim in the text the agent saw) but compare on a normalized form:

  • NFKC-normalize + collapse whitespace runs on both sides before the substring test;
  • try both hyphenation joins for end-of-line hyphens (-\n → "" and -\n → -);
  • strip standalone page-number lines;
  • as a fallback, match against the overlap-deduplicated concatenation of the agent's chunk group (treating chunk_index/page_number as provenance hints rather than exact pointers), so boundary-spanning quotes pass.

We implemented exactly this as a local wrapper around _validate_research_draft and confirmed: (a) all previously rejected faithful quotes pass, (b) a deliberately fabricated quote is still rejected, and (c) PaperFlow then completes end-to-end on 1706.03762v7. Happy to turn it into a PR if the direction sounds right.

Environment

  • quant-mind @ 10e9dbd0 (v0.2.0), editable install, Python 3.12.13 (aarch64 linux)
  • Model: litellm/moonshot/kimi-k3 (also reproduced with a local qwen3:14b via litellm/ollama_chat/...)
  • Same behavior at temperature=0.0 and 1.0
Lingua principale
Python
Stelle
3.1k
Fork
492
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di LLMQuant/quant-mind

Tutte le issue di LLMQuant/quant-mind

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.