Cache stores markdown generation errors and failed LLM extractions as successful results

Aperta
#2,520 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start at DefaultMarkdownGenerator.generate_markdown, where an exception becomes a MarkdownGenerationResult whose raw_markdown starts with "Error in markdown generation", and at LLMExtractionStrategy as reached through aprocess_html, which writes {"error": True} blocks into extracted_content. Then read the cache-write check in arun that #2519 added. Done means these failures are not written to the cache and a later CacheMode.ENABLED run retries them; the signal design is still open, so agree on it in the thread before coding.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐞 Bug 🩺 Needs Triage

Follow-up to #2519, which keeps crawls with success=False and results with no markdown out of the cache. Two kinds of failure still look like a success to that check, so they get cached.

Markdown generation errors. When DefaultMarkdownGenerator.generate_markdown hits an unexpected exception, it returns a MarkdownGenerationResult with raw_markdown="Error in markdown generation: ...". The crawl keeps success=True and the markdown is not None, so the error text is written to the cache. It replaces any good copy and replays as the page's markdown on later CacheMode.ENABLED runs. Confirmed by forcing an exception in the generator on the #2519 branch: the next crawl returned success=True, a cache hit, and the error text as markdown.

Failed LLM extractions. When an LLM call fails (rate limit, bad key), LLMExtractionStrategy returns a block like [{"error": True, "tags": ["error"], ...}]. aprocess_html serializes it into extracted_content with success=True, so it is cached and replayed without retrying the extraction. This one comes from reading the code and has not been reproduced.

Both need a failure signal the cache check in arun can see, since the crawl itself did not fail.

Lingua principale
Python
Stelle
87.2k
Fork
9.1k
Merge medio
1g 6h
PR unite (30g)
25

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.