Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: Text after an inline <link>, <meta>, <style> or <noscript> is dropped from cleaned_html and markdown

Aperta Adatta ai principianti
  • Qualcosa non va?
#2,416 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@zhongxiao-chang ci sta già lavorando.

Dal 19/10/2026.

  • #2421 di @zhongxiao-chang — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
85/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, web-dev

Direzione di ricerca

Leggi il ciclo di rimozione dei tag in crawl4ai/content_scraping_strategy.py intorno alle righe 827–830 e confronta il comportamento di remove() con l’approccio drop_tree() descritto nell’issue e nell’issue correlata #2377. Riproduci il problema con lo snippet HTML fornito e conferma che il testo successivo a ciascun elemento rimosso rimane in cleaned_html e nel Markdown.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.4 (reproduced). The same code is in v0.9.5 and on main and develop (checked 2026-10-19).

Expected Behavior

When the scraper strips <style>, <link>, <meta> and <noscript> from the body, the text that follows each of them stays in cleaned_html and in the markdown.

Current Behavior

The text that follows each removed element is lost. LXMLWebScrapingStrategy._scrap removes these tags with lxml's remove() (crawl4ai/content_scraping_strategy.py, lines 827-830 in v0.9.5):

for tag in ["style", "link", "meta", "noscript"]:
    for element in body.xpath(f".//{tag}"):
        if element.getparent() is not None:
            element.getparent().remove(element)

lxml stores the text after an element as that element's .tail, and remove() discards it together with the element.

Wikipedia articles are hit in particular, because MediaWiki puts <link rel="mw:PageProp/Category"> inside paragraphs. https://en.wikipedia.org/wiki/Tokyo has 18 such <link> elements followed by text inside a <p>; one of them is followed by ", Tokyo is home to 26 of the world's 500 largest companies, ...", which never reaches the output. The crawl still reports success, so nothing signals the loss.

Is this reproducible?

Yes

Inputs Causing the Bug

Any HTML where <link>, <meta>, <style> or <noscript> is followed by text in the same parent, for example https://en.wikipedia.org/wiki/Tokyo.

Steps to Reproduce
  1. Run the snippet below.
  2. Compare the printed markdown with the paragraphs in HTML.
Code snippets
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

HTML = """<html><body><article>
<p>The port grew quickly, and as of 2020<link rel="mw:PageProp/Category" href="/x">, it handles more cargo than any other port.</p>
<p>The old warehouses stood on the quay<meta property="mw:PageProp/toc"> until the great fire destroyed them.</p>
<p>Styled text<style>.a{color:red}</style> continues after a style element.</p>
<p>Plain text<noscript>Enable JavaScript</noscript> continues after a noscript element.</p>
</article></body></html>"""

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("raw:" + HTML, config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS))
        print(result.markdown.raw_markdown)

asyncio.run(main())

Output on 0.9.4:

The port grew quickly, and as of 2020
The old warehouses stood on the quay
Styled text
Plain text

Expected:

The port grew quickly, and as of 2020, it handles more cargo than any other port.
The old warehouses stood on the quay until the great fire destroyed them.
Styled text continues after a style element.
Plain text continues after a noscript element.
Suggested fix

Replace element.getparent().remove(element) with element.drop_tree(). drop_tree() comes with lxml.html and moves the element's tail text onto the previous sibling or the parent. With this one-line change the snippet prints the expected output above.

Related:

  • #1938 / #1939 fixed the same loss in remove_empty_elements_fast() by moving .tail by hand.
  • #2377 (open) fixes it for links removed by exclude_external_links and similar options, using drop_tree().

This loop is a third place with the same pattern.

OS

Debian GNU/Linux 12 (bookworm), in the official unclecode/crawl4ai:0.9.4 Docker image (lxml 6.1.3)

Python version

3.12.14

Error logs & Screenshots (if applicable)

None. The crawl succeeds and the text is silently missing.

Lingua principale
Python
Stelle
85.9k
Fork
8.9k
Merge medio
5g 6h
PR unite (30g)
24

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.