Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug]: Text after an inline <link>, <meta>, <style> or <noscript> is dropped from cleaned_html and markdown

Abierto Apto para principiantes
  • ¿Algo no está bien?
#2,416 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@zhongxiao-chang ya está trabajando en esto.

Desde el 19/10/2026.

  • #2421 de @zhongxiao-chang — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
85/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, web-dev

Línea de trabajo

Lee el bucle de eliminación de etiquetas en crawl4ai/content_scraping_strategy.py alrededor de las líneas 827–830 y compara el comportamiento de remove() con el enfoque de drop_tree() descrito en el issue y en el issue relacionado #2377. Reproduce el problema con el fragmento HTML proporcionado y confirma que el texto posterior a cada elemento eliminado permanece en cleaned_html y en Markdown.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

crawl4ai version

0.9.4 (reproduced). The same code is in v0.9.5 and on main and develop (checked 2026-10-19).

Expected Behavior

When the scraper strips <style>, <link>, <meta> and <noscript> from the body, the text that follows each of them stays in cleaned_html and in the markdown.

Current Behavior

The text that follows each removed element is lost. LXMLWebScrapingStrategy._scrap removes these tags with lxml's remove() (crawl4ai/content_scraping_strategy.py, lines 827-830 in v0.9.5):

for tag in ["style", "link", "meta", "noscript"]:
    for element in body.xpath(f".//{tag}"):
        if element.getparent() is not None:
            element.getparent().remove(element)

lxml stores the text after an element as that element's .tail, and remove() discards it together with the element.

Wikipedia articles are hit in particular, because MediaWiki puts <link rel="mw:PageProp/Category"> inside paragraphs. https://en.wikipedia.org/wiki/Tokyo has 18 such <link> elements followed by text inside a <p>; one of them is followed by ", Tokyo is home to 26 of the world's 500 largest companies, ...", which never reaches the output. The crawl still reports success, so nothing signals the loss.

Is this reproducible?

Yes

Inputs Causing the Bug

Any HTML where <link>, <meta>, <style> or <noscript> is followed by text in the same parent, for example https://en.wikipedia.org/wiki/Tokyo.

Steps to Reproduce
  1. Run the snippet below.
  2. Compare the printed markdown with the paragraphs in HTML.
Code snippets
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

HTML = """<html><body><article>
<p>The port grew quickly, and as of 2020<link rel="mw:PageProp/Category" href="/x">, it handles more cargo than any other port.</p>
<p>The old warehouses stood on the quay<meta property="mw:PageProp/toc"> until the great fire destroyed them.</p>
<p>Styled text<style>.a{color:red}</style> continues after a style element.</p>
<p>Plain text<noscript>Enable JavaScript</noscript> continues after a noscript element.</p>
</article></body></html>"""

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("raw:" + HTML, config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS))
        print(result.markdown.raw_markdown)

asyncio.run(main())

Output on 0.9.4:

The port grew quickly, and as of 2020
The old warehouses stood on the quay
Styled text
Plain text

Expected:

The port grew quickly, and as of 2020, it handles more cargo than any other port.
The old warehouses stood on the quay until the great fire destroyed them.
Styled text continues after a style element.
Plain text continues after a noscript element.
Suggested fix

Replace element.getparent().remove(element) with element.drop_tree(). drop_tree() comes with lxml.html and moves the element's tail text onto the previous sibling or the parent. With this one-line change the snippet prints the expected output above.

Related:

  • #1938 / #1939 fixed the same loss in remove_empty_elements_fast() by moving .tail by hand.
  • #2377 (open) fixes it for links removed by exclude_external_links and similar options, using drop_tree().

This loop is a third place with the same pattern.

OS

Debian GNU/Linux 12 (bookworm), in the official unclecode/crawl4ai:0.9.4 Docker image (lxml 6.1.3)

Python version

3.12.14

Error logs & Screenshots (if applicable)

None. The crawl succeeds and the text is silently missing.

Lenguaje dominante
Python
Estrellas
85.9k
Forks
8.9k
Merge medio
5 d 7 h
PR fusionados (30 d)
23

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.