Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug]: whitespace-only inline elements are removed from cleaned_html, merging words in raw_markdown (prose, not only <pre>/<code>)

Cerrado Apto para principiantes
  • ¿Algo no está bien?
#2,328 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Ya se ha fusionado un pull request relacionado.

  • #2329 de @drakeo338 — fusionado

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
83/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
data

Línea de trabajo

Comienza en content_scraping_strategy.py, en remove_empty_elements_fast, y rastrea cómo se gestionan el texto y los tails de los elementos eliminados. Reproduce el problema con la entrada HTML sin procesar proporcionada de /crawl usando content_source="cleaned_html"; se considera terminado cuando los elementos inline que solo contienen espacios en blanco ya no unen palabras en raw_markdown, preservando el comportamiento existente para el resto del contenido.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

⚙ Done 🐞 Bug 🥸 - Medium
crawl4ai version

0.9.3 (Docker image, REST /crawl and /md)

Expected Behavior

Text whose words are split into inline elements keeps its word spacing in raw_markdown, as it does when the markdown is generated from the raw HTML.

Current Behavior

remove_empty_elements_fast (content_scraping_strategy.py) removes whitespace-only inline elements outside <pre>/<code>, and the whitespace inside them goes with them. With the default content_source="cleaned_html", raw_markdown then merges the words: Hi there becomes Hithere.

Real-world trigger: pages whose scripts split text into per-character or per-word spans for animation. Today https://example.com rewrites its paragraph into one <span> per character with each space in its own <span> </span>. Through /md or /crawl (default cleaned_html), the paragraph comes back as Thisdomainisforuseindocumentationexampleswithoutneedingpermission.. Text on the same page that is not split into spans keeps its spaces.

Is this reproducible?

Yes

Inputs Causing the Bug
POST /crawl
{
  "urls": ["raw:<html><body><p><span>H</span><span>i</span><span> </span><span>t</span><span>h</span><span>e</span><span>r</span><span>e</span></p><p>plain text stays fine</p></body></html>"],
  "crawler_config": {"type": "CrawlerRunConfig", "params": {"cache_mode": "bypass",
    "markdown_generator": {"type": "DefaultMarkdownGenerator", "params": {"content_source": "cleaned_html"}}}}
}
Output
content_source raw_markdown
cleaned_html (default) Hithere\nplain text stays fine
raw_html Hi there\nplain text stays fine

cleaned_html no longer contains the <span> </span>:
<p><span>H</span><span>i</span><span>t</span><span>h</span><span>e</span><span>r</span><span>e</span></p>

Root cause

In remove_empty_elements_fast, a childless element whose stripped text has fewer than word_count_threshold words is removed. A whitespace-only element has zero words, so it is always removed. The function preserves the removed element's .tail, but not its own .text, so the space disappears. #1181 hit the same thing inside code and was fixed (v0.7.8) by skipping elements inside <pre>/<code>. Prose split into inline elements has no such guard. #2110 reports the same family of failure in PruningContentFilter on the fit_markdown path.

Suggested fix

When removing an element whose text is whitespace-only, keep the whitespace: append a single space to the previous sibling's .tail (or to parent.text), as is already done for .tail. Alternatively, never remove whitespace-only inline phrasing elements (span, em, strong, b, i, …).

Workaround

DefaultMarkdownGenerator(content_source="raw_html") (via /crawl; /md cannot set it). On the pages we checked it keeps the same words and links as cleaned_html and restores the spacing. fit_html also keeps spacing, but it drops links and pruned content.

Lenguaje dominante
Python
Estrellas
85.4k
Forks
8.9k
Merge medio
5 d 7 h
PR fusionados (30 d)
23

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.