Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug]: Text after an inline <link>, <meta>, <style> or <noscript> is dropped from cleaned_html and markdown

未关闭 适合新手
  • 哪里不对？
#2,416 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

@zhongxiao-chang 已经在做这个了。

开始于 2026年10月19日。

  • #2421 来自 @zhongxiao-chang —— 未关闭

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
85/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
backend, web-dev

调研方向

阅读crawl4ai/content_scraping_strategy.py中约第827–830行的标签移除循环，并将remove()的行为与 issue 和相关 issue #2377 中描述的drop_tree()方法进行比较。使用提供的 HTML 片段复现，并确认每个被移除元素后面的文本仍保留在cleaned_html和 Markdown 中。

由索引模型根据 Issue 内容生成。

描述

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.4 (reproduced). The same code is in v0.9.5 and on main and develop (checked 2026-10-19).

Expected Behavior

When the scraper strips <style>, <link>, <meta> and <noscript> from the body, the text that follows each of them stays in cleaned_html and in the markdown.

Current Behavior

The text that follows each removed element is lost. LXMLWebScrapingStrategy._scrap removes these tags with lxml's remove() (crawl4ai/content_scraping_strategy.py, lines 827-830 in v0.9.5):

for tag in ["style", "link", "meta", "noscript"]:
    for element in body.xpath(f".//{tag}"):
        if element.getparent() is not None:
            element.getparent().remove(element)

lxml stores the text after an element as that element's .tail, and remove() discards it together with the element.

Wikipedia articles are hit in particular, because MediaWiki puts <link rel="mw:PageProp/Category"> inside paragraphs. https://en.wikipedia.org/wiki/Tokyo has 18 such <link> elements followed by text inside a <p>; one of them is followed by ", Tokyo is home to 26 of the world's 500 largest companies, ...", which never reaches the output. The crawl still reports success, so nothing signals the loss.

Is this reproducible?

Yes

Inputs Causing the Bug

Any HTML where <link>, <meta>, <style> or <noscript> is followed by text in the same parent, for example https://en.wikipedia.org/wiki/Tokyo.

Steps to Reproduce
  1. Run the snippet below.
  2. Compare the printed markdown with the paragraphs in HTML.
Code snippets
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

HTML = """<html><body><article>
<p>The port grew quickly, and as of 2020<link rel="mw:PageProp/Category" href="/x">, it handles more cargo than any other port.</p>
<p>The old warehouses stood on the quay<meta property="mw:PageProp/toc"> until the great fire destroyed them.</p>
<p>Styled text<style>.a{color:red}</style> continues after a style element.</p>
<p>Plain text<noscript>Enable JavaScript</noscript> continues after a noscript element.</p>
</article></body></html>"""

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("raw:" + HTML, config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS))
        print(result.markdown.raw_markdown)

asyncio.run(main())

Output on 0.9.4:

The port grew quickly, and as of 2020
The old warehouses stood on the quay
Styled text
Plain text

Expected:

The port grew quickly, and as of 2020, it handles more cargo than any other port.
The old warehouses stood on the quay until the great fire destroyed them.
Styled text continues after a style element.
Plain text continues after a noscript element.
Suggested fix

Replace element.getparent().remove(element) with element.drop_tree(). drop_tree() comes with lxml.html and moves the element's tail text onto the previous sibling or the parent. With this one-line change the snippet prints the expected output above.

Related:

  • #1938 / #1939 fixed the same loss in remove_empty_elements_fast() by moving .tail by hand.
  • #2377 (open) fixes it for links removed by exclude_external_links and similar options, using drop_tree().

This loop is a third place with the same pattern.

OS

Debian GNU/Linux 12 (bookworm), in the official unclecode/crawl4ai:0.9.4 Docker image (lxml 6.1.3)

Python version

3.12.14

Error logs & Screenshots (if applicable)

None. The crawl succeeds and the text is silently missing.

主要语言
Python
星标
85.9k
派生
8.9k
平均合并
5 天 6 小时
30 天内合并 PR
24

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。