Relative <base href> produces scheme-less extracted links
维护者通常 1 天内回复
评估
调研方向
The relevant code is LXMLWebScrapingStrategy._process_element() in the content scraping strategy module, which reads <base href> and uses it as url without resolving it, plus the normalize_url helper it should match. Reproduce with the snippet in the issue to confirm the scheme-less /sub/page.html output. Done means relative and absolute <base href> values both yield absolute internal links. Pull request #2552 is already open against this issue, so check with the maintainers before starting.
由索引模型根据 Issue 内容生成。
描述
Description
LXMLWebScrapingStrategy._process_element() reads <base href> and assigns it to url as-is. When the page declares a relative base (e.g. <base href="/sub/">), every relative link is resolved against that relative string, so the extracted links come out scheme- and host-less (/sub/page.html) even though the same document without a <base> element yields an absolute link.
Reproduction
import sys
from lxml import html as lhtml
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
strategy = LXMLWebScrapingStrategy()
html = "<html><head><base href='/sub/'></head><body><a href='page.html'>Page</a></body></html>"
result = strategy.process_element("https://example.com/app/index.html", lhtml.fromstring(html))
print(list(result["internal_links_dict"]))
- Actual:
['/sub/page.html'] - Expected:
['https://example.com/sub/page.html']
Absolute <base href="https://cdn.example.com/docs/"> behaves as before.
Expected fix
Resolve the declared base against the page URL with urljoin(url, base_href) before it is used, so relative and absolute <base> values both produce absolute links. This is the same resolution normalize_url already performs when no <base> is present.
I will open a small PR with a regression test covering root-relative, document-relative, absolute and missing <base> values.
- 主要语言
- Python
- 星标
- 87.2k
- 派生
- 9.1k
- 平均合并
- 1 天 6 小时
- 30 天内合并 PR
- 25
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
unclecode/crawl4ai 的其他 Issue
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stack可能已有人在做 @HhH666-ux 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.config可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 70/100
unclecode/crawl4ai#2494 · 1 条评论 ·
维护者通常 1 天内回复
-
⚙ Done ✨ Enhancement
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 1 天内回复
-
⚙ Done 🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.media未关闭⚙ Done 🐞 Bug
难度 2/5 1-3 小时 新手友好度 78/100
unclecode/crawl4ai#2469 · 1 条评论 ·
维护者通常 1 天内回复
查看 unclecode/crawl4ai 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 85/100
521xueweihan/HelloGitHub#4385 ·
-
🐛 bug
难度 2/5 1-3 小时 新手友好度 72/100
pdm-project/pdm#4012 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 82/100
pollen-robotics/reachy_mini_conversation_app#611 ·
维护者通常 1 天内回复
-
难度 1/5 1-3 小时 新手友好度 91/100
aws-samples/sample-per-model-bedrock#913 ·
维护者通常 1 天内回复
-
lesson-submission pending-review
难度 1/5 1 小时以内 新手友好度 72/100
Ikalus1988/MisakaNet#5115 ·
维护者通常 1 天内回复