Relative <base href> produces scheme-less extracted links

未关闭
#2,550 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@HhH666-ux 已经在做这个了。

开始于 2026年11月13日。

  • #2552 来自 @HhH666-ux —— 未关闭

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
25/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
html, python
领域
backend

调研方向

The relevant code is LXMLWebScrapingStrategy._process_element() in the content scraping strategy module, which reads <base href> and uses it as url without resolving it, plus the normalize_url helper it should match. Reproduce with the snippet in the issue to confirm the scheme-less /sub/page.html output. Done means relative and absolute <base href> values both yield absolute internal links. Pull request #2552 is already open against this issue, so check with the maintainers before starting.

由索引模型根据 Issue 内容生成。

描述

Description

LXMLWebScrapingStrategy._process_element() reads <base href> and assigns it to url as-is. When the page declares a relative base (e.g. <base href="/sub/">), every relative link is resolved against that relative string, so the extracted links come out scheme- and host-less (/sub/page.html) even though the same document without a <base> element yields an absolute link.

Reproduction
import sys
from lxml import html as lhtml
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy

strategy = LXMLWebScrapingStrategy()
html = "<html><head><base href='/sub/'></head><body><a href='page.html'>Page</a></body></html>"
result = strategy.process_element("https://example.com/app/index.html", lhtml.fromstring(html))
print(list(result["internal_links_dict"]))
  • Actual: ['/sub/page.html']
  • Expected: ['https://example.com/sub/page.html']

Absolute <base href="https://cdn.example.com/docs/"> behaves as before.

Expected fix

Resolve the declared base against the page URL with urljoin(url, base_href) before it is used, so relative and absolute <base> values both produce absolute links. This is the same resolution normalize_url already performs when no <base> is present.

I will open a small PR with a regression test covering root-relative, document-relative, absolute and missing <base> values.

主要语言
Python
星标
87.2k
派生
9.1k
平均合并
1 天 6 小时
30 天内合并 PR
25

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。