Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
调研方向
Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.
由索引模型根据 Issue 内容生成。
描述
Summary
The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.
Why it matters
With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.
Reproduction (crawl4ai 0.9.5):
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
async def main():
cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler() as crawler:
r = await crawler.arun("https://expired.badssl.com/", config=cfg)
print("browser:", r.status_code, r.markdown.raw_markdown.strip())
async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
r = await crawler.arun("https://expired.badssl.com/", config=cfg)
print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))
asyncio.run(main())
Output:
browser: 200 # expired.
badssl.com
http: False True
The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.
What I'm not proposing
- Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
- Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
- Next minor release: track whether
ignore_https_errorswas set explicitly. Resolve an unset value toTruebefore every browser and context consumer (the managed launch flag inbrowser_manager.pyand both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction throughto_dict/cloneanddump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-timeFutureWarningsaying the default will change and thatignore_https_errors=Truekeeps the current behaviour. ExplicitTrueorFalsenever warns. - Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and
docs/examples/cli/browser.yml(both showignore_https_errors: true), and say how CLI users keepTrue; without a config file the CLI follows theBrowserConfigdefault, so it would migrate with the library. - One or two minors later: change the default to
False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an externalcdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
- Is a default change acceptable in principle with this notice period? If you'd rather keep
Truepermanently, would a plain notice for insecure mode (without promising a change) be welcome instead? - Preferred surface for the warning:
warnings.warn(FutureWarning), the crawler logger, or both? - Would you take a PR for step 1 alone first?
Happy to draft the PR for whichever steps you agree to.
- 主要语言
- Python
- 星标
- 87.2k
- 派生
- 9.1k
- 平均合并
- 1 天 6 小时
- 30 天内合并 PR
- 25
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
unclecode/crawl4ai 的其他 Issue
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stack可能已有人在做 @HhH666-ux 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.config可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 70/100
unclecode/crawl4ai#2494 · 1 条评论 ·
维护者通常 1 天内回复
-
⚙ Done ✨ Enhancement
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 1 天内回复
-
⚙ Done 🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.media未关闭⚙ Done 🐞 Bug
难度 2/5 1-3 小时 新手友好度 78/100
unclecode/crawl4ai#2469 · 1 条评论 ·
维护者通常 1 天内回复
查看 unclecode/crawl4ai 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
documentation
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
Install the skills as a Claude Code plugin可能已有人在做 @ruban-24 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 6 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
data
难度 2/5 1-3 小时 新手友好度 66/100