Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)

未关闭
#2,437 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
playwright, python
领域
backend, security

调研方向

Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.

由索引模型根据 Issue 内容生成。

描述

🩺 Needs Triage
Summary

The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.

Why it matters

With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.

Reproduction (crawl4ai 0.9.5):

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

async def main():
    cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("browser:", r.status_code, r.markdown.raw_markdown.strip())
    async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))

asyncio.run(main())

Output:

browser: 200 #  expired.
badssl.com
http: False True

The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.

What I'm not proposing
  • Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
  • Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
  1. Next minor release: track whether ignore_https_errors was set explicitly. Resolve an unset value to True before every browser and context consumer (the managed launch flag in browser_manager.py and both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction through to_dict/clone and dump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-time FutureWarning saying the default will change and that ignore_https_errors=True keeps the current behaviour. Explicit True or False never warns.
  2. Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and docs/examples/cli/browser.yml (both show ignore_https_errors: true), and say how CLI users keep True; without a config file the CLI follows the BrowserConfig default, so it would migrate with the library.
  3. One or two minors later: change the default to False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an external cdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
  • Is a default change acceptable in principle with this notice period? If you'd rather keep True permanently, would a plain notice for insecure mode (without promising a change) be welcome instead?
  • Preferred surface for the warning: warnings.warn(FutureWarning), the crawler logger, or both?
  • Would you take a PR for step 1 alone first?

Happy to draft the PR for whichever steps you agree to.

主要语言
Python
星标
87.2k
派生
9.1k
平均合并
1 天 6 小时
30 天内合并 PR
25

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。