Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)

オープン
#2,437 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
playwright, python
領域
backend, security

調査の方向性

Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.

索引モデルが issue の本文から書いたものです。

説明

🩺 Needs Triage
Summary

The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.

Why it matters

With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.

Reproduction (crawl4ai 0.9.5):

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

async def main():
    cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("browser:", r.status_code, r.markdown.raw_markdown.strip())
    async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))

asyncio.run(main())

Output:

browser: 200 #  expired.
badssl.com
http: False True

The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.

What I'm not proposing
  • Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
  • Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
  1. Next minor release: track whether ignore_https_errors was set explicitly. Resolve an unset value to True before every browser and context consumer (the managed launch flag in browser_manager.py and both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction through to_dict/clone and dump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-time FutureWarning saying the default will change and that ignore_https_errors=True keeps the current behaviour. Explicit True or False never warns.
  2. Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and docs/examples/cli/browser.yml (both show ignore_https_errors: true), and say how CLI users keep True; without a config file the CLI follows the BrowserConfig default, so it would migrate with the library.
  3. One or two minors later: change the default to False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an external cdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
  • Is a default change acceptable in principle with this notice period? If you'd rather keep True permanently, would a plain notice for insecure mode (without promising a change) be welcome instead?
  • Preferred surface for the warning: warnings.warn(FutureWarning), the crawler logger, or both?
  • Would you take a PR for step 1 alone first?

Happy to draft the PR for whichever steps you agree to.

主要言語
Python
スター
87.2k
フォーク
9.1k
平均マージ
1日 6時間
マージ済み PR(30日)
25

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

unclecode/crawl4ai のほかの issue

unclecode/crawl4ai の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。