Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
調査の方向性
Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.
索引モデルが issue の本文から書いたものです。
説明
Summary
The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.
Why it matters
With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.
Reproduction (crawl4ai 0.9.5):
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
async def main():
cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler() as crawler:
r = await crawler.arun("https://expired.badssl.com/", config=cfg)
print("browser:", r.status_code, r.markdown.raw_markdown.strip())
async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
r = await crawler.arun("https://expired.badssl.com/", config=cfg)
print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))
asyncio.run(main())
Output:
browser: 200 # expired.
badssl.com
http: False True
The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.
What I'm not proposing
- Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
- Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
- Next minor release: track whether
ignore_https_errorswas set explicitly. Resolve an unset value toTruebefore every browser and context consumer (the managed launch flag inbrowser_manager.pyand both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction throughto_dict/cloneanddump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-timeFutureWarningsaying the default will change and thatignore_https_errors=Truekeeps the current behaviour. ExplicitTrueorFalsenever warns. - Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and
docs/examples/cli/browser.yml(both showignore_https_errors: true), and say how CLI users keepTrue; without a config file the CLI follows theBrowserConfigdefault, so it would migrate with the library. - One or two minors later: change the default to
False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an externalcdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
- Is a default change acceptable in principle with this notice period? If you'd rather keep
Truepermanently, would a plain notice for insecure mode (without promising a change) be welcome instead? - Preferred surface for the warning:
warnings.warn(FutureWarning), the crawler logger, or both? - Would you take a PR for step 1 alone first?
Happy to draft the PR for whichever steps you agree to.
- 主要言語
- Python
- スター
- 87.2k
- フォーク
- 9.1k
- 平均マージ
- 1日 6時間
- マージ済み PR(30日)
- 25
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
unclecode/crawl4ai のほかの issue
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stack対応中かも @HhH666-ux が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
メンテナーはふだん 1 日以内に返信
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.config対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープン🐞 Bug 📌 Root caused
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
unclecode/crawl4ai#2494 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
⚙ Done ✨ Enhancement
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
メンテナーはふだん 1 日以内に返信
-
⚙ Done 🐞 Bug 📌 Root caused
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaオープン⚙ Done 🐞 Bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
unclecode/crawl4ai#2469 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
unclecode/crawl4ai の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
メンテナーはふだん 1 日以内に返信
-
documentation
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
メンテナーはふだん 1 日以内に返信
-
Install the skills as a Claude Code plugin対応中かも @ruban-24 が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
メンテナーはふだん 6 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: Publicationオープンdata
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100