Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)

Đang mở
#2,437 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
playwright, python
Lĩnh vực
backend, security

Hướng nghiên cứu

Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

🩺 Needs Triage
Summary

The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.

Why it matters

With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.

Reproduction (crawl4ai 0.9.5):

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

async def main():
    cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("browser:", r.status_code, r.markdown.raw_markdown.strip())
    async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))

asyncio.run(main())

Output:

browser: 200 #  expired.
badssl.com
http: False True

The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.

What I'm not proposing
  • Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
  • Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
  1. Next minor release: track whether ignore_https_errors was set explicitly. Resolve an unset value to True before every browser and context consumer (the managed launch flag in browser_manager.py and both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction through to_dict/clone and dump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-time FutureWarning saying the default will change and that ignore_https_errors=True keeps the current behaviour. Explicit True or False never warns.
  2. Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and docs/examples/cli/browser.yml (both show ignore_https_errors: true), and say how CLI users keep True; without a config file the CLI follows the BrowserConfig default, so it would migrate with the library.
  3. One or two minors later: change the default to False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an external cdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
  • Is a default change acceptable in principle with this notice period? If you'd rather keep True permanently, would a plain notice for insecure mode (without promising a change) be welcome instead?
  • Preferred surface for the warning: warnings.warn(FutureWarning), the crawler logger, or both?
  • Would you take a PR for step 1 alone first?

Happy to draft the PR for whichever steps you agree to.

Ngôn ngữ chính
Python
Star
87.2k
Fork
9.1k
Merge trung bình
1 ngày 6 giờ
Pull request đã merge (30 ngày)
25

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của unclecode/crawl4ai

Tất cả issue của unclecode/crawl4ai

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.