Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)

Aberta
#2,437 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
playwright, python
Domínio
backend, security

Direção de pesquisa

Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

🩺 Needs Triage
Summary

The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.

Why it matters

With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.

Reproduction (crawl4ai 0.9.5):

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

async def main():
    cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("browser:", r.status_code, r.markdown.raw_markdown.strip())
    async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))

asyncio.run(main())

Output:

browser: 200 #  expired.
badssl.com
http: False True

The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.

What I'm not proposing
  • Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
  • Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
  1. Next minor release: track whether ignore_https_errors was set explicitly. Resolve an unset value to True before every browser and context consumer (the managed launch flag in browser_manager.py and both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction through to_dict/clone and dump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-time FutureWarning saying the default will change and that ignore_https_errors=True keeps the current behaviour. Explicit True or False never warns.
  2. Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and docs/examples/cli/browser.yml (both show ignore_https_errors: true), and say how CLI users keep True; without a config file the CLI follows the BrowserConfig default, so it would migrate with the library.
  3. One or two minors later: change the default to False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an external cdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
  • Is a default change acceptable in principle with this notice period? If you'd rather keep True permanently, would a plain notice for insecure mode (without promising a change) be welcome instead?
  • Preferred surface for the warning: warnings.warn(FutureWarning), the crawler logger, or both?
  • Would you take a PR for step 1 alone first?

Happy to draft the PR for whichever steps you agree to.

Linguagem predominante
Python
Estrelas
87.2k
Forks
9.1k
Merge médio
1d 6h
PRs com merge (30d)
25

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de unclecode/crawl4ai

Todas as issues de unclecode/crawl4ai

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.