Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)

Abierto
#2,437 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
playwright, python
Área
backend, security

Línea de trabajo

Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

🩺 Needs Triage
Summary

The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.

Why it matters

With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.

Reproduction (crawl4ai 0.9.5):

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy

async def main():
    cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("browser:", r.status_code, r.markdown.raw_markdown.strip())
    async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
        r = await crawler.arun("https://expired.badssl.com/", config=cfg)
        print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))

asyncio.run(main())

Output:

browser: 200 #  expired.
badssl.com
http: False True

The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.

What I'm not proposing
  • Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
  • Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
  1. Next minor release: track whether ignore_https_errors was set explicitly. Resolve an unset value to True before every browser and context consumer (the managed launch flag in browser_manager.py and both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction through to_dict/clone and dump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-time FutureWarning saying the default will change and that ignore_https_errors=True keeps the current behaviour. Explicit True or False never warns.
  2. Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and docs/examples/cli/browser.yml (both show ignore_https_errors: true), and say how CLI users keep True; without a config file the CLI follows the BrowserConfig default, so it would migrate with the library.
  3. One or two minors later: change the default to False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an external cdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
  • Is a default change acceptable in principle with this notice period? If you'd rather keep True permanently, would a plain notice for insecure mode (without promising a change) be welcome instead?
  • Preferred surface for the warning: warnings.warn(FutureWarning), the crawler logger, or both?
  • Would you take a PR for step 1 alone first?

Happy to draft the PR for whichever steps you agree to.

Lenguaje dominante
Python
Estrellas
87.2k
Forks
9.1k
Merge medio
1 d 6 h
PR fusionados (30 d)
25

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.