Discussion: migrate BrowserConfig.ignore_https_errors default to False (deprecation path)
Mantenedores costumam responder em até 1 dia
Ninguém assumiu esta issue ainda.
Avaliação
- Dificuldade
- 5/5
- Tempo estimado
- Mais de uma semana
- Facilidade para iniciantes
- 35/100
Direção de pesquisa
Start by reading BrowserConfig and the consumers named in the proposal: browser_manager.py, both context creation calls, and the config serialization paths. Then review the BrowserConfig reference and CLI sample locations mentioned in the issue. The proposal is to stage a default change while preserving explicit True behavior; done means the maintainers agree on the migration and warning approach, since this issue asks for a decision rather than a patch.
Escrita pelo modelo de indexação a partir do texto da issue.
Descrição
Summary
The browser crawler accepts any TLS certificate by default (BrowserConfig.ignore_https_errors=True). The HTTP crawler verifies by default (HTTPCrawlerConfig.verify_ssl=True), and the Docker server forces verification on (enforce_egress in deploy/docker/egress_broker.py sets ignore_https_errors=False unless CRAWL4AI_ALLOW_INSECURE_TLS is set). This follows the 0.9.5 fix in GHSA-j2qh-f73r-mg3q, which made an explicit False effective for launched Chromium. I'd like to propose a staged change of the library's browser default, not a flip in one release, and ask whether you would accept it.
Why it matters
With the current default, a library caller who never chose a TLS setting gets content from whoever answers on the network path: an intercepting proxy, a captive portal, a hijacked DNS answer. By default CrawlResult has no dedicated certificate-validation outcome (ssl_certificate is only filled when fetch_ssl_certificate=True), so the content looks like the real site's. That matters for the uses crawl4ai is promoted for: RAG ingestion and agents acting on what they read.
Reproduction (crawl4ai 0.9.5):
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
async def main():
cfg = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler() as crawler:
r = await crawler.arun("https://expired.badssl.com/", config=cfg)
print("browser:", r.status_code, r.markdown.raw_markdown.strip())
async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) as crawler:
r = await crawler.arun("https://expired.badssl.com/", config=cfg)
print("http:", r.success, "CERTIFICATE_VERIFY_FAILED" in (r.error_message or ""))
asyncio.run(main())
Output:
browser: 200 # expired.
badssl.com
http: False True
The browser loads the expired-certificate page and returns its content. (On this tiny test page success is then False because the anti-bot minimal-text check fires, not because of the certificate.) The HTTP strategy refuses it.
What I'm not proposing
- Not changing the library/Docker trust boundary. I understand #2146 keeps destination filtering opt-in for trusted library callers. Would you treat the TLS verification default as a separate compatibility decision?
- Not breaking existing crawls of internal, dev or self-signed sites in one release.
Proposed migration
- Next minor release: track whether
ignore_https_errorswas set explicitly. Resolve an unset value toTruebefore every browser and context consumer (the managed launch flag inbrowser_manager.pyand both context-creation calls), so behaviour is unchanged. Keep the set/unset distinction throughto_dict/cloneanddump/load, so a cloned or serialized config doesn't silently pin the old default. When a browser starts with it unset, emit a one-timeFutureWarningsaying the default will change and thatignore_https_errors=Truekeeps the current behaviour. ExplicitTrueorFalsenever warns. - Docs and CLI: document the change in the BrowserConfig reference. Update the CLI help sample and
docs/examples/cli/browser.yml(both showignore_https_errors: true), and say how CLI users keepTrue; without a config file the CLI follows theBrowserConfigdefault, so it would migrate with the library. - One or two minors later: change the default to
False. The option already controls newly created Playwright contexts, persistent contexts and locally launched managed Chromium. Browsers reached through an externalcdp_url, whose existing contexts crawl4ai doesn't create, need separate consideration.
Questions for maintainers
- Is a default change acceptable in principle with this notice period? If you'd rather keep
Truepermanently, would a plain notice for insecure mode (without promising a change) be welcome instead? - Preferred surface for the warning:
warnings.warn(FutureWarning), the crawler logger, or both? - Would you take a PR for step 1 alone first?
Happy to draft the PR for whichever steps you agree to.
- Linguagem predominante
- Python
- Estrelas
- 87.2k
- Forks
- 9.1k
- Merge médio
- 1d 6h
- PRs com merge (30d)
- 25
Preparar o ambiente
- Inclui um Dockerfile ou arquivo Docker Compose
- Tem um modelo de pull request
- Ler o guia de contribuição
Primeiros passos
- Leia a issue inteira e depois o guia de contribuição do projeto.
- Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
- Faça um fork do repositório e trabalhe em uma branch.
- Abra um pull request que referencie o número da issue.
Mais de unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackTalvez já em andamento @HhH666-ux assumiu hoje. Aberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 85/100
Mantenedores costumam responder em até 1 dia
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configTalvez já em andamento Um pull request vinculado a esta issue está aberto ou já foi mesclado. Aberta🐞 Bug 📌 Root caused
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 70/100
unclecode/crawl4ai#2494 · 1 comentário ·
Mantenedores costumam responder em até 1 dia
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAberta⚙ Done ✨ Enhancement
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 77/100
Mantenedores costumam responder em até 1 dia
-
⚙ Done 🐞 Bug 📌 Root caused
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 72/100
Mantenedores costumam responder em até 1 dia
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAberta⚙ Done 🐞 Bug
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 78/100
unclecode/crawl4ai#2469 · 1 comentário ·
Mantenedores costumam responder em até 1 dia
Todas as issues de unclecode/crawl4ai
Issues semelhantes
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 76/100
Mantenedores costumam responder em até 1 dia
-
documentation
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 85/100
Mantenedores costumam responder em até 1 dia
-
Install the skills as a Claude Code pluginTalvez já em andamento @ruban-24 assumiu hoje. Aberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 76/100
Mantenedores costumam responder em até 6 dias
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemAberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationAbertadata
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 66/100