Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: browser_mode=builtin fails with "CDP endpoint not ready" — Chromium segfaults (SIGSEGV) when spawned from gunicorn worker (v0.9.3)

Aperta
  • Qualcosa non va?
#2,238 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
docker, playwright, python
Ambito
api, backend, devops

Direzione di ricerca

Reproduce the failure with the Docker POST request and compare it with the working docker exec and standalone subprocess cases. Start in ManagedBrowser.start() and _initial_startup_check, then trace the call from api.py handle_crawl_request through crawler_pool.py. Done means builtin mode starts Chromium from the gunicorn worker, the CDP endpoint responds, and the crawl request succeeds.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

❓ Question 🐞 Bug
crawl4ai version

0.9.3

Expected Behavior

With browser_config.browser_mode = "builtin", the Docker server should launch its managed Chromium and return the crawled page. This worked on 0.9.1 — we use builtin mode as the only persistent-profile path that passes the server's UNTRUSTED validation (cookies/fingerprints accumulate across calls).

Current Behavior

Any POST /crawl with browser_mode="builtin" returns HTTP 500:

Exception: CDP endpoint at http://localhost:9222 is not ready after startup

ManagedBrowser.start() spawns Chromium via subprocess.Popen(args, stdout=PIPE, stderr=PIPE, preexec_fn=os.setpgrp), and the browser dies within ~500ms with SIGSEGV — logged by _initial_startup_check as:

Browser process terminated during startup | Code: -11 | STDOUT: (empty) | STDERR: (only dbus noise)

The interesting part: the exact same command line (copied from the verbose "Starting browser with args" log), same user, same env and same ulimits works fine when launched (1) via docker exec shell and (2) via a standalone python3 subprocess.Popen with identical pipes/preexec_fn — CDP answers on :9222 within ~1s in both cases. It only segfaults when the parent is the gunicorn UvicornWorker (1 worker, 4 threads, asyncio loop running). Deterministic: persists across container restarts; ~100% failure from the worker, 100% success from any other parent.

Is this reproducible?

Yes

Inputs Causing the Bug
# Any URL triggers it — only browser_mode matters
{
  "urls": ["https://example.com"],
  "browser_config": {"headless": true, "browser_mode": "builtin"},
  "crawler_config": {"cache_mode": "BYPASS"}
}
Steps to Reproduce
docker run -d --name c4ai -p 11235:11235 unclecode/crawl4ai:0.9.3
TOKEN=...  # CRAWL4AI_API_TOKEN from container env
curl -X POST localhost:11235/crawl \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com"],"browser_config":{"headless":true,"browser_mode":"builtin"},"crawler_config":{"cache_mode":"BYPASS"}}'
# → 500 {"error":"CDP endpoint at http://localhost:9222 is not ready after startup"}

# Contrast test inside the same container — works:
docker exec c4ai /home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome \
  --remote-debugging-port=9222 --user-data-dir=/tmp/t --headless=new --no-sandbox \
  --disable-dev-shm-usage about:blank
# → "DevTools listening on ws://127.0.0.1:9222/..." within ~1s
Code snippets
# Standalone inside the container — this WORKS (CDP OK in 1s),
# which isolates the bug to the gunicorn-worker parent context:
import subprocess, os, time, urllib.request
args = ["/home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome",
        "--remote-debugging-port=9222", "--user-data-dir=/tmp/bt", "--headless=new",
        "--window-size=1080,600", "--no-sandbox", "--disable-dev-shm-usage",
        "--disable-gpu", "about:blank"]
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=os.setpgrp)
time.sleep(1)
print(urllib.request.urlopen("http://localhost:9222/json/version", timeout=2).read())
OS

Linux x86_64 (Docker image unclecode/crawl4ai:0.9.3; host with standard Docker defaults, shm_size 4gb, ulimits nofile 65536)

Python version

3.12.14

Browser

No response

Browser version

Chromium (Playwright-installed, launched by ManagedBrowser)

Error logs & Screenshots (if applicable)

[CRAWL] api - ERROR - Crawl error: CDP endpoint at http://localhost:9222 is not ready after startup
Traceback (most recent call last):
File "/app/api.py", line 714, in handle_crawl_request
crawler = await get_crawler(browser_config)
File "/app/crawler_pool.py", line 114, in get_crawler
await crawler.start()
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 183, in start
...
Exception: CDP endpoint at http://localhost:9222 is not ready after startup

[BROWSER] Browser process terminated during startup | Code: -11 | STDOUT: | STDERR: [45969:45984:ERROR:dbus/bus.cc:405] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

Launch args used (from verbose log):

/home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome --remote-debugging-port=9222 --user-data-dir=/tmp/browser-profile-XXXX --headless=new --window-size=1080,600 --no-sandbox --disable-dev-shm-usage --no-first-run --no-default-browser-check --disable-infobars --ignore-certificate-errors --disable-blink-features=AutomationControlled --disable-gpu ...

Suspects: something in the worker's process state (event-loop signal handlers?

threaded fork+exec interaction?) trips Chromium's early startup — inherited

signal dispositions or seccomp/rlimit nuance not visible in /proc//limits.

Workaround: none found short of avoiding builtin mode.

Lingua principale
Python
Stelle
84.5k
Fork
8.7k
Merge medio
3g 9h
PR unite (30g)
17

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.