Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug]: browser_mode=builtin fails with "CDP endpoint not ready" — Chromium segfaults (SIGSEGV) when spawned from gunicorn worker (v0.9.3)

Đang mở
  • Có gì không ổn?
#2,238 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
docker, playwright, python
Lĩnh vực
api, backend, devops

Hướng nghiên cứu

Reproduce the failure with the Docker POST request and compare it with the working docker exec and standalone subprocess cases. Start in ManagedBrowser.start() and _initial_startup_check, then trace the call from api.py handle_crawl_request through crawler_pool.py. Done means builtin mode starts Chromium from the gunicorn worker, the CDP endpoint responds, and the crawl request succeeds.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

❓ Question 🐞 Bug
crawl4ai version

0.9.3

Expected Behavior

With browser_config.browser_mode = "builtin", the Docker server should launch its managed Chromium and return the crawled page. This worked on 0.9.1 — we use builtin mode as the only persistent-profile path that passes the server's UNTRUSTED validation (cookies/fingerprints accumulate across calls).

Current Behavior

Any POST /crawl with browser_mode="builtin" returns HTTP 500:

Exception: CDP endpoint at http://localhost:9222 is not ready after startup

ManagedBrowser.start() spawns Chromium via subprocess.Popen(args, stdout=PIPE, stderr=PIPE, preexec_fn=os.setpgrp), and the browser dies within ~500ms with SIGSEGV — logged by _initial_startup_check as:

Browser process terminated during startup | Code: -11 | STDOUT: (empty) | STDERR: (only dbus noise)

The interesting part: the exact same command line (copied from the verbose "Starting browser with args" log), same user, same env and same ulimits works fine when launched (1) via docker exec shell and (2) via a standalone python3 subprocess.Popen with identical pipes/preexec_fn — CDP answers on :9222 within ~1s in both cases. It only segfaults when the parent is the gunicorn UvicornWorker (1 worker, 4 threads, asyncio loop running). Deterministic: persists across container restarts; ~100% failure from the worker, 100% success from any other parent.

Is this reproducible?

Yes

Inputs Causing the Bug
# Any URL triggers it — only browser_mode matters
{
  "urls": ["https://example.com"],
  "browser_config": {"headless": true, "browser_mode": "builtin"},
  "crawler_config": {"cache_mode": "BYPASS"}
}
Steps to Reproduce
docker run -d --name c4ai -p 11235:11235 unclecode/crawl4ai:0.9.3
TOKEN=...  # CRAWL4AI_API_TOKEN from container env
curl -X POST localhost:11235/crawl \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com"],"browser_config":{"headless":true,"browser_mode":"builtin"},"crawler_config":{"cache_mode":"BYPASS"}}'
# → 500 {"error":"CDP endpoint at http://localhost:9222 is not ready after startup"}

# Contrast test inside the same container — works:
docker exec c4ai /home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome \
  --remote-debugging-port=9222 --user-data-dir=/tmp/t --headless=new --no-sandbox \
  --disable-dev-shm-usage about:blank
# → "DevTools listening on ws://127.0.0.1:9222/..." within ~1s
Code snippets
# Standalone inside the container — this WORKS (CDP OK in 1s),
# which isolates the bug to the gunicorn-worker parent context:
import subprocess, os, time, urllib.request
args = ["/home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome",
        "--remote-debugging-port=9222", "--user-data-dir=/tmp/bt", "--headless=new",
        "--window-size=1080,600", "--no-sandbox", "--disable-dev-shm-usage",
        "--disable-gpu", "about:blank"]
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=os.setpgrp)
time.sleep(1)
print(urllib.request.urlopen("http://localhost:9222/json/version", timeout=2).read())
OS

Linux x86_64 (Docker image unclecode/crawl4ai:0.9.3; host with standard Docker defaults, shm_size 4gb, ulimits nofile 65536)

Python version

3.12.14

Browser

No response

Browser version

Chromium (Playwright-installed, launched by ManagedBrowser)

Error logs & Screenshots (if applicable)

[CRAWL] api - ERROR - Crawl error: CDP endpoint at http://localhost:9222 is not ready after startup
Traceback (most recent call last):
File "/app/api.py", line 714, in handle_crawl_request
crawler = await get_crawler(browser_config)
File "/app/crawler_pool.py", line 114, in get_crawler
await crawler.start()
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 183, in start
...
Exception: CDP endpoint at http://localhost:9222 is not ready after startup

[BROWSER] Browser process terminated during startup | Code: -11 | STDOUT: | STDERR: [45969:45984:ERROR:dbus/bus.cc:405] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

Launch args used (from verbose log):

/home/appuser/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome --remote-debugging-port=9222 --user-data-dir=/tmp/browser-profile-XXXX --headless=new --window-size=1080,600 --no-sandbox --disable-dev-shm-usage --no-first-run --no-default-browser-check --disable-infobars --ignore-certificate-errors --disable-blink-features=AutomationControlled --disable-gpu ...

Suspects: something in the worker's process state (event-loop signal handlers?

threaded fork+exec interaction?) trips Chromium's early startup — inherited

signal dispositions or seccomp/rlimit nuance not visible in /proc//limits.

Workaround: none found short of avoiding builtin mode.

Ngôn ngữ chính
Python
Star
84.5k
Fork
8.7k
Merge trung bình
3 ngày 9 giờ
Pull request đã merge (30 ngày)
17

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của unclecode/crawl4ai

Tất cả issue của unclecode/crawl4ai

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.