Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Bug]: Docker server egress proxy resolves DNS synchronously on the event loop, stalling the whole server under load

オープン
  • 何か問題がありますか？
#2,423 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

@greerso がすでに取り組んでいます。

2026年10月21日 から。

  • #2428 @greerso による — オープン

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
72/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
docker, python
領域
backend, performance

調査の方向性

Read deploy/docker/egress_proxy.py and egress_broker.py, then api.py; trace the DNS resolution calls from the proxy handlers and seed validation. The issue describes blocking socket.getaddrinfo on the event loop and provides a production fix to compare against. Reproduce with a slow fake getaddrinfo and verify that the event loop stays responsive while four CONNECT requests resolve.

索引モデルが issue の本文から書いたものです。

説明

⚙️ In-progress 📌 Root caused
crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5). deploy/docker/egress_proxy.py and egress_broker.py are unchanged on develop.

Expected Behavior

The Docker server keeps answering requests while crawls run.

Current Behavior

Under concurrent crawls of ordinary, ad-heavy pages, the whole server stalls for seconds at a time. Even a request rejected at the auth gate (POST /crawl with body {} and no token, which should answer 401 at once) takes 10 s at the median and up to 39 s. The gunicorn worker is mostly idle (about 22% CPU) during those stalls.

Cause: since 0.9.5 every Chromium connection goes through the in-process egress proxy (PinningProxy). For each CONNECT, _handle_connect (and _handle_absolute for plain HTTP) calls egress_broker.resolve_and_pin(). That calls socket.getaddrinfo, which blocks, directly on the asyncio event loop. A page with many third-party hosts (ads, trackers, CDNs) opens hundreds of connections, so the server's single event loop spends most of its time blocked in DNS lookups, and every other request waits behind them.

py-spy record --idle on the gunicorn worker, under 16 concurrent /crawl requests: 3,398 of 3,550 samples (96%) on the event-loop thread were in this stack:

getaddrinfo (socket.py)
 <- _resolve (egress_broker.py:97)
 <- resolve_and_pin (egress_broker.py:140)
 <- _handle_connect (egress_proxy.py:181)
 <- _handle (egress_proxy.py:165)
 <- asyncio base_events._run_once

Most of the rest (233 samples) was the same lookup via _normalize_and_validate_seeds in handle_crawl_request (api.py), which validates each seed URL on the loop.

Is this reproducible?

Yes

Inputs Causing the Bug

Any steady concurrent /crawl traffic to real-world pages with many third-party hosts (news sites, retail product pages).

Steps to Reproduce
  1. Run unclecode/crawl4ai:0.9.5 with an API token set.
  2. Send 16 concurrent single-URL /crawl requests in a loop, to a list of real-world pages (we used 120 news and retail pages).
  3. Every second, send POST /crawl with body {} and no token, and time the 401.
Code snippets
# deploy/docker/egress_proxy.py, PinningProxy._handle_connect (0.9.5)
try:
    pin = resolve_and_pin(f"https://{host}:{port_s}")   # blocking getaddrinfo on the event loop
except EgressBlocked:
    ...

Fix we run in production: resolve in a worker thread. The egress rule is unchanged (resolve once, reject any non-global answer, pin the IP).

pin = await asyncio.to_thread(resolve_and_pin, f"https://{host}:{port_s}")   # _handle_connect
pin = await asyncio.to_thread(resolve_and_pin, f"http://{sp.hostname}:{port}")  # _handle_absolute
urls, refused = await asyncio.to_thread(_normalize_and_validate_seeds, urls)  # api.py, both handlers

Same container image, same load (60 s warm-up, then 4 min at 16 concurrent crawls):

Probe p50 Probe max Probes over 8 s Crawls finished Successful Median crawl
stock 0.9.5 10.9 s 39.1 s 11 of 19 29 0 90.1 s
with the fix 0.02 s 1.14 s 0 of 215 178 134 15.6 s

Stock 0.9.5 in that table already carries our fixes for #2419 and for dead pooled browsers. Neither touches the egress proxy. We can open a PR against develop with this change and a test (a slow fake getaddrinfo; the event loop must stay responsive while 4 CONNECTs resolve).

OS

Linux (Docker, host kernel 7.0)

Python version

3.12 (image default)

Browser

Chromium headless shell (image default)

Browser version

Playwright 1.64.0, chromium_headless_shell-1248

Error logs & Screenshots (if applicable)

No error is logged. The only visible symptom is latency. Under load, the monitor also logs Failed to persist endpoint stats: Timeout reading from crawl4ai-redis:6379, because the Redis call waits behind the blocked loop.

主要言語
Python
スター
85.9k
フォーク
8.9k
平均マージ
5日 8時間
マージ済み PR（30日）
24

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

unclecode/crawl4ai のほかの issue

unclecode/crawl4ai の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。