Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug]: Docker server egress proxy resolves DNS synchronously on the event loop, stalling the whole server under load

未关闭
  • 哪里不对？
#2,423 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

@greerso 已经在做这个了。

开始于 2026年10月21日。

  • #2428 来自 @greerso —— 未关闭

评估

难度
3/5
预计耗时
1-2 天
新手友好度
72/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
docker, python
领域
backend, performance

调研方向

Read deploy/docker/egress_proxy.py and egress_broker.py, then api.py; trace the DNS resolution calls from the proxy handlers and seed validation. The issue describes blocking socket.getaddrinfo on the event loop and provides a production fix to compare against. Reproduce with a slow fake getaddrinfo and verify that the event loop stays responsive while four CONNECT requests resolve.

由索引模型根据 Issue 内容生成。

描述

⚙️ In-progress 📌 Root caused
crawl4ai version

0.9.5 (Docker image unclecode/crawl4ai:0.9.5). deploy/docker/egress_proxy.py and egress_broker.py are unchanged on develop.

Expected Behavior

The Docker server keeps answering requests while crawls run.

Current Behavior

Under concurrent crawls of ordinary, ad-heavy pages, the whole server stalls for seconds at a time. Even a request rejected at the auth gate (POST /crawl with body {} and no token, which should answer 401 at once) takes 10 s at the median and up to 39 s. The gunicorn worker is mostly idle (about 22% CPU) during those stalls.

Cause: since 0.9.5 every Chromium connection goes through the in-process egress proxy (PinningProxy). For each CONNECT, _handle_connect (and _handle_absolute for plain HTTP) calls egress_broker.resolve_and_pin(). That calls socket.getaddrinfo, which blocks, directly on the asyncio event loop. A page with many third-party hosts (ads, trackers, CDNs) opens hundreds of connections, so the server's single event loop spends most of its time blocked in DNS lookups, and every other request waits behind them.

py-spy record --idle on the gunicorn worker, under 16 concurrent /crawl requests: 3,398 of 3,550 samples (96%) on the event-loop thread were in this stack:

getaddrinfo (socket.py)
 <- _resolve (egress_broker.py:97)
 <- resolve_and_pin (egress_broker.py:140)
 <- _handle_connect (egress_proxy.py:181)
 <- _handle (egress_proxy.py:165)
 <- asyncio base_events._run_once

Most of the rest (233 samples) was the same lookup via _normalize_and_validate_seeds in handle_crawl_request (api.py), which validates each seed URL on the loop.

Is this reproducible?

Yes

Inputs Causing the Bug

Any steady concurrent /crawl traffic to real-world pages with many third-party hosts (news sites, retail product pages).

Steps to Reproduce
  1. Run unclecode/crawl4ai:0.9.5 with an API token set.
  2. Send 16 concurrent single-URL /crawl requests in a loop, to a list of real-world pages (we used 120 news and retail pages).
  3. Every second, send POST /crawl with body {} and no token, and time the 401.
Code snippets
# deploy/docker/egress_proxy.py, PinningProxy._handle_connect (0.9.5)
try:
    pin = resolve_and_pin(f"https://{host}:{port_s}")   # blocking getaddrinfo on the event loop
except EgressBlocked:
    ...

Fix we run in production: resolve in a worker thread. The egress rule is unchanged (resolve once, reject any non-global answer, pin the IP).

pin = await asyncio.to_thread(resolve_and_pin, f"https://{host}:{port_s}")   # _handle_connect
pin = await asyncio.to_thread(resolve_and_pin, f"http://{sp.hostname}:{port}")  # _handle_absolute
urls, refused = await asyncio.to_thread(_normalize_and_validate_seeds, urls)  # api.py, both handlers

Same container image, same load (60 s warm-up, then 4 min at 16 concurrent crawls):

Probe p50 Probe max Probes over 8 s Crawls finished Successful Median crawl
stock 0.9.5 10.9 s 39.1 s 11 of 19 29 0 90.1 s
with the fix 0.02 s 1.14 s 0 of 215 178 134 15.6 s

Stock 0.9.5 in that table already carries our fixes for #2419 and for dead pooled browsers. Neither touches the egress proxy. We can open a PR against develop with this change and a test (a slow fake getaddrinfo; the event loop must stay responsive while 4 CONNECTs resolve).

OS

Linux (Docker, host kernel 7.0)

Python version

3.12 (image default)

Browser

Chromium headless shell (image default)

Browser version

Playwright 1.64.0, chromium_headless_shell-1248

Error logs & Screenshots (if applicable)

No error is logged. The only visible symptom is latency. Under load, the monitor also logs Failed to persist endpoint stats: Timeout reading from crawl4ai-redis:6379, because the Redis call waits behind the blocked loop.

主要语言
Python
星标
85.9k
派生
8.9k
平均合并
5 天 8 小时
30 天内合并 PR
24

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。