Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug]: `crawler_configs` is silently ignored for a single URL and on `/crawl/stream`

已关闭
  • 哪里不对？
#2,287 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
76/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
api, backend, testing-qa

调研方向

Start with handle_crawl_request in deploy/docker/api.py and compare its single-URL and multi-URL paths, then trace stream_process in deploy/docker/server.py into handle_stream_crawl_request. Run tests/test_issue_1837_config_list.py::test_single_url_ignores_crawler_configs and reproduce the documented /crawl and /crawl/stream cases. Done means crawler_configs are honored or unsupported cases return 400 instead of silently returning 200.

由索引模型根据 Issue 内容生成。

描述

⚙ Done 🐞 Bug 📌 Root caused 🥸 - Medium
crawl4ai version

0.9.4

Expected Behavior

crawler_configs applies to every request that carries it: a /crawl with one URL, and /crawl/stream (or /crawl with stream: true). If some case can't support it, a 400 rather than a 200 that ran with a different config.

Current Behavior

Two gaps in the config-list support from #1852:

  1. handle_crawl_request uses the list only when len(urls) > 1 (api.py#L726); with one URL it calls arun() with crawler_config (#L749).
  2. stream_process doesn't pass crawler_configs down, and handle_stream_crawl_request has no parameter for it (server.py#L1030-L1036, api.py#L889-L895).

Both answer 200, so the caller can't tell its per-URL settings were dropped. The single-URL case is pinned by tests/test_issue_1837_config_list.py::test_single_url_ignores_crawler_configs ("arun only takes one config"), but arun_many handles a one-URL list fine.

cc @hafezparast @ntohidi

Is this reproducible?

Yes

Inputs Causing the Bug
- /crawl, one URL, crawler_configs: [{url_matcher: "*", css_selector: ".only-a"}]
- /crawl/stream, two URLs, the same list
Steps to Reproduce
1. Page with two sections, .only-a and .only-b
2. POST /crawl with that one URL and the crawler_configs above
3. The markdown still contains the .only-b section
4. Same list on /crawl/stream: same result
Code snippets
import httpx

# URL: a page with
#   <div class="only-a"><p>SECTION-A is the part a per-URL css_selector keeps.</p></div>
#   <div class="only-b"><p>SECTION-B is the part it drops.</p></div>
per_url = [{"type": "CrawlerRunConfig",
            "params": {"url_matcher": "*", "css_selector": ".only-a", "cache_mode": "bypass"}}]
r = httpx.post("http://localhost:11235/crawl", headers={"Authorization": f"Bearer {TOKEN}"},
               json={"urls": [URL], "crawler_configs": per_url}, timeout=60)
md = r.json()["results"][0]["markdown"]["raw_markdown"]
print("SECTION-B" in md)   # True: the per-URL css_selector was not applied
OS

Linux (Docker image built from develop @ 1f68e5b)

Python version

3.12 (the image's)

Browser

Chromium (Playwright, bundled)

Browser version

No response

Error logs & Screenshots (if applicable)

Local test page with .only-a / .only-b sections (CRAWL4AI_ALLOW_INTERNAL_URLS=true); A / B = that section is in the markdown:
[configs] /crawl, one URL, per-URL css_selector=.only-a -> HTTP 200: A=True B=True
[configs] /crawl/stream, two URLs, same list -> HTTP 200: A=True B=True; A=True B=True

主要语言
Python
星标
85.9k
派生
8.9k
平均合并
5 天 6 小时
30 天内合并 PR
24

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。