[Bug]: Docker requests can no longer specify deep crawl params
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 55/100
调研方向
Start by reproducing the 400 response from the Docker /crawl endpoint when deep_crawl_strategy is supplied, then trace the request validation and compare it with the limits in deploy/docker/governor.py. Check how filter_chain, url_scorer, max_pages, and max_depth are handled. Done means supported deep-crawl parameters work through Docker while configured safety limits remain enforced.
由索引模型根据 Issue 内容生成。
描述
crawl4ai version
0.9.4
Expected Behavior
When calling the Docker /crawl endpoint I should be able to specify:
- Deep or single-page crawl
filter_chainwith params e.g. URL pattern filtersurl_scorerwith params e.g. keywordsmax_pagesandmax_depth(clamped to a configurable limit to prevent DoS)
Current Behavior
Since 0.9.0 any requests with deep_crawl_strategy are rejected with 400.
This removes a ton of functionality that I'm currently relying on - for example:
- If I want to conditionally run single-page vs deep crawls, I would now need to run two docker containers in parallel, each statically configured to do each type.
- I currently dynamically generate URL path filters based on varying input URLs. This is now impossible because they cannot be defined statically.
According to the changelog, this was removed from the API to improve security. However, I believe this could be re-added with no increased security risk, beyond clamping a few values (e.g. max pages) to a server-configurable limit to prevent DoS - which seems to already be implemented in deploy/docker/governor.py.
An alternative would be to allow users to opt-out of such security limitations, for example in environments where they are already validating inputs before calling the API.
OS
Linux
Python version
3.12
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
No response
- 主要语言
- Python
- 星标
- 86.4k
- 派生
- 9k
- 平均合并
- 3 天 5 小时
- 30 天内合并 PR
- 23
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
unclecode/crawl4ai 的其他 Issue
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requires可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭⚙️ In-progress ✨ Enhancement
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 2 天内回复
-
Bug]: Crawl4aiDockerClient gets 400 with any BrowserConfig: dump() always sends the generated `headers`可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭⚙️ In-progress 🐞 Bug 📌 Root caused
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 2 天内回复
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.media可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 2 天内回复
-
[Bug]: Text after an inline <link>, <meta>, <style> or <noscript> is dropped from cleaned_html and markdown可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭🐞 Bug 🩺 Needs Triage
难度 2/5 1-3 小时 新手友好度 85/100
unclecode/crawl4ai#2416 · 1 条评论 ·
维护者通常 2 天内回复
-
🐞 Bug 🩺 Needs Triage
难度 2/5 1-3 小时 新手友好度 72/100
unclecode/crawl4ai#2382 · 2 条评论 ·
维护者通常 2 天内回复
查看 unclecode/crawl4ai 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 86/100
NousResearch/hermes-agent#147104 ·
维护者通常 1 天内回复
-
[oblt-aw][autodoc] Document new required `shared-org-key` input for `obs-aw-dependency-review`可能已有人在做 @github-actions 今天认领。 未关闭
难度 1/5 1 小时以内 新手友好度 90/100
维护者通常 1 天内回复
-
bot-triaged module: inductor oncall: pt2 topic: fuzzer triaged
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
getchaintips leaves out the active tip once a header is staged on it, where Core always lists it未关闭
难度 2/5 1-3 小时 新手友好度 88/100
btclib-org/btclib-node#2158 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
patrick-hudson/put.gd-desktop#979 · 1 条评论 ·
维护者通常 1 天内回复