Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: `crawler_configs` is silently ignored for a single URL and on `/crawl/stream`

Aperta
  • Qualcosa non va?
#2,287 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@talelboussetta ci sta già lavorando.

Dal 25/9/2026.

  • #2290 di @talelboussetta — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
76/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
api, backend, testing-qa

Direzione di ricerca

Start with handle_crawl_request in deploy/docker/api.py and compare its single-URL and multi-URL paths, then trace stream_process in deploy/docker/server.py into handle_stream_crawl_request. Run tests/test_issue_1837_config_list.py::test_single_url_ignores_crawler_configs and reproduce the documented /crawl and /crawl/stream cases. Done means crawler_configs are honored or unsupported cases return 400 instead of silently returning 200.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.4

Expected Behavior

crawler_configs applies to every request that carries it: a /crawl with one URL, and /crawl/stream (or /crawl with stream: true). If some case can't support it, a 400 rather than a 200 that ran with a different config.

Current Behavior

Two gaps in the config-list support from #1852:

  1. handle_crawl_request uses the list only when len(urls) > 1 (api.py#L726); with one URL it calls arun() with crawler_config (#L749).
  2. stream_process doesn't pass crawler_configs down, and handle_stream_crawl_request has no parameter for it (server.py#L1030-L1036, api.py#L889-L895).

Both answer 200, so the caller can't tell its per-URL settings were dropped. The single-URL case is pinned by tests/test_issue_1837_config_list.py::test_single_url_ignores_crawler_configs ("arun only takes one config"), but arun_many handles a one-URL list fine.

cc @hafezparast @ntohidi

Is this reproducible?

Yes

Inputs Causing the Bug
- /crawl, one URL, crawler_configs: [{url_matcher: "*", css_selector: ".only-a"}]
- /crawl/stream, two URLs, the same list
Steps to Reproduce
1. Page with two sections, .only-a and .only-b
2. POST /crawl with that one URL and the crawler_configs above
3. The markdown still contains the .only-b section
4. Same list on /crawl/stream: same result
Code snippets
import httpx

# URL: a page with
#   <div class="only-a"><p>SECTION-A is the part a per-URL css_selector keeps.</p></div>
#   <div class="only-b"><p>SECTION-B is the part it drops.</p></div>
per_url = [{"type": "CrawlerRunConfig",
            "params": {"url_matcher": "*", "css_selector": ".only-a", "cache_mode": "bypass"}}]
r = httpx.post("http://localhost:11235/crawl", headers={"Authorization": f"Bearer {TOKEN}"},
               json={"urls": [URL], "crawler_configs": per_url}, timeout=60)
md = r.json()["results"][0]["markdown"]["raw_markdown"]
print("SECTION-B" in md)   # True: the per-URL css_selector was not applied
OS

Linux (Docker image built from develop @ 1f68e5b)

Python version

3.12 (the image's)

Browser

Chromium (Playwright, bundled)

Browser version

No response

Error logs & Screenshots (if applicable)

Local test page with .only-a / .only-b sections (CRAWL4AI_ALLOW_INTERNAL_URLS=true); A / B = that section is in the markdown:
[configs] /crawl, one URL, per-URL css_selector=.only-a -> HTTP 200: A=True B=True
[configs] /crawl/stream, two URLs, same list -> HTTP 200: A=True B=True; A=True B=True

Lingua principale
Python
Stelle
84.8k
Fork
8.8k
Merge medio
14h 10m
PR unite (30g)
12

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.