[Bug]: AsyncUrlSeeder.urls() always idles ~5 s after the last sitemap URL (worker polls queue.get() with a 5 s timeout)
I maintainer di solito rispondono entro 1 giorno
@PiUnknown ci sta già lavorando.
Dal 7/11/2026.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 35/100
Direzione di ricerca
Start in crawl4ai/async_url_seeder.py around the worker loop at line 486 and trace how producer completion and the max_urls early-stop path interact with the queue. Compare the same timeout pattern in extract_head_for_urls(). The issue suggests sentinels or waiting on both events; done means workers wake promptly after production ends without breaking early-stop behavior, with coverage for both paths.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
crawl4ai version
0.9.4 (the same code is on main today)
Expected behavior
AsyncUrlSeeder.urls() returns as soon as the producer has yielded the last URL and the workers have processed the queue.
Current behavior
Every call takes about 5 seconds longer than reading the sitemap. In urls() each worker loops:
if queue.empty() and producer_done.is_set():
break
try:
url = await asyncio.wait_for(queue.get(), 5)
except asyncio.TimeoutError:
continue
(crawl4ai/async_url_seeder.py, line 486 on main.) A worker that finds the queue empty while the producer is still running parks in queue.get() with a 5 s timeout. When the producer then finishes without putting another item, nothing wakes that worker, so it sits out the full 5 s before it re-checks producer_done. With extract_head=False and live_check=False the workers do almost no work, so this wait is nearly the whole cost of the call.
In our crawler (sitemap source only, extract_head=False, concurrency=5) the wait was about 13 % of total crawl time per site across 23 real sites, and the first page started after 9.5 s instead of 6.1 s (median) once the wait was gone.
Steps to reproduce
import asyncio, time
from crawl4ai import AsyncUrlSeeder, SeedingConfig
async def main():
async with AsyncUrlSeeder() as seeder:
started = time.perf_counter()
urls = await seeder.urls("https://www.python.org", SeedingConfig(source="sitemap", extract_head=False, force=True))
print(len(urls), "URLs in", round(time.perf_counter() - started, 1), "s")
asyncio.run(main())
Timing the producer separately (iterating seeder._from_sitemaps(domain, "*", force=True)) shows the sitemap itself is read about 5 s faster than urls() returns.
Possible fix
Wake the workers when the producer finishes, for example with one sentinel per worker:
# producer, in its finally block
producer_done.set()
for _ in range(concurrency):
await queue.put(None)
# worker
url = await queue.get()
if url is None:
queue.task_done()
break
(The max_urls early-stop path that flushes the queue would need to keep the sentinels, or use asyncio.wait on queue.get() and producer_done.wait() instead of the polling timeout.) extract_head_for_urls() has the same pattern with a 1 s timeout.
OS
Linux (Debian, Docker), macOS
Python version
3.15
- Lingua principale
- Python
- Stelle
- 87.2k
- Fork
- 9.1k
- Merge medio
- 1g 6h
- PR unite (30g)
- 25
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackForse già presa @HhH666-ux l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
I maintainer di solito rispondono entro 1 giorno
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configForse già presa Una pull request collegata a questa issue è aperta o già unita. Aperta🐞 Bug 📌 Root caused
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
unclecode/crawl4ai#2494 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAperta⚙ Done ✨ Enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
I maintainer di solito rispondono entro 1 giorno
-
⚙ Done 🐞 Bug 📌 Root caused
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAperta⚙ Done 🐞 Bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
unclecode/crawl4ai#2469 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di unclecode/crawl4ai
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 73/100
Netcracker/qubership-ai-packages#266 ·
I maintainer di solito rispondono entro 1 giorno
-
PROCESSED_DATA_FOLDER: strip whitespace and treat empty as unsetForse già presa @TorecLuik l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
NL-BioImaging/BIOMERO.importer#63 ·
I maintainer di solito rispondono entro 1 giorno
-
MLX training-complete chat/merge suggestions omit --backend mlxForse già presa @vaputa l’ha presa oggi. Apertaneeds-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
MakazhanAlpamys/Soup#2913 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
area:security effort:S priority:P3 status:needs-triage type:chore
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
randomparity/kdive#3657 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
aws/mcp-proxy-for-aws#476 ·
I maintainer di solito rispondono entro 1 giorno