[Bug]: AsyncUrlSeeder.urls() always idles ~5 s after the last sitemap URL (worker polls queue.get() with a 5 s timeout)

Aperta
#2,509 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@PiUnknown ci sta già lavorando.

Dal 7/11/2026.

  • #2511 di @anubhavlal07 — aperta
  • #2512 di @PiUnknown — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start in crawl4ai/async_url_seeder.py around the worker loop at line 486 and trace how producer completion and the max_urls early-stop path interact with the queue. Compare the same timeout pattern in extract_head_for_urls(). The issue suggests sentinels or waiting on both events; done means workers wake promptly after production ends without breaking early-stop behavior, with coverage for both paths.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.4 (the same code is on main today)

Expected behavior

AsyncUrlSeeder.urls() returns as soon as the producer has yielded the last URL and the workers have processed the queue.

Current behavior

Every call takes about 5 seconds longer than reading the sitemap. In urls() each worker loops:

if queue.empty() and producer_done.is_set():
    break
try:
    url = await asyncio.wait_for(queue.get(), 5)
except asyncio.TimeoutError:
    continue

(crawl4ai/async_url_seeder.py, line 486 on main.) A worker that finds the queue empty while the producer is still running parks in queue.get() with a 5 s timeout. When the producer then finishes without putting another item, nothing wakes that worker, so it sits out the full 5 s before it re-checks producer_done. With extract_head=False and live_check=False the workers do almost no work, so this wait is nearly the whole cost of the call.

In our crawler (sitemap source only, extract_head=False, concurrency=5) the wait was about 13 % of total crawl time per site across 23 real sites, and the first page started after 9.5 s instead of 6.1 s (median) once the wait was gone.

Steps to reproduce
import asyncio, time
from crawl4ai import AsyncUrlSeeder, SeedingConfig

async def main():
    async with AsyncUrlSeeder() as seeder:
        started = time.perf_counter()
        urls = await seeder.urls("https://www.python.org", SeedingConfig(source="sitemap", extract_head=False, force=True))
        print(len(urls), "URLs in", round(time.perf_counter() - started, 1), "s")

asyncio.run(main())

Timing the producer separately (iterating seeder._from_sitemaps(domain, "*", force=True)) shows the sitemap itself is read about 5 s faster than urls() returns.

Possible fix

Wake the workers when the producer finishes, for example with one sentinel per worker:

# producer, in its finally block
producer_done.set()
for _ in range(concurrency):
    await queue.put(None)

# worker
url = await queue.get()
if url is None:
    queue.task_done()
    break

(The max_urls early-stop path that flushes the queue would need to keep the sentinels, or use asyncio.wait on queue.get() and producer_done.wait() instead of the polling timeout.) extract_head_for_urls() has the same pattern with a 1 s timeout.

OS

Linux (Debian, Docker), macOS

Python version

3.15

Lingua principale
Python
Stelle
87.2k
Fork
9.1k
Merge medio
1g 6h
PR unite (30g)
25

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.