Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: After a streamed deep crawl is stopped early, the next deep crawl returns only the start page

Aperta
  • Qualcosa non va?
#2,380 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@PiUnknown ci sta già lavorando.

Dal 15/10/2026.

  • #2386 di @PiUnknown — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
12/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start at DeepCrawlDecorator, where the deep_crawl_active ContextVar is set to True, and at the finally in its stream wrapper that resets it. Run the reproduction script in the issue: a full crawl, a crawl stopped after two pages, then a third crawl that should return six pages but returns one. Done means the third crawl is no longer treated as nested. Pull request #2386 already targets this, so check it before starting.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

crawl4ai version

0.9.5 (same on 0.9.0)

Expected Behavior

Every crawler.arun(url, config) with a deep_crawl_strategy and stream=True runs a deep crawl, including after a previous streamed deep crawl was stopped early.

Current Behavior

If the caller stops iterating a streamed deep crawl (break), the next deep crawl from the same task returns only the start page.

DeepCrawlDecorator sets the deep_crawl_active ContextVar to True and resets it in the finally of the stream wrapper. When the caller leaves the stream without closing it, that finally runs later, when the generator is garbage-collected, in another context. The flag stays True in the caller's context, so the next arun() is treated as a nested call and falls through to a single-page crawl. The same happens if the stream is consumed from another task (for example asyncio.create_task(anext(stream))): the reset only reaches that task's copy of the context.

Is this reproducible?

Yes

Code snippets
import asyncio

from aiohttp import web
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

PAGES = 5


async def handle(request):
    links = "".join(f'<a href="/p{i}">p{i}</a>' for i in range(PAGES)) if request.path == "/" else ""
    return web.Response(text=f"<html><body>{links}<p>{'text ' * 200}</p></body></html>", content_type="text/html")


async def deep_crawl(crawler, url, stop_after=None):
    config = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1), stream=True)
    pages = []
    async for result in await crawler.arun(url, config=config):
        pages.append(result.url)
        if stop_after and len(pages) == stop_after:
            break  # e.g. the caller restarts the crawl with another browser config
    return pages


async def main():
    app = web.Application()
    app.router.add_get("/{tail:.*}", handle)
    runner = web.AppRunner(app)
    await runner.setup()
    await web.TCPSite(runner, "127.0.0.1", 8765).start()
    url = "http://127.0.0.1:8765/"
    async with AsyncWebCrawler() as crawler:
        print("full crawl:", len(await deep_crawl(crawler, url)))  # 6
        print("stopped early:", len(await deep_crawl(crawler, url, stop_after=2)))  # 2
        print("next crawl:", len(await deep_crawl(crawler, url)))  # 1, expected 6
    await runner.cleanup()


asyncio.run(main())
Suggested fix

Reset the flag as soon as deep_crawl_strategy.arun() has returned the stream, with the token from set(), instead of in the stream's finally. The strategies' own batches already run with deep_crawl_strategy=None, so the guard is not needed while the stream is consumed. Callers can work around it by closing the stream in their own task (async with contextlib.aclosing(stream)).

Lingua principale
Python
Stelle
85.4k
Fork
8.9k
Merge medio
5g 7h
PR unite (30g)
23

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di unclecode/crawl4ai

Tutte le issue di unclecode/crawl4ai

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.