Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug]: After a streamed deep crawl is stopped early, the next deep crawl returns only the start page

Abierto
  • ¿Algo no está bien?
#2,380 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@PiUnknown ya está trabajando en esto.

Desde el 15/10/2026.

  • #2386 de @PiUnknown — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
1-2 días
Aptitud para principiantes
12/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend

Línea de trabajo

Start at DeepCrawlDecorator, where the deep_crawl_active ContextVar is set to True, and at the finally in its stream wrapper that resets it. Run the reproduction script in the issue: a full crawl, a crawl stopped after two pages, then a third crawl that should return six pages but returns one. Done means the third crawl is no longer treated as nested. Pull request #2386 already targets this, so check it before starting.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

crawl4ai version

0.9.5 (same on 0.9.0)

Expected Behavior

Every crawler.arun(url, config) with a deep_crawl_strategy and stream=True runs a deep crawl, including after a previous streamed deep crawl was stopped early.

Current Behavior

If the caller stops iterating a streamed deep crawl (break), the next deep crawl from the same task returns only the start page.

DeepCrawlDecorator sets the deep_crawl_active ContextVar to True and resets it in the finally of the stream wrapper. When the caller leaves the stream without closing it, that finally runs later, when the generator is garbage-collected, in another context. The flag stays True in the caller's context, so the next arun() is treated as a nested call and falls through to a single-page crawl. The same happens if the stream is consumed from another task (for example asyncio.create_task(anext(stream))): the reset only reaches that task's copy of the context.

Is this reproducible?

Yes

Code snippets
import asyncio

from aiohttp import web
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

PAGES = 5


async def handle(request):
    links = "".join(f'<a href="/p{i}">p{i}</a>' for i in range(PAGES)) if request.path == "/" else ""
    return web.Response(text=f"<html><body>{links}<p>{'text ' * 200}</p></body></html>", content_type="text/html")


async def deep_crawl(crawler, url, stop_after=None):
    config = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1), stream=True)
    pages = []
    async for result in await crawler.arun(url, config=config):
        pages.append(result.url)
        if stop_after and len(pages) == stop_after:
            break  # e.g. the caller restarts the crawl with another browser config
    return pages


async def main():
    app = web.Application()
    app.router.add_get("/{tail:.*}", handle)
    runner = web.AppRunner(app)
    await runner.setup()
    await web.TCPSite(runner, "127.0.0.1", 8765).start()
    url = "http://127.0.0.1:8765/"
    async with AsyncWebCrawler() as crawler:
        print("full crawl:", len(await deep_crawl(crawler, url)))  # 6
        print("stopped early:", len(await deep_crawl(crawler, url, stop_after=2)))  # 2
        print("next crawl:", len(await deep_crawl(crawler, url)))  # 1, expected 6
    await runner.cleanup()


asyncio.run(main())
Suggested fix

Reset the flag as soon as deep_crawl_strategy.arun() has returned the stream, with the token from set(), instead of in the stream's finally. The strategies' own batches already run with deep_crawl_strategy=None, so the guard is not needed while the stream is consumed. Callers can work around it by closing the stream in their own task (async with contextlib.aclosing(stream)).

Lenguaje dominante
Python
Estrellas
85.4k
Forks
8.9k
Merge medio
5 d 7 h
PR fusionados (30 d)
23

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.