Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Bug]: After a streamed deep crawl is stopped early, the next deep crawl returns only the start page

オープン
  • 何か問題がありますか？
#2,380 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

@PiUnknown がすでに取り組んでいます。

2026年10月15日 から。

  • #2386 @PiUnknown による — オープン

評価

難易度
4/5
見積もり時間
1〜2日
初心者へのやさしさ
12/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
backend

調査の方向性

Start at DeepCrawlDecorator, where the deep_crawl_active ContextVar is set to True, and at the finally in its stream wrapper that resets it. Run the reproduction script in the issue: a full crawl, a crawl stopped after two pages, then a third crawl that should return six pages but returns one. Done means the third crawl is no longer treated as nested. Pull request #2386 already targets this, so check it before starting.

索引モデルが issue の本文から書いたものです。

説明

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.5 (same on 0.9.0)

Expected Behavior

Every crawler.arun(url, config) with a deep_crawl_strategy and stream=True runs a deep crawl, including after a previous streamed deep crawl was stopped early.

Current Behavior

If the caller stops iterating a streamed deep crawl (break), the next deep crawl from the same task returns only the start page.

DeepCrawlDecorator sets the deep_crawl_active ContextVar to True and resets it in the finally of the stream wrapper. When the caller leaves the stream without closing it, that finally runs later, when the generator is garbage-collected, in another context. The flag stays True in the caller's context, so the next arun() is treated as a nested call and falls through to a single-page crawl. The same happens if the stream is consumed from another task (for example asyncio.create_task(anext(stream))): the reset only reaches that task's copy of the context.

Is this reproducible?

Yes

Code snippets
import asyncio

from aiohttp import web
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

PAGES = 5


async def handle(request):
    links = "".join(f'<a href="/p{i}">p{i}</a>' for i in range(PAGES)) if request.path == "/" else ""
    return web.Response(text=f"<html><body>{links}<p>{'text ' * 200}</p></body></html>", content_type="text/html")


async def deep_crawl(crawler, url, stop_after=None):
    config = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1), stream=True)
    pages = []
    async for result in await crawler.arun(url, config=config):
        pages.append(result.url)
        if stop_after and len(pages) == stop_after:
            break  # e.g. the caller restarts the crawl with another browser config
    return pages


async def main():
    app = web.Application()
    app.router.add_get("/{tail:.*}", handle)
    runner = web.AppRunner(app)
    await runner.setup()
    await web.TCPSite(runner, "127.0.0.1", 8765).start()
    url = "http://127.0.0.1:8765/"
    async with AsyncWebCrawler() as crawler:
        print("full crawl:", len(await deep_crawl(crawler, url)))  # 6
        print("stopped early:", len(await deep_crawl(crawler, url, stop_after=2)))  # 2
        print("next crawl:", len(await deep_crawl(crawler, url)))  # 1, expected 6
    await runner.cleanup()


asyncio.run(main())
Suggested fix

Reset the flag as soon as deep_crawl_strategy.arun() has returned the stream, with the token from set(), instead of in the stream's finally. The strategies' own batches already run with deep_crawl_strategy=None, so the guard is not needed while the stream is consumed. Callers can work around it by closing the stream in their own task (async with contextlib.aclosing(stream)).

主要言語
Python
スター
85.9k
フォーク
8.9k
平均マージ
5日 6時間
マージ済み PR（30日）
24

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

unclecode/crawl4ai のほかの issue

unclecode/crawl4ai の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。