Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug]: Reusing BFSDeepCrawlStrategy leaks the previous crawl's max_pages budget into a fresh run

Abierto Apto para principiantes
  • ¿Algo no está bien?
#2,309 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend

Línea de trabajo

Comienza con BFSDeepCrawlStrategy e inspecciona las ramas de inicialización sin reanudación de _arun_batch y _arun_stream. Verifica llamadas repetidas en modo batch, stream y mixto con el crawler determinista del harness de regresión enlazado, conservando el contador guardado para los crawls reanudados. Se considera terminado cuando el harness de 12 casos pasa y cada ejecución nueva recibe su propio presupuesto de max_pages.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Description

BFSDeepCrawlStrategy resets its cancellation event at the start of each batch/stream run, but the fresh-run branch does not reset _pages_crawled. Consequently, sequential independent crawls using the same strategy consume one shared page budget. A later run can return no results even though its start URL was never crawled.

This is distinct from resuming a saved crawl: _resume_state is absent in the failing case. A resumed run should continue to use its saved counter.

Minimal reproduction

This uses the public strategy with a deterministic crawler boundary; no browser or HTTP request is needed:

import asyncio
from types import SimpleNamespace
from crawl4ai import CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

class Crawler:
    async def arun_many(self, urls, config):
        return [SimpleNamespace(url=u, success=True, metadata={}, links={'internal': []}) for u in urls]

async def main():
    strategy = BFSDeepCrawlStrategy(max_depth=0, max_pages=1)
    config = CrawlerRunConfig(stream=False)
    for url in ['https://example.test/first', 'https://example.test/second']:
        results = await strategy.arun(start_url=url, crawler=Crawler(), config=config)
        print([r.url for r in results])

asyncio.run(main())

Expected: each call returns its respective start URL. Current behavior: the first call returns its URL; the second returns [] before invoking the crawler.

Candidate fix

Reset self._pages_crawled = 0 in the non-resume initialization branch of both _arun_batch and _arun_stream. Leave the resume branch's pages_crawled restoration unchanged.

Validation

Tested develop at 1f68e5bd7c29f2067a1ef74f28dbf4dc20686a06.

Before: 10 failed, 2 passed. After: 12 passed. Tests exercise the complete BFS/base strategy and filter/scorer modules with a deterministic async crawler and lightweight configuration/result/statistics objects. They cover repeated batch/stream/mixed-mode calls, cancellation followed by a fresh run, and preservation of a saved resume counter. All test crawls have max_depth=0, so URL normalization and browser/network behavior are outside this test scope. The full repository suite was not run.

Python 3.13.5, pytest 9.0.2, Linux. Issue searches for reuse max_pages and _pages_crawled reset, and PR search for the latter, did not find a specific existing report of sequential fresh-run counter leakage.

Patch and tests

Source-pinned patch and 12-case regression harness. From the bundle root, run python reproduce.py --fetch --case crawl4ai-fresh-run-budget --variant after. The runner checks the original source hash and patch application before testing.

Lenguaje dominante
Python
Estrellas
84.5k
Forks
8.7k
Merge medio
3 d 20 h
PR fusionados (30 d)
15

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.