Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug]: Failed CrawlResult has `links: {}` / `media: {}` instead of the usual keys

Abierto
  • ¿Algo no está bien?
#2,286 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
api, backend

Línea de trabajo

Start with CrawlResult defaults in crawl4ai/models.py and the result construction branches in crawl4ai/async_webcrawler.py, especially robots.txt refusal, proxy failure, exception, and anti-bot fallback paths. Reproduce the two-URL /crawl case with check_robots_txt=true, then verify that failed and fallback results expose the same internal/external link and image/video/audio media keys as successful results.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

🐞 Bug 🩺 Needs Triage
crawl4ai version

0.9.4 (also on 0.9.2)

Expected Behavior

A result that failed (robots.txt refusal, fetch error, "All proxies failed") has the same links and media keys as a successful one:

"links": {"internal": [], "external": []},
"media": {"images": [], "videos": [], "audios": []}

so a client can read links["internal"] without special-casing failures.

Current Behavior

CrawlResult.links and CrawlResult.media default to {} (models.py#L136-L137). Successful crawls fill them from the scraper's typed Links / Media models, but the results AsyncWebCrawler.arun builds itself never set them:

So in a mixed batch /crawl returns "links": {}, "media": {} for exactly the entries a client has to handle differently, and in the SDK result.links["internal"] raises KeyError. Since #2134 keeps failed results in /crawl responses, every batch with a refused or failed URL has this shape. In one client that read links.internal.length, the resulting TypeError was taken for a failed request and the healthy pages of the batch were re-crawled several times over.

cc @nightcityblade (#2134) @ntohidi

Is this reproducible?

Yes

Inputs Causing the Bug
- Two URLs on one site, one of them disallowed by its robots.txt
- crawler_config params: {"check_robots_txt": true}
Steps to Reproduce
1. Run the Docker image (0.9.4, or built from develop)
2. POST /crawl with both URLs and check_robots_txt=true
3. Compare `links` / `media` of the two results
Code snippets
from crawl4ai.models import CrawlResult

# What AsyncWebCrawler.arun returns for a URL robots.txt disallows
r = CrawlResult(url="https://example.com/private", html="", success=False,
                status_code=403, error_message="Access denied by robots.txt")
print(r.links, r.media)   # {} {}
r.links["internal"]       # KeyError: 'internal'
OS

Linux (Docker image built from develop @ 1f68e5b)

Python version

3.12 (the image's)

Browser

Chromium (Playwright, bundled)

Browser version

No response

Error logs & Screenshots (if applicable)

POST /crawl, check_robots_txt=true, two URLs of a local test site (CRAWL4AI_ALLOW_INTERNAL_URLS=true), /private disallowed by its robots.txt:
[shape] HTTP 200
/index.html success=True status=200 links=['external', 'internal'] media=['audios', 'images', 'videos'] error=''
/private/index.html success=False status=403 links={} media={} error='Access denied by robots.txt'

Lenguaje dominante
Python
Estrellas
84.5k
Forks
8.7k
Merge medio
3 d 9 h
PR fusionados (30 d)
17

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.