Bug]: Crawl4aiDockerClient gets 400 with any BrowserConfig: dump() always sends the generated `headers`

Abierto Apto para principiantes
  • ¿Algo no está bien?
#2,472 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
72/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
api, backend

Línea de trabajo

Empieza leyendo BrowserConfig.dump() y la inicialización descrita de BrowserConfig que genera sec-ch-ua; después, comprueba la reproducción de la carga no confiable en la issue. Verifica que la configuración serializada ya no envíe el campo headers generado y que el servidor acepte una solicitud con BrowserConfig().

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

⚙ Done 🐞 Bug 📌 Root caused
crawl4ai version

0.9.5 (also on develop; since 0.9.0)

Expected Behavior

Crawl4aiDockerClient.crawl(..., browser_config=BrowserConfig(headless=True)) is accepted by the server. The user did not set any headers.

Current Behavior

Every request with a BrowserConfig is rejected:

400 Rejected request: field 'headers' is not permitted on BrowserConfig from an untrusted request

BrowserConfig.__init__ always adds a sec-ch-ua header derived from the user agent:

self.browser_hint = UAGen.generate_client_hints(self.user_agent)
self.headers.setdefault("sec-ch-ua", self.browser_hint)

So headers is never empty, and dump() always writes it. The server forbids headers from an untrusted request (UNTRUSTED_FORBIDDEN_FIELDS["BrowserConfig"]) and rejects it whatever its value. The client sends browser_config.dump(), so it fails with any BrowserConfig, even BrowserConfig(). Without a BrowserConfig the request works.

The server rule is right. The problem is that dump() sends a header the user never set, and the server's __init__ derives the same value again from user_agent anyway.

#2262 hit the same 400 in the playground and worked around it there by forwarding only the typed params. The client was not fixed.

Is this reproducible?

Yes

Inputs Causing the Bug

Any BrowserConfig passed to Crawl4aiDockerClient, or any BrowserConfig(...).dump() posted to /crawl, /crawl/stream or /crawl/job.

Steps to Reproduce

Run the snippet below; it reproduces the server's check in-process. Against a running server, Crawl4aiDockerClient().crawl(["https://example.com"], browser_config=BrowserConfig()) returns the same 400.

Code snippets
from crawl4ai import BrowserConfig
from crawl4ai.async_configs import Provenance

data = BrowserConfig(headless=True).dump()
print(data["params"]["headers"])
# {'type': 'dict', 'value': {'sec-ch-ua': '"Chromium";v="116", "Not_A Brand";v="8", "Google Chrome";v="116"'}}

BrowserConfig.load(data, provenance=Provenance.UNTRUSTED)
# UntrustedConfigError: field 'headers' is not permitted on BrowserConfig from an untrusted request
OS

macOS 26.6 (platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

See above. I'll open a PR.

Lenguaje dominante
Python
Estrellas
86.8k
Forks
9k
Merge medio
2 d 3 h
PR fusionados (30 d)
26

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.