[Bug]: URL seeder drops .gz sitemaps that are served with Content-Encoding: gzip

Abierto
#2,541 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@adityajai119 ya está trabajando en esto.

Desde el 12/11/2026.

  • #2542 de @adityajai119 — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
50/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend

Línea de trabajo

The gzip check is in _iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py, where the body is decompressed only when the URL ends in .gz. Start there, then run the MockTransport script from the issue to see the empty result. Done means that script prints both example.com/posts URLs, and tests cover a gzipped body without a .gz URL and a .gz URL served with Content-Encoding gzip. The reporter says they are opening a PR, so check for it before starting.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

crawl4ai version

0.9.5 (also reproduced on develop at 1a6eddb)

Expected Behavior

AsyncUrlSeeder returns the URLs of a .xml.gz sitemap whether the server sends the gzip file as it is, or with Content-Encoding: gzip.

Current Behavior

_iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py gunzip the body whenever the sitemap URL ends with .gz:

data = gzip.decompress(r.content) if url.endswith(".gz") else r.content

When the server sends the file with Content-Encoding: gzip (nginx gzip_static, Apache AddEncoding gzip .gz, S3/CloudFront objects with that metadata), httpx removes that encoding before the seeder sees the body. The seeder then gets plain XML, gzip.decompress() raises BadGzipFile, and every URL of that sitemap is dropped.

The other way round, gzipped content at a URL that doesn't end in .gz (for example sitemap-posts.xml.gz?v=2) is never decompressed, and no <loc> is found.

Is this reproducible?

Yes

Inputs Causing the Bug

A sitemap index that lists https://example.com/sitemap-posts.xml.gz, served with Content-Type: application/xml and Content-Encoding: gzip.

Steps to Reproduce

Run the script below. It uses httpx's MockTransport, so no network is needed.

Code snippets
import asyncio
import gzip
import tempfile

import httpx
from crawl4ai import AsyncUrlSeeder, SeedingConfig

INDEX = b"""<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-posts.xml.gz</loc></sitemap>
</sitemapindex>"""

POSTS = b"""<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://example.com/posts/1</loc></url>
  <url><loc>https://example.com/posts/2</loc></url>
</urlset>"""


def handler(request: httpx.Request) -> httpx.Response:
    if request.url.path == "/sitemap.xml":
        return httpx.Response(200, headers={"Content-Type": "application/xml"}, content=INDEX)
    if request.url.path == "/sitemap-posts.xml.gz":
        return httpx.Response(
            200,
            headers={"Content-Type": "application/xml", "Content-Encoding": "gzip"},
            content=gzip.compress(POSTS),
        )
    return httpx.Response(404)


async def main():
    client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
    with tempfile.TemporaryDirectory() as tmp:
        seeder = AsyncUrlSeeder(client=client, base_directory=tmp, cache_root=tmp)
        results = await seeder.urls("example.com", SeedingConfig(source="sitemap", force=True))
    print("URLs found:", [r["url"] for r in results])


asyncio.run(main())

Output:

URLs found: []

Expected:

URLs found: ['https://example.com/posts/1', 'https://example.com/posts/2']
OS

Windows 11

Python version

3.13.5

Browser

Not used (the URL seeder only uses httpx)

Browser version
Error logs & Screenshots (if applicable)

With logger=AsyncLogger(verbose=True) passed to AsyncUrlSeeder:

Error processing sub-sitemap https://example.com/sitemap-posts.xml.gz: Not a gzipped file (b'<?')

I'm opening a PR that checks the gzip magic bytes (1f 8b) instead of the URL suffix, with tests.

AI-assisted: found and reproduced with an AI coding agent; the example above was run locally.

Lenguaje dominante
Python
Estrellas
87.2k
Forks
9.1k
Merge medio
1 d 6 h
PR fusionados (30 d)
25

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.