[Bug]: URL seeder drops .gz sitemaps that are served with Content-Encoding: gzip
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Aptitud para principiantes
- 50/100
Línea de trabajo
The gzip check is in _iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py, where the body is decompressed only when the URL ends in .gz. Start there, then run the MockTransport script from the issue to see the empty result. Done means that script prints both example.com/posts URLs, and tests cover a gzipped body without a .gz URL and a .gz URL served with Content-Encoding gzip. The reporter says they are opening a PR, so check for it before starting.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
crawl4ai version
0.9.5 (also reproduced on develop at 1a6eddb)
Expected Behavior
AsyncUrlSeeder returns the URLs of a .xml.gz sitemap whether the server sends the gzip file as it is, or with Content-Encoding: gzip.
Current Behavior
_iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py gunzip the body whenever the sitemap URL ends with .gz:
data = gzip.decompress(r.content) if url.endswith(".gz") else r.content
When the server sends the file with Content-Encoding: gzip (nginx gzip_static, Apache AddEncoding gzip .gz, S3/CloudFront objects with that metadata), httpx removes that encoding before the seeder sees the body. The seeder then gets plain XML, gzip.decompress() raises BadGzipFile, and every URL of that sitemap is dropped.
The other way round, gzipped content at a URL that doesn't end in .gz (for example sitemap-posts.xml.gz?v=2) is never decompressed, and no <loc> is found.
Is this reproducible?
Yes
Inputs Causing the Bug
A sitemap index that lists https://example.com/sitemap-posts.xml.gz, served with Content-Type: application/xml and Content-Encoding: gzip.
Steps to Reproduce
Run the script below. It uses httpx's MockTransport, so no network is needed.
Code snippets
import asyncio
import gzip
import tempfile
import httpx
from crawl4ai import AsyncUrlSeeder, SeedingConfig
INDEX = b"""<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://example.com/sitemap-posts.xml.gz</loc></sitemap>
</sitemapindex>"""
POSTS = b"""<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://example.com/posts/1</loc></url>
<url><loc>https://example.com/posts/2</loc></url>
</urlset>"""
def handler(request: httpx.Request) -> httpx.Response:
if request.url.path == "/sitemap.xml":
return httpx.Response(200, headers={"Content-Type": "application/xml"}, content=INDEX)
if request.url.path == "/sitemap-posts.xml.gz":
return httpx.Response(
200,
headers={"Content-Type": "application/xml", "Content-Encoding": "gzip"},
content=gzip.compress(POSTS),
)
return httpx.Response(404)
async def main():
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
with tempfile.TemporaryDirectory() as tmp:
seeder = AsyncUrlSeeder(client=client, base_directory=tmp, cache_root=tmp)
results = await seeder.urls("example.com", SeedingConfig(source="sitemap", force=True))
print("URLs found:", [r["url"] for r in results])
asyncio.run(main())
Output:
URLs found: []
Expected:
URLs found: ['https://example.com/posts/1', 'https://example.com/posts/2']
OS
Windows 11
Python version
3.13.5
Browser
Not used (the URL seeder only uses httpx)
Browser version
Error logs & Screenshots (if applicable)
With logger=AsyncLogger(verbose=True) passed to AsyncUrlSeeder:
Error processing sub-sitemap https://example.com/sitemap-posts.xml.gz: Not a gzipped file (b'<?')
I'm opening a PR that checks the gzip magic bytes (1f 8b) instead of the URL suffix, with tests.
AI-assisted: found and reproduced with an AI coding agent; the example above was run locally.
- Lenguaje dominante
- Python
- Estrellas
- 87.2k
- Forks
- 9.1k
- Merge medio
- 1 d 6 h
- PR fusionados (30 d)
- 25
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackPosiblemente ocupada @HhH666-ux la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
unclecode/crawl4ai#2494 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresAbierto⚙ Done ✨ Enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 1 día
-
⚙ Done 🐞 Bug 📌 Root caused
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaAbierto⚙ Done 🐞 Bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
unclecode/crawl4ai#2469 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de unclecode/crawl4ai
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
Install the skills as a Claude Code pluginPosiblemente ocupada @ruban-24 la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
Los mantenedores suelen responder en 6 días
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationAbiertodata
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100