[Bug]: URL seeder drops .gz sitemaps that are served with Content-Encoding: gzip

Offen
#2,541 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@adityajai119 arbeitet bereits daran.

Seit 12.11.2026.

  • #2542 von @adityajai119 — offen

Bewertung

Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Anfängerfreundlichkeit
50/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
backend

Rechercherichtung

The gzip check is in _iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py, where the body is decompressed only when the URL ends in .gz. Start there, then run the MockTransport script from the issue to see the empty result. Done means that script prints both example.com/posts URLs, and tests cover a gzipped body without a .gz URL and a .gz URL served with Content-Encoding gzip. The reporter says they are opening a PR, so check for it before starting.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

crawl4ai version

0.9.5 (also reproduced on develop at 1a6eddb)

Expected Behavior

AsyncUrlSeeder returns the URLs of a .xml.gz sitemap whether the server sends the gzip file as it is, or with Content-Encoding: gzip.

Current Behavior

_iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py gunzip the body whenever the sitemap URL ends with .gz:

data = gzip.decompress(r.content) if url.endswith(".gz") else r.content

When the server sends the file with Content-Encoding: gzip (nginx gzip_static, Apache AddEncoding gzip .gz, S3/CloudFront objects with that metadata), httpx removes that encoding before the seeder sees the body. The seeder then gets plain XML, gzip.decompress() raises BadGzipFile, and every URL of that sitemap is dropped.

The other way round, gzipped content at a URL that doesn't end in .gz (for example sitemap-posts.xml.gz?v=2) is never decompressed, and no <loc> is found.

Is this reproducible?

Yes

Inputs Causing the Bug

A sitemap index that lists https://example.com/sitemap-posts.xml.gz, served with Content-Type: application/xml and Content-Encoding: gzip.

Steps to Reproduce

Run the script below. It uses httpx's MockTransport, so no network is needed.

Code snippets
import asyncio
import gzip
import tempfile

import httpx
from crawl4ai import AsyncUrlSeeder, SeedingConfig

INDEX = b"""<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-posts.xml.gz</loc></sitemap>
</sitemapindex>"""

POSTS = b"""<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://example.com/posts/1</loc></url>
  <url><loc>https://example.com/posts/2</loc></url>
</urlset>"""


def handler(request: httpx.Request) -> httpx.Response:
    if request.url.path == "/sitemap.xml":
        return httpx.Response(200, headers={"Content-Type": "application/xml"}, content=INDEX)
    if request.url.path == "/sitemap-posts.xml.gz":
        return httpx.Response(
            200,
            headers={"Content-Type": "application/xml", "Content-Encoding": "gzip"},
            content=gzip.compress(POSTS),
        )
    return httpx.Response(404)


async def main():
    client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
    with tempfile.TemporaryDirectory() as tmp:
        seeder = AsyncUrlSeeder(client=client, base_directory=tmp, cache_root=tmp)
        results = await seeder.urls("example.com", SeedingConfig(source="sitemap", force=True))
    print("URLs found:", [r["url"] for r in results])


asyncio.run(main())

Output:

URLs found: []

Expected:

URLs found: ['https://example.com/posts/1', 'https://example.com/posts/2']
OS

Windows 11

Python version

3.13.5

Browser

Not used (the URL seeder only uses httpx)

Browser version
Error logs & Screenshots (if applicable)

With logger=AsyncLogger(verbose=True) passed to AsyncUrlSeeder:

Error processing sub-sitemap https://example.com/sitemap-posts.xml.gz: Not a gzipped file (b'<?')

I'm opening a PR that checks the gzip magic bytes (1f 8b) instead of the URL suffix, with tests.

AI-assisted: found and reproduced with an AI coding agent; the example above was run locally.

Vorherrschende Sprache
Python
Sterne
87.2k
Forks
9.1k
Ø Merge
1 T. 6 Std.
Gemergte PRs (30 T.)
25

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus unclecode/crawl4ai

Alle Issues in unclecode/crawl4ai

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.