[Bug]: URL seeder drops .gz sitemaps that are served with Content-Encoding: gzip
Maintainer antworten meist innerhalb von 1 Tag
Bewertung
- Schwierigkeit
- 2/5
- Geschätzter Aufwand
- 1-3 Stunden
- Anfängerfreundlichkeit
- 50/100
Rechercherichtung
The gzip check is in _iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py, where the body is decompressed only when the URL ends in .gz. Start there, then run the MockTransport script from the issue to see the empty result. Done means that script prints both example.com/posts URLs, and tests cover a gzipped body without a .gz URL and a .gz URL served with Content-Encoding gzip. The reporter says they are opening a PR, so check for it before starting.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
crawl4ai version
0.9.5 (also reproduced on develop at 1a6eddb)
Expected Behavior
AsyncUrlSeeder returns the URLs of a .xml.gz sitemap whether the server sends the gzip file as it is, or with Content-Encoding: gzip.
Current Behavior
_iter_sitemap() and _iter_sitemap_content() in crawl4ai/async_url_seeder.py gunzip the body whenever the sitemap URL ends with .gz:
data = gzip.decompress(r.content) if url.endswith(".gz") else r.content
When the server sends the file with Content-Encoding: gzip (nginx gzip_static, Apache AddEncoding gzip .gz, S3/CloudFront objects with that metadata), httpx removes that encoding before the seeder sees the body. The seeder then gets plain XML, gzip.decompress() raises BadGzipFile, and every URL of that sitemap is dropped.
The other way round, gzipped content at a URL that doesn't end in .gz (for example sitemap-posts.xml.gz?v=2) is never decompressed, and no <loc> is found.
Is this reproducible?
Yes
Inputs Causing the Bug
A sitemap index that lists https://example.com/sitemap-posts.xml.gz, served with Content-Type: application/xml and Content-Encoding: gzip.
Steps to Reproduce
Run the script below. It uses httpx's MockTransport, so no network is needed.
Code snippets
import asyncio
import gzip
import tempfile
import httpx
from crawl4ai import AsyncUrlSeeder, SeedingConfig
INDEX = b"""<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://example.com/sitemap-posts.xml.gz</loc></sitemap>
</sitemapindex>"""
POSTS = b"""<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://example.com/posts/1</loc></url>
<url><loc>https://example.com/posts/2</loc></url>
</urlset>"""
def handler(request: httpx.Request) -> httpx.Response:
if request.url.path == "/sitemap.xml":
return httpx.Response(200, headers={"Content-Type": "application/xml"}, content=INDEX)
if request.url.path == "/sitemap-posts.xml.gz":
return httpx.Response(
200,
headers={"Content-Type": "application/xml", "Content-Encoding": "gzip"},
content=gzip.compress(POSTS),
)
return httpx.Response(404)
async def main():
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
with tempfile.TemporaryDirectory() as tmp:
seeder = AsyncUrlSeeder(client=client, base_directory=tmp, cache_root=tmp)
results = await seeder.urls("example.com", SeedingConfig(source="sitemap", force=True))
print("URLs found:", [r["url"] for r in results])
asyncio.run(main())
Output:
URLs found: []
Expected:
URLs found: ['https://example.com/posts/1', 'https://example.com/posts/2']
OS
Windows 11
Python version
3.13.5
Browser
Not used (the URL seeder only uses httpx)
Browser version
Error logs & Screenshots (if applicable)
With logger=AsyncLogger(verbose=True) passed to AsyncUrlSeeder:
Error processing sub-sitemap https://example.com/sitemap-posts.xml.gz: Not a gzipped file (b'<?')
I'm opening a PR that checks the gzip magic bytes (1f 8b) instead of the URL suffix, with tests.
AI-assisted: found and reproduced with an AI coding agent; the example above was run locally.
- Vorherrschende Sprache
- Python
- Sterne
- 87.2k
- Forks
- 9.1k
- Ø Merge
- 1 T. 6 Std.
- Gemergte PRs (30 T.)
- 25
Entwicklungsumgebung
- Enthält ein Dockerfile oder eine Docker-Compose-Datei
- Hat eine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus unclecode/crawl4ai
-
BUG: UserAgentGenerator.generate ignores browser_type and returns a random browser stackEvtl. vergeben @HhH666-ux hat das heute übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 85/100
Maintainer antworten meist innerhalb von 1 Tag
-
[Bug]: Hardened docker-compose (read_only) keeps full Chromium from starting: crashpad needs a writable ~/.configEvtl. vergeben Ein verknüpfter Pull Request ist offen oder bereits gemergt. Offen🐞 Bug 📌 Root caused
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
unclecode/crawl4ai#2494 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 1 Tag
-
[Feature]: Crawl4aiDockerClient has no public option for the API token the 0.9 server requiresOffen⚙ Done ✨ Enhancement
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 77/100
Maintainer antworten meist innerhalb von 1 Tag
-
⚙ Done 🐞 Bug 📌 Root caused
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
Maintainer antworten meist innerhalb von 1 Tag
-
[Bug]: PDF images saved with save_images_locally=True come back with an empty src in result.mediaOffen⚙ Done 🐞 Bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
unclecode/crawl4ai#2469 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 1 Tag
Alle Issues in unclecode/crawl4ai
Ähnliche Issues
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
Maintainer antworten meist innerhalb von 1 Tag
-
documentation
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 85/100
Maintainer antworten meist innerhalb von 1 Tag
-
Install the skills as a Claude Code pluginEvtl. vergeben @ruban-24 hat das heute übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
Maintainer antworten meist innerhalb von 6 Tagen
-
Make the usage of time-zone aware UTC timestamps the norm across the pynxtools NOMAD ecosystemOffen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 62/100
FAIRmat-NFDI/pynxtools#874 ·
-
Eval: PublicationOffendata
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 66/100