Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Sitemap Extractor: optimize download & parsing (streaming XML, gzip, keep-alive, concurrent child fetch, incremental skip, dedupe)

Abierto
  • ¿Algo no está bien?
#340 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
typescript
Área
backend, performance

Línea de trabajo

Start by locating the sitemap download and parsing path in the TypeScript project; the issue does not name specific files or tests. Read that implementation and check how it handles sitemap indexes, child fetches, and repeat runs. The proposed work spans streaming parsing, transport behavior, concurrency, deduplication, and conditional requests; done means faster, lower-memory enumeration and fewer timeouts without excessive per-host concurrency.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

t-universe

Summary

Independent of the overlap change (#339), the sitemap download/parse path itself can be made substantially faster and lighter for large sitemap trees.

Context

Same 500-domain test as #339 (250 most-popular + 250 random domains, Apr–Jun 2026). Large sites carry sitemap indexes referencing hundreds of child sitemaps, each up to 50k URLs. 448,562 pages were enumerated across the test; the largest sites timed out at 1h primarily in the sitemap phase.

Proposed optimizations

  • Streaming XML parsing (SAX, e.g. sax / saxes) instead of buffering + DOM-parsing each child sitemap — lower memory and faster time-to-first-URL on 50k-entry files.
  • Guarantee gzip transfer (Accept-Encoding: gzip) with transparent decompression — sitemaps are frequently served as .xml.gz; large savings on the wire.
  • HTTP keep-alive / connection reuse (shared agent) across the many child-sitemap requests to the same host.
  • Concurrent child-sitemap fetching with a bounded pool (also enables the overlap in #339).
  • Dedupe child-sitemap URLs — indexes commonly list duplicates.
  • Incremental skip via sitemap-index <lastmod>: skip child sitemaps unchanged since the previous run using If-Modified-Since / If-None-Match (conditional GET). Big win for re-crawls, and foundational for a future "crawl only new content" feature.
  • Bound per-host concurrency to avoid anti-bot triggers.

Expected impact

Faster sitemap enumeration, lower memory footprint, fewer timeouts on large sites, and materially cheaper re-crawls.

🤖 Filed from a data-driven analysis with Claude Code, based on a 500-domain Sitemap Extractor test (Apr–Jun 2026).

Lenguaje dominante
TypeScript
Estrellas
149
Forks
59
Merge medio
9 d 13 h
PR fusionados (30 d)
3

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apify/actor-scraper

Todos los issues de apify/actor-scraper

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.