Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Sitemap Extractor: fall back to Apify Unblocker when sitemap/robots.txt fetch is blocked (HTTP 403)

Abierto
  • ¿Algo no está bien?
#341 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
typescript
Área
web-dev

Línea de trabajo

The issue does not name relevant files, tests, or entry points. Start by locating sitemap and robots.txt fetching and the existing proxy retry path, then check how Apify Unblocker is configured in this repository. Done means blocked fetches retry through Unblocker while the default path remains unchanged for unblocked responses, with tests covering both paths.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

t-universe

Summary

When the sitemap (or the robots.txt used to discover it) is fetched through the default proxy and the target returns an anti-bot block — typically HTTP 403 — the run fails with "No valid sitemaps were discovered" even though a sitemap actually exists. Propose falling back to Apify Unblocker (or a residential proxy with a browser-like fingerprint) for sitemap/robots.txt retrieval when a block is detected.

Evidence

From the 500-domain test (Apr–Jun 2026), 101 runs FAILED. Classifying their logs:

Reason Count %
No sitemap genuinely absent 63 62.4%
Blocked / anti-bot (sitemap likely exists) 21 20.8%
DNS / domain unresolvable 15 14.9%
TLS/SSL error 1 1.0%
Server 5xx while fetching 1 1.0%

~1 in 5 failures were false negatives — real, sitemap-having sites returning 403 to the sitemap/robots fetch. Examples: help.shopify.com, coupang.com, tableau.com, mayoclinic.org, docs.redhat.com, qconcursos.com.

The current logs already show the pattern:

WARN  Sitemap discovery through proxy failed or returned no sitemaps. Retrying once without proxy.
ERROR [Status message]: No valid sitemaps were discovered from the provided startUrls.

The "retry without proxy" step is not enough — these hosts block datacenter IPs and basic fingerprints outright.

Proposed fix

  • Detect anti-bot blocks (HTTP 401/403/429; Cloudflare / "just a moment" / captcha markers) on sitemap and robots.txt fetches.
  • On block, retry the fetch through Apify Unblocker (or residential proxy with a browser-like fingerprint) before giving up.
  • Escalate to Unblocker only on block (cost control) — keep the fast default-proxy path first.

Expected impact

Recovers ~20% of current sitemap failures — real sites we drop today — and does so on exactly the high-value large/popular domains most likely to sit behind anti-bot.

🤖 Filed from a data-driven analysis with Claude Code, based on a 500-domain Sitemap Extractor test (Apr–Jun 2026).

Lenguaje dominante
TypeScript
Estrellas
149
Forks
59
Merge medio
9 d 13 h
PR fusionados (30 d)
3

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apify/actor-scraper

Todos los issues de apify/actor-scraper

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.