Performance regressions in generic scrapers

Abierto
  • ¿Algo no está bien?
#362 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 12 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
typescript
Área
backend, performance

Línea de trabajo

Start by identifying the builds and scraper versions named in the issue, then inspect the changes between them and use the reported Snowflake APIFY_ANALYTICS comparisons to narrow down the regressions. Prioritize the playwright-scraper failure and restart spike, then investigate the cheerio-scraper memory spike and scaling-related web-scraper slowdown. Done means the regressions are root-caused and corrected, with the comparisons rerun to verify performance and reliability.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

t-universe

Summary

Build-over-build performance comparison across the scrapers in this repo (last 7 days, Snowflake APIFY_ANALYTICS, all-runs aggregate + a controlled 1-clean-item short-run subset). Three of the four scrapers show a real regression in the newer build; puppeteer-scraper looks like a clean improvement and is not part of the problem list below — see the note at the end.

cheerio-scraper — memory/efficiency regression

Builds: gTZUgxTMzuh0BBr4b (old) → 6EleDKkR2avUKBelE (new)

  • Peak memory on minimal (1-item output) runs more than doubled: 1,733 MB → 6,567 MB. The exact same peak value shows up in the full aggregate too, so this looks like one run spiking to 6.5 GB — worth pulling that run to check for a leak.
  • Avg CU per clean item worsened ~56% (avg items/run dropped 107.7 → 70.6, while avg CU/run stayed roughly flat).
  • Avg allocated memory +15%, avg CPU +26%.
  • Minor reliability dip: abort rate 0.1% → 0.4%, restarts introduced (0 → 0.05 avg/run).
  • Not all bad: median duration improved (10.7s → 6.6s).

web-scraper — slower and less efficient at scale

Builds: byk3boe85HewTmSD9 (old) → XHPrjngguiKJ17ZsN (new)

  • Avg duration +29.7%, p95 duration +35.5% in the full run mix; avg CU +21.9%.
  • CU per clean item worse by ~12% even after crediting the new build for producing slightly more items.
  • Reliability regressed: success rate -0.9pp, fail rate +0.3pp, timeout rate +0.5pp.
  • Regression is concentrated in heavier/larger runs — the controlled short-run (1-item) subset is nearly flat, so this isn't a fixed per-call overhead issue, it's something that scales with run size.
  • One improvement: peak memory down 32%.

playwright-scraper — severe regression, do not roll out further

Builds: 8Dwn9FInP9Z5gsRy1 (old) → vi1WR785wACojOJg7 (new)

  • Fail rate: 0.7% → 14.2% (+13.5pp) — roughly 1 in 7 runs now fail outright.
  • Avg restarts: 0.02 → 0.43 (~21.5×), consistent with a crash/retry loop.
  • Avg clean items/run collapsed 42% (24.1 → 13.9).
  • Even on minimal 1-item runs, avg duration is +35% and p95 is +47% slower.
  • CU per clean item worse by ~62%.
  • Recommend root-causing the failure spike before this build goes any further — this reads as a functional regression, not a tuning tradeoff.

puppeteer-scraper — looks good, please retest after the above are fixed

Builds: WGHufYdES3oOahj9x (old) → 9Ho7gB6bqt4AU1UtL (new)

  • Median duration nearly halved (72.4s → 36.1s), consistent across both the aggregate and controlled short-run subset.
  • CU per clean item slightly improved despite a higher raw avg-CU/run number (fully explained by producing ~25% more items/run).
  • Peak memory down ~20%, reliability unchanged.
  • No action needed here, but please re-run this comparison once the other three scrapers are patched, in case a shared dependency/base-image bump introduced in this cycle also affects puppeteer-scraper in ways not yet visible in the current traffic mix.

Generated from a 7-day build comparison (all-runs aggregate + 1-clean-item short-run subset) via Redash/Snowflake APIFY_ANALYTICS.

Lenguaje dominante
TypeScript
Estrellas
149
Forks
59
Merge medio
11 d 9 h
PR fusionados (30 d)
6

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apify/actor-scraper

Todos los issues de apify/actor-scraper

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.