NikolaiT/se-scraper

Rotating proxy avaible ?

Open

#27 opened on Jun 7, 2019

View on GitHub
 (8 comments) (0 reactions) (0 assignees)HTML (523 stars) (124 forks)batch import
help wanted

Description

Hi,

Is there any possibility to retry request if response code is != 200 ? or if connection to proxy failed ?

/BR

Contributor guide

Tech stack
javascriptnodejs
Domain
backendapi
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
puppeteer basicsproxy configuration concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue requests the ability to retry requests on non 200 status codes or proxy connection failures, and to rotate proxies. To implement this, one would need to examine the existing request handling in the scraping module, likely in files like 'src/scraper.js' or similar. The maintainers have not clarified the desired behavior or acceptance criteria, so further investigation into the codebase and discussion with maintainers is needed to define the feature scope.