matthewmueller/x-ray

Can't catch an error thrown when there's an invalid status code 522

Open

#90 opened on Sep 2, 2015

View on GitHub
 (1 comment) (1 reaction) (0 assignees)JavaScript (5,804 stars) (415 forks)batch import
bughelp wanted

Description

Hi,

I'm trying to scrape the url below but after some time it shows an error: http://www.lazygamer.net/boost/the-top-10-gaming-youtubers-in-2014-so-far/

assert.js:86
  throw new assert.AssertionError({
        ^
AssertionError: invalid status code: 522
    at Object.status (/Users/jonathan/git/youtube_urls_from_url/node_modules/x-ray/node_modules/http-context/lib/response.js:81:5)
    at context.status (/Users/jonathan/git/youtube_urls_from_url/node_modules/x-ray/node_modules/http-context/node_modules/delegates/index.js:92:31)
    at /Users/jonathan/git/youtube_urls_from_url/node_modules/x-ray/node_modules/x-ray-crawler/lib/http-driver.js:30:20
    at Request.callback (/Users/jonathan/git/youtube_urls_from_url/node_modules/x-ray/node_modules/x-ray-crawler/node_modules/superagent/lib/node/index.js:797:3)
    at IncomingMessage.<anonymous> (/Users/jonathan/git/youtube_urls_from_url/node_modules/x-ray/node_modules/x-ray-crawler/node_modules/superagent/lib/node/index.js:990:12)
    at IncomingMessage.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

I also attached a screenshot of the webpage loaded from the browser if it's helpful.

Contributor guide

Tech stack
javascriptnodejs
Domain
backend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Node.js basicserror handling
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
The issue occurs when an HTTP status code 522 is encountered, causing an unhandled AssertionError in `http context/lib/response.js` (line 81). The error propagates from x ray crawler's http driver.js via superagent. To fix, consider catching the assertion error or validating status codes before they are passed to the response object. No linked PRs or maintainer comments exist, so investigate whether the assertion should be wrapped in a try catch or the status code should be allowed.