rmax/scrapy-redis

Should we auto reconnect to redis?

Open

#167 opened on May 1, 2020

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Python (1,586 forks)batch import
featurehelp wanted

Repository metrics

Stars
 (5,337 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

I get the following error while the consumer spider running:

RedisMixin.spider_idle of <MySpider 'my_spider' at 0x7f5c2cb335c0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/signal.py", line 32, in send_catch_log
    *arguments, **named)
  File "/usr/local/lib/python3.6/dist-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 121, in spider_idle
    self.schedule_next_requests()
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 116, in schedule_next_requests
    self.crawler.engine.crawl(req, spider=self)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 216, in crawl
    self.schedule(request, spider)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 222, in schedule
    if not self.slot.scheduler.enqueue_request(request):
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/scheduler.py", line 167, in enqueue_request
    self.queue.push(request)
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/queue.py", line 104, in push
    self.server.execute_command('ZADD', self.key, score, data)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 730, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 321, in read_response
    raw = self._buffer.readline()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
2020-05-01 07:38:27 [scrapy.core.engine] INFO: Closing spider (finished)

I think we need to reconnect to redis if we have cases like this.

Contributor guide