rmax/scrapy-redis

Should we auto reconnect to redis?

Open

#167 aperta il 1 mag 2020

Vedi su GitHub
 (2 commenti) (0 reazioni) (0 assegnatari)Python (1586 fork)batch import
featurehelp wanted

Metriche repository

Star
 (5337 star)
Metriche merge PR
 (Nessuna PR mergiata in 30 g)

Descrizione

I get the following error while the consumer spider running:

RedisMixin.spider_idle of <MySpider 'my_spider' at 0x7f5c2cb335c0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/signal.py", line 32, in send_catch_log
    *arguments, **named)
  File "/usr/local/lib/python3.6/dist-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 121, in spider_idle
    self.schedule_next_requests()
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 116, in schedule_next_requests
    self.crawler.engine.crawl(req, spider=self)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 216, in crawl
    self.schedule(request, spider)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 222, in schedule
    if not self.slot.scheduler.enqueue_request(request):
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/scheduler.py", line 167, in enqueue_request
    self.queue.push(request)
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/queue.py", line 104, in push
    self.server.execute_command('ZADD', self.key, score, data)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 730, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 321, in read_response
    raw = self._buffer.readline()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
2020-05-01 07:38:27 [scrapy.core.engine] INFO: Closing spider (finished)

I think we need to reconnect to redis if we have cases like this.

Guida contributor