featurehelp wanted
Metriche repository
- Star
- (5337 star)
- Metriche merge PR
- (Nessuna PR mergiata in 30 g)
Descrizione
I get the following error while the consumer spider running:
RedisMixin.spider_idle of <MySpider 'my_spider' at 0x7f5c2cb335c0>>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/signal.py", line 32, in send_catch_log
*arguments, **named)
File "/usr/local/lib/python3.6/dist-packages/pydispatch/robustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 121, in spider_idle
self.schedule_next_requests()
File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 116, in schedule_next_requests
self.crawler.engine.crawl(req, spider=self)
File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 216, in crawl
self.schedule(request, spider)
File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 222, in schedule
if not self.slot.scheduler.enqueue_request(request):
File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/scheduler.py", line 167, in enqueue_request
self.queue.push(request)
File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/queue.py", line 104, in push
self.server.execute_command('ZADD', self.key, score, data)
File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 730, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 321, in read_response
raw = self._buffer.readline()
File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 256, in readline
self._read_from_socket()
File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 201, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
2020-05-01 07:38:27 [scrapy.core.engine] INFO: Closing spider (finished)
I think we need to reconnect to redis if we have cases like this.