Cookies from the Cookie request header are not processed
#1.992 aberto em 16 de mai. de 2016
Métricas do repositório
- Stars
- (61.658 stars)
- Métricas de merge de PR
- (Mesclagem média 42d 12h) (76 fundiu PRs em 30d)
Description
I am new in scrapy, and I meet some problems which I can not get answer from google, so I post it here:
1 Cookie not work even set in DEFAULT_REQUEST_HEADERS:
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'accept-encoding': 'gzip, deflate, sdch',
'cache-control': 'no-cache',
'cookie': 'xx=yy',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
}
class MySpider(scrapy.Spider):
def make_requests_from_url(self, url):
return scrapy.http.Request(url, headers=DEFAULT_REQUEST_HEADERS)
I know the make_requests_from_url will only called once for the start_urls, and in my opinion, the first request will send the cookie I set in the DEFAULT_REQUEST_HEADERS, however it does not.
2 Share settings between spiders.
I have multiple spiders in the project which share most of the settings like RandomAgentMiddleware RandomProxyMiddleware UserAgent DEFAULT_REQUEST_HEADERS and etc, however they are configured inside the settings.py for each spider.
Is it possible to share these settings?
The
COOKIES_ENABLED is set to true.