scrapy/scrapy

Cookies from the Cookie request header are not processed

Open

#1,992 创建于 2016年5月16日

在 GitHub 查看
 (8 评论) (0 反应) (0 负责人)Python (61,658 star) (11,547 fork)batch import
bughelp wanted

描述

I am new in scrapy, and I meet some problems which I can not get answer from google, so I post it here:

1 Cookie not work even set in DEFAULT_REQUEST_HEADERS:

DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'accept-encoding': 'gzip, deflate, sdch',
    'cache-control': 'no-cache',
    'cookie': 'xx=yy',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
}

class MySpider(scrapy.Spider):
    def make_requests_from_url(self, url):
        return scrapy.http.Request(url, headers=DEFAULT_REQUEST_HEADERS)

I know the make_requests_from_url will only called once for the start_urls, and in my opinion, the first request will send the cookie I set in the DEFAULT_REQUEST_HEADERS, however it does not.

2 Share settings between spiders.

I have multiple spiders in the project which share most of the settings like RandomAgentMiddleware RandomProxyMiddleware UserAgent DEFAULT_REQUEST_HEADERS and etc, however they are configured inside the settings.py for each spider.

Is it possible to share these settings?

The COOKIES_ENABLED is set to true.

贡献者指南

技术栈
python
领域
backend
议题类型
bug
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
half day
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
Basic understanding of Scrapy request/response flowFamiliarity with cookie handling in Python
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
40
研究方向
Investigate how Scrapy processes cookies from request headers. Look at the 'scrapy.downloadermiddlewares.cookies' module and the 'CookiesMiddleware' implementation. Check if there is a conflict between manually set 'Cookie' header and Scrapy's internal cookie handling. The issue may require modifying the middleware to respect pre set cookies or clarifying documentation. The comments in the issue may provide additional context.
Cookies from the Cookie request header are not processed · scrapy/scrapy#1992 | Good First Issue