[BUG] Scrape tools reject the documented cookies mapping and drop cookies when the URL is passed at runtime

Open Beginner friendly
  • Is something wrong?
#9,007 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Luckyou-d is already working on this.

Since Nov 4, 2026.

  • #9060 by @Luckyou-d — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
security, tooling

Research direction

Start with the cookies handling in lib/crewai-tools/src/crewai_tools/tools/scrape_website_tool/scrape_website_tool.py (around lines 66-67), then make the same change in scrape_element_from_website/scrape_element_from_website.py. Keep the assignment inside the website_url branch so cookies still apply only to a pinned URL, and check that _run passes self.cookies to requests unchanged. Done when a plain {"sessionid": "abc123"} mapping is sent with the request and the tests cover both tools.

Written by the indexing model from the issue text.

Description

Description

ScrapeWebsiteTool and ScrapeElementFromWebsiteTool declare cookies as a plain mapping and document it as such, but their constructors read two literal keys from it and treat the second as the name of an environment variable:

    cookies: dict[str, str] | None = None
    ...

        if website_url is not None:
            ...
            if cookies is not None:
                self.cookies = {cookies["name"]: os.getenv(cookies["value"]) or ""}

(lib/crewai-tools/src/crewai_tools/tools/scrape_website_tool/scrape_website_tool.py:66-67, identical in scrape_element_from_website/scrape_element_from_website.py:67-68)

The docs describe cookies as "A dictionary containing cookies to be sent with the request" (docs/edge/en/tools/web-scraping/scrapeelementfromwebsitetool.mdx:78), and _run passes self.cookies straight to requests. The documented input therefore fails:

ScrapeWebsiteTool(
    website_url="https://example.com",
    cookies={"sessionid": "abc123"},
)
# KeyError: 'name'

Only the undocumented shape {"name": "sessionid", "value": "SESSION_COOKIE_ENV_VAR"} produces a cookie, and its value comes from the environment rather than from the caller.

Suggested fix

Store the mapping as given, while keeping the existing restriction that cookies only apply to a tool constructed with a pinned website_url:

if website_url is not None:
    ...
    if cookies is not None:
        self.cookies = cookies

Keeping the assignment inside the website_url branch is deliberate: the pinned-URL schema removes website_url from the agent's arguments, so the cookie jar stays scoped to the configured origin. A tool without a pinned URL accepts model-chosen URLs, and attaching developer credentials to those would leak them to an arbitrary host.

A regression test covering both tools is included in the PR.

Prepared with an AI coding agent. Please apply the llm-generated label; this account cannot set repository labels.

Dominant language
Python
Stars
60.4k
Forks
9k
Avg merge
1d 4h
Merged PRs (30d)
210

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from crewAIInc/crewAI

All issues in crewAIInc/crewAI

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.