Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug]: When extracting data with scroll_full_page, only the final elements get parsed

未关闭
  • 哪里不对？
#731 22 条评论 3 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

  • #1877 来自 @hafezparast —— 已关闭，未合并

评估

难度
4/5
预计耗时
3-5 天
新手友好度
47/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
冷清
技术栈
python
领域
backend

调研方向

Reproduce the issue with AsyncWebCrawler and CrawlerRunConfig against the supplied X URL, comparing scroll_full_page on and off. Inspect how result.html, screenshots, and JsonCssExtractionStrategy process the scrolled page. Done means extraction retains the full set of visible profiles rather than only the final elements, with the existing reproduction succeeding.

由索引模型根据 Issue 内容生成。

描述

⚙️ In-progress 🐞 Bug
crawl4ai version

0.4.248

Expected Behavior

I'm crawiling Twitter, specifically the "following" section of a profile. I have a css selector for the relevant data (user's names and bios) and set up a JsonCssExtractionStrategy. If I don't use scroll_full_page, I understandably only expect to get the first N user profiles. But if I do enable scroll_full_page, I expect the returned data to contain the list to the fullest extent as visible when browsing manually.

Current Behavior

When not using scroll_full_page, I do get the first 16 profiles in this case. However, when setting scroll_full_page, I only get the LAST 12. It's important to note that there are over 40 profiles listed, so none of the 12 profiles intersect with the first 16. I checked the result's html property and it does only contain information about the last 12. However, strangely the screenshot saved contains all the profiles.

Is this reproducible?

Yes

Inputs Causing the Bug
https://x.com/SomeTwitterProfile/following
Steps to Reproduce
Execute the following snippet with both scoll_full_page on and off.
Code snippets
site_url = "https://x.com/elonmusk/following"

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode, BrowserConfig, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
import json


async def main():
    schema = {
        "name": "Followers",
        "baseSelector": "button[data-testid='UserCell']",
        "fields": [
            {
              "name": "name",
              "selector": "span",
              "type": "text"
	    },
            {
              "name": "handle",
              "selector": 'a[role="link"] > div > div[dir="ltr"]:only-child > span ',
              "type": "text"
	    },
            {
              "name": "bio",
              "selector": "div[dir='auto'] > span",
              "type": "html",
	     }
                        
	 ]
    }
    
    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
    
    browser_conf = BrowserConfig(
        extra_args=['--disable-web-security'],
        cookies=[
            {"name": "auth_token", "value": "YOURAUTHTOKEN", "domain": ".x.com", "path": "/"},
        ],
    )
    
    crawler_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS, #        cache_mode=CacheMode.BYPASS/DISABLED,
        screenshot=True,
        wait_for="css:button[data-testid='UserCell']:nth-child(1)",
        page_timeout=5000,
        extraction_strategy=extraction_strategy,
        # scan_full_page=True,
        scroll_delay=1.5,
    )

    try:
        async with AsyncWebCrawler(
            config=browser_conf,
            verbose=True,
        ) as crawler:
            result = await crawler.arun(
                url=site_url ,
                config=crawler_config,
            )
                
            from base64 import b64decode
            with open("screenshot.png", "wb") as f:
              f.write(b64decode(result.screenshot))
              
            with open("index.html", "w") as f:
              f.write(result.html)
                                
            data = json.loads(result.extracted_content)
            print(f"Extracted {len(data)} users")
            print(json.dumps(data, indent=2) if data else "No data found")
                    
    except Exception as e:
        print(f"Something happened: {e}")
        raise e

if __name__ == "__main__":
    asyncio.run(main())
OS

Linux

Python version

3.12.9

Browser

Default

Browser version

No response

Error logs & Screenshots (if applicable)

No response

主要语言
Python
星标
85.4k
派生
8.9k
平均合并
5 天 7 小时
30 天内合并 PR
23

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

unclecode/crawl4ai 的其他 Issue

查看 unclecode/crawl4ai 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。