MLSAKIIT/pixly

Implement a more robust web scrapper

开放

#12 创建于 2025年10月14日

 (2 条评论) (0 个反应) (1 位负责人)Python (15 个派生)auto 404
enhancementhacktoberfest

仓库指标

星标
 (20 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Web Scrappers in knowledge_manager.py, extract_wiki_content() and extract_forum_content() sometimes gets blocked by certain websites, (namely the ones present in minecraft.csv)

Recommended Solutions :

  • Use an existing tool such as scrappy or playwright.
  • Mask the scrappers to behave to more like human by introducing delays between different searches.
  • Make the scrapper behave like a browser as much as possible
  • Use Proxies to circumvent IP bans.
  • Rotate a list of User Agents and headers.
  • Make it asynchronous using asyncio + httpx

贡献者指南