MLSAKIIT/pixly

Implement a more robust web scrapper

開放

#12 建立於 2025年10月14日

 (2 則留言) (0 個反應) (1 位負責人)Python (15 個分叉)auto 404
enhancementhacktoberfest

倉庫指標

星標
 (20 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Web Scrappers in knowledge_manager.py, extract_wiki_content() and extract_forum_content() sometimes gets blocked by certain websites, (namely the ones present in minecraft.csv)

Recommended Solutions :

  • Use an existing tool such as scrappy or playwright.
  • Mask the scrappers to behave to more like human by introducing delays between different searches.
  • Make the scrapper behave like a browser as much as possible
  • Use Proxies to circumvent IP bans.
  • Rotate a list of User Agents and headers.
  • Make it asynchronous using asyncio + httpx

貢獻者指南