reworkd/AgentGPT

馃寧 Add web scraping capabilities

Open

#684 opened on Jun 5, 2023

View on GitHub
聽(2 comments)聽(4 reactions)聽(0 assignees)TypeScript聽(34,594 stars)聽(9,446 forks)batch import
enhancementhelp wantedjavascriptpython

Description

Currently we just do a simple Google search and summarize the descriptions of websites. We should also parse each website individual and aggregate information.

This would be a large undertaking

Contributor guide

Tech stack
typescriptnodejs
Domain
backenddata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
TypeScriptweb scraping conceptsNode.js
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Review the existing Google search integration in the codebase to understand the current data flow. Examine potential scraping libraries compatible with TypeScript, such as Cheerio or Puppeteer. Assess the structure of target websites to determine parsing strategies. Consider error handling and rate limiting to avoid IP blocks. Propose a design that aggregates scraped data with existing search results.