New Integration: Firecrawl

Description

Add Firecrawl integration to sync web scraping results, crawl data, and extracted structured content into CORE.

Reference Implementations

Existing Integrations (use as templates)

• integrations/github/ - API-based data integration
• integrations/linear/ - Project management integration
• integrations/slack/ - For reference

Required Files Structure

integrations/firecrawl/
├── src/
│   ├── index.ts          # Main entry, API key spec
│   ├── schedule.ts       # Sync logic
│   ├── utils.ts          # Firecrawl API utilities
│   ├── account-create.ts # API key setup
│   └── create-activity.ts # Activity formatting
├── package.json
├── tsup.config.ts
└── README.md

Firecrawl API Integration

Authentication Setup

• Use API Key authentication (Bearer token)
• API keys are generated at firecrawl.dev dashboard
• Base URL: https://api.firecrawl.dev/v1

Key Endpoints

• POST /scrape - Scrape a single URL, returns markdown/HTML/structured data
• POST /crawl - Crawl an entire website recursively, returns all subpages
• GET /crawl/{id} - Check crawl job status
• POST /map - URL discovery and sitemap generation
• POST /extract - Extract structured data using LLM

Output Formats

• Markdown, HTML, rawHtml, screenshot, links, JSON, branding

Events to Track

1. Scrape Jobs - Single URL scrape completions with extracted content
2. Crawl Jobs - Multi-page crawl progress and completion
3. Map Results - URL discovery results
4. Extract Results - Structured data extraction completions

Implementation Tasks

• Set up basic integration structure following integrations/github/src/index.ts pattern
• Implement API key authentication flow in account-create.ts
• Create API utilities in utils.ts for scrape, crawl, map, and extract endpoints
• Implement sync logic in schedule.ts for polling crawl/scrape job results
• Convert scrape/crawl results to CORE activity format
• Add error handling and rate limiting (500 free credits, then paid plans)
• Create integration documentation
• Add to integrations/README.md

Technical Notes

• Firecrawl handles JavaScript-rendered dynamic content automatically
• Crawl jobs are asynchronous - need polling via GET /crawl/{id}
• Free tier: 500 scraped pages, then Hobby/Standard/Growth plans
• SDKs available for Python and Node.js
• Consider webhook support for async job completion notifications

Resources

• Firecrawl Documentation
• Firecrawl Website
• Firecrawl GitHub

Labels

enhancement, integration, new-feature