Phase 4: Resource ingestion pipeline for automated content discovery

Start by reading src/lib/ingestion/run-ingestion.ts, src/lib/ingestion/sources/index.ts, and src/lib/ingestion/classify-match.ts, then trace the cron route and the admin actions. Review vercel.json and the existing ingestion flow before deciding how to divide the source adapters, scheduling, matching, monitoring, and review-workflow work. Done means the listed ingestion stages and priority sources operate through admin approval.

Description

The codebase already has a partial ingestion pipeline implementation in src/lib/ingestion/. This issue tracks completing it for automated content discovery from external sources.

Current State (from codebase exploration)

• src/lib/ingestion/types.ts - TypeScript types for candidates
• src/lib/ingestion/sources/index.ts - Source adapters (GitHub Student Pack, etc.)
• src/lib/ingestion/classify-match.ts - Duplicate detection logic
• src/lib/ingestion/run-ingestion.ts - Main ingestion orchestration
• src/lib/ingestion/queries.ts - Database queries for candidates
• src/app/api/cron/ingest-resources/route.ts - Cron endpoint
• src/app/admin/(dashboard)/ingestion/page.tsx - Admin review UI
• src/app/admin/(dashboard)/ingestion/actions.ts - Approve/reject actions

What's Needed to Complete

1. Source implementations: Complete adapters for:
   • GitHub Student Pack (official API or scraping)
   • Major cloud providers (AWS, Azure, GCP student pages)
   • Popular tool vendor pages (JetBrains, Figma, Notion, etc.)
2. Scheduling: Vercel Cron to run ingestion weekly
3. Deduplication improvement: Better matching logic in classify-match.ts
4. Admin review workflow: Already partially built, needs polish
5. Source monitoring: Track which sources are healthy/stale

Files to Enhance

• src/lib/ingestion/sources/ - add more source adapters
• src/lib/ingestion/classify-match.ts - improve matching algorithm
• src/app/api/cron/ingest-resources/route.ts - ensure cron works
• vercel.json - add ingestion cron schedule

Ingestion Flow

1. Cron triggers /api/cron/ingest-resources
2. For each source: fetch raw data → normalize → classify (new vs possible duplicate)
3. Store in ingestion_candidates table with matchType and matchedResourceId
4. Admin reviews in /admin/ingestion → Approve (creates resource) or Reject
5. Approved candidates → inserted into resources with approved = true

Priority Sources

1. GitHub Student Pack (official partner list)
2. AWS / Azure / GCP student pages
3. Major IDE vendors (JetBrains, VS Code extensions)
4. Design tools (Figma, Canva, Framer, Sketch)
5. Productivity tools (Notion, Linear, Raycast, etc.)

Notes

• This reduces manual admin work for keeping directory fresh
• Complements community submissions (Phase 4)
• Can run more frequently than link-health cron