Cross-Source Deduplication & Topic Clustering (Timelines + Newsletter Top Stories)

🔎 Summary

Reduce noise and improve signal by automatically grouping near-duplicate articles from multiple sources into topic clusters. Each cluster shows a canonical headline, AI-fused summary, key entities, and a timeline of updates. Search/list pages collapse duplicates; the newsletter highlights top clusters (not every single article).

---

💡 Why

• Current feed can repeat the same story across sources.
• Curators want one view per story with attribution to all sources, plus how it evolved over time.
• Newsletters should surface top clusters instead of spamming multiple near-duplicates.

---

✅ Acceptance Criteria

• Clustering

  • New articles are assigned to an existing cluster or create a new one within a rolling window (e.g., 7–14 days).
  • Duplicate/near-duplicate articles are collapsed on list/search pages.
  • Each cluster stores: canonical title, fused AI summary, list of articles (URLs, sources), entities, first/last seen timestamps.

• Timelines

  • Cluster detail page shows chronological timeline of significant updates (e.g., first report → official statement → policy action).

• Frontend

  • /clusters index: paginated clusters with canonical headline, sources count, and last update time.
  • Article list/search view: duplicates collapsed into cluster cards (badge: “+4 sources”).
  • Cluster detail: timeline, entities, per-source snippets, and link-outs.

• Newsletter

  • Daily email shows Top N clusters (ranked) with canonical summary and “sources included.”

• API

  • REST endpoints exist to query clusters, timelines, and map article→cluster.

• Perf

  • Incremental clustering under < 300ms p95 per article (excluding AI calls).

• Reliability

  • If AI services rate-limit, system falls back to non-AI similarity and queues summary fusion.

---

🧠 Approach (Similarity & Fusion)

1. Text normalization: strip boilerplate, remove tracking params, normalize quotes/whitespace.

2. Similarity (multi-stage, fast→smart):

   • Stage A: MinHash/LSH on 5-gram shingles for near-duplicate detection (fast, cheap).
   • Stage B: TF-IDF cosine on normalized title+lead for borderline cases.
   • Stage C (fallback/upgrade): LLM embeddings or cross-encoder similarity (queued) to refine clusters.

3. Canonical headline selection: pick the earliest reputable or longest informative title.

4. AI-fused summary: prompt Gemini to produce a concise, de-duplicated summary citing differences across sources.

5. Entity extraction: use NER (simple regex + AI fallback) for people, orgs, places, and policy areas.

---

🗄 Data Model (MongoDB / Mongoose)

• articles (existing; unchanged fields kept)

  • add: clusterId?: ObjectId, normalizedTitle, normalizedLead, signatures: { minhash: string, tfidf?: string }, entities?: { persons:[], orgs:[], places:[], topics:[] }
  • indexes: { clusterId: 1 }, { createdAt: -1 }

• clusters (new)

  • _id
  • canonicalTitle: string
  • summary: string (AI-fused)
  • entityBag: { persons:[], orgs:[], places:[], topics:[] }
  • articleIds: ObjectId[]
  • sourceCounts: Record<string, number>
  • firstSeen: Date, lastUpdated: Date
  • quality: { coherence: number, size: number }
  • indexes: { lastUpdated: -1 }, { firstSeen: -1 }, { size: -1 }

• cluster_events (optional; for timeline)

  • _id, clusterId, articleId, ts, kind: 'first_report'|'official'|'update'|'analysis', note?

---

🔌 API Endpoints (Express in Next.js)

• GET /api/clusters?page&limit&since → list clusters (paginated, newest first)
• GET /api/clusters/:id → cluster detail (articles, timeline, entities)
• GET /api/articles/:id/cluster → resolve an article’s cluster
• POST /api/admin/clusters/rebuild (protected) → rebuild clusters over a window
• POST /api/admin/clusters/merge (protected) → merge two cluster IDs
• POST /api/admin/clusters/split (protected) → split by article IDs

All reads are public; admin routes gated by JWT role (e.g., admin).

---

🕸 Crawler Changes

• After extraction, compute MinHash signature + basic TF-IDF vector.
• Call POST /internal/cluster/ingest (new internal route) to assign the article to a cluster.
• On 403/dynamic pages, Puppeteer path still applies; normalization runs after content resolution.

---

🧭 Frontend (Next.js + Tailwind + Shadcn)

• Routes

  • /clusters (index)
  • /clusters/[id] (detail: timeline + sources list)

• Components

  • ClusterCard (canonical title, sources count, key entities, lastUpdated)
  • ClusterTimeline (events grouped by day; expandable)
  • EntityChips (pill badges, filterable)
  • CollapsedSources (hover to preview per-source headlines)

• Article List/Search

  • Replace repeated items with a single ClusterCard + “View sources”
  • Toggle: “Show all sources” (expands inline)

---

📨 Newsletter (Resend)

• Group stories by cluster.
• Ranking: size (sources) + recency decay + source diversity.
• Section: Top N clusters today with canonical summary and “Includes: SourceA, SourceB, …”
• Fallback to per-article if clustering queue hasn’t processed (mark as “(early)”).

---

🧱 Infra / Caching / Rate Limits

• Redis: cache GET /api/clusters* for 60–120s; key by query params.

• Background worker (Vercel CRON every 10–15 min) to:

  • Re-score clusters
  • Generate/refresh AI-fused summaries for large clusters
  • Extract entities for new articles

• Respect Gemini rate limits via token bucket + queue; exponential backoff on 429/5xx.

---

🔐 Security / Roles

• Public reads; admin write ops (merge/split/rebuild).
• Internal ingest route authenticated via service token (crawler→backend).
• Sanitize HTML and strip scripts in any stored snippet.

---

📊 Observability & Metrics

• Dashboards

  • Cluster count/day, avg cluster size, duplicate-collapse rate
  • AI summary latency, error rates, retry counts
  • Newsletter CTR per cluster

• Alerts

  • Spike in unassigned articles
  • AI failures > X% over 10 min

---

🧪 Testing

• Unit: MinHash/TF-IDF similarity thresholds; entity extraction; canonical title picker.
• Integration: Ingest flow → cluster assignment; merge/split behavior; Redis caching.
• E2E (Playwright): clusters index loads; search collapses duplicates; timeline renders; newsletter sample groups by cluster.
• Load: Ingest 5k articles/hour maintains p95 < 300ms for assignment (without AI).

---

🗺 Rollout Plan

1. Ship clustering with MinHash + TF-IDF only (AI fusion behind flag).
2. Add clusters index/detail routes + collapsed search.
3. Enable newsletter grouping (A/B: grouped vs non-grouped).
4. Turn on AI-fused summaries for top clusters only → gradually expand.

---

⚠️ Risks & Mitigations

• Over-merging (false positives): conservative thresholds; manual merge/split tools.
• AI cost/latency: batch & cache; only summarize clusters over size≥2 or trending.
• Vercel limits: keep heavy ops in scheduled jobs; store pre-computed summaries.

---

📋 Tasks

• DB: clusters, optional cluster_events; indexes
• Article schema: clusterId, signatures, entities
• Similarity lib: MinHash/LSH + TF-IDF helper
• Ingest route (internal) + assignment logic
• Admin: rebuild/merge/split endpoints + minimal UI
• Frontend: /clusters, detail page, collapsed search results
• AI summary fusion + entity extraction jobs (queued)
• Newsletter: render grouped clusters + ranking
• Redis cache for cluster reads
• Metrics + logs + alerts
• Tests: unit/integration/E2E + load scripts
• Docs: README sections + screenshots

---