Privacy & Data Governance Epic - PII Redaction, User Controls, and Vector Purge

🔎 Summary

Tighten privacy across the stack. This epic introduces client- and server-side PII redaction, export / delete-all controls, ephemeral “incognito” chats, and vector store purge by user. Goal: users keep full control of their data; the system stores only what’s needed—and stores it safely.

---

🎯 Goals

• No raw PII (emails, phones, addresses, IDs, credit cards, etc.) is persisted in MongoDB/Pinecone unless user chooses to save.
• One-click Export (JSON bundle) and Delete-All (MongoDB rows + Pinecone vectors) for any account.
• Incognito mode for chats: no DB writes, no vector upserts.
• Redaction before embedding so the vector DB never sees sensitive strings.
• TTL retention options per user (auto-delete after N days).

---

✅ Acceptance Criteria

• PII guardrail (server):

  • Incoming chat/query + documents are run through a deterministic PII scrubber before any persistence or embedding.
  • Redaction tokens (<EMAIL>, <PHONE>, <ADDR>, <SSN>, etc.) replace matches; original plaintext never leaves memory.

• Incognito mode:

  • POST /api/chat?mode=incognito streams an answer without saving messages or upserting vectors.
  • UI toggle “Incognito” persists per-session; visible badge in chat header.

• User controls (frontend + API):

  • Export my data: returns a ZIP of all conversations + metadata (/api/me/export).
  • Delete all data: hard-deletes user’s conversations in MongoDB and purges Pinecone namespace (/api/me/purge).
  • Retention setting: user can set retention_days (e.g., 0/7/30/90); a daily job removes expired data.

• Vector hygiene:

  • Pinecone operations are namespaced by userId (hashed).
  • Purge removes all vectors for that namespace within ≤ 5 minutes (async job status exposed).

• Observability & logs:

  • No logs contain PII; redaction verified via tests.
  • Metrics for export/purge counts, purge latency, redaction hit-rate.

---

🧱 Architecture / Design

• Redaction pipeline (server):

  • New utility redactPII(text) (fast regex + checksum validation; no LLM).
  • Apply in chat route before: saving messages, calling embedder, or writing to logs.
  • Apply in any document ingestion/store scripts.

• MongoDB:

  • Add user_settings collection: { userId, retentionDays, redactionEnabled, incognitoDefault }.
  • Add TTL index on messages.createdAt when retentionDays > 0 (or nightly sweeper if per-user TTL not feasible).

• Pinecone:

  • Namespace = sha256(userId).
  • New worker vectorPurgeJob(userIdHash) to delete-by-namespace; store job status in purge_jobs.

• Exports:

  • Server assembles a JSON bundle from conversations/messages + minimal metadata (no secrets), zips in memory, streams to client.

• Delete-All:

  • Transactional sequence: soft-lock user → Mongo deleteMany → enqueue Pinecone purge → finalize when job completes.

---

🔌 API Additions (documented in openapi.yaml)

• POST /api/chat?mode=incognito: stream response; skip persistence/embedding.
• GET /api/me/export: returns application/zip (auth required).
• POST /api/me/purge: enqueue delete-all, returns { jobId }.
• GET /api/me/purge/:jobId: returns purge job status.
• PUT /api/me/settings: update { retentionDays, redactionEnabled, incognitoDefault }.

Auth: existing JWT middleware. Rate-limit export/purge endpoints.

---

🖥 Frontend (client)

• Chat header: Incognito toggle (with tooltip); badge when active.

• Settings → Privacy:

  • Redaction toggle (default on).
  • Retention dropdown (0/7/30/90 days).
  • Buttons: Export my data, Delete all (2-step confirm + warning).

• Toasts & states: progress spinners for export download and purge status polling.

---

🔐 Security

• Never store raw PII in vectors or logs.
• Use hashed userId for namespaces (no direct identifiers).
• Strict CORS + HttpOnly; Secure; SameSite=Lax cookies if used.
• Limit export/purge frequency per user; audit entries (timestamps only, no payloads).

---

🧪 Testing

• Unit: redactPII() regexes (edge cases, false positives), TTL computation, namespace hashing.
• Integration: chat incognito flow (no DB writes), export JSON schema, purge job end-to-end.
• E2E: UI toggles, export download, purge confirmation, settings persistence.
• Snapshot: verify logs contain redacted tokens only.

---

📊 Observability

• Add counters: exports_total, purges_total, redactions_applied_total.
• Histogram: purge_latency_seconds.
• Alert on purge failures > 1% or redaction errors.

---

🧰 Ops / CI

• GitHub Actions: run redaction/test suite, forbid console.log of request bodies, secret scan.
• Nightly job: enforce retention (Mongo deletions + Pinecone cleanups for users with retentionDays > 0).

---

📦 Env Vars (new)

• PINECONE_NAMESPACE_SALT (for hashing userId)
• RETENTION_DEFAULT_DAYS=30
• EXPORT_MAX_MB=50
• PURGE_POLL_INTERVAL_MS=2000
• PRIVACY_FEATURES_ENABLED=1

---

📝 Tasks

• Server: implement redactPII() and integrate in chat + ingestion paths
• Server: incognito query mode (skip persistence & embeddings)
• Server: export route (+ ZIP stream), purge routes, purge job worker
• Server: user_settings model; retention sweeper; TTL strategy
• Pinecone: namespace hashing + delete-by-namespace job
• Client: privacy settings screen + chat incognito toggle
• OpenAPI: document new endpoints & models
• Tests: unit/integration/E2E + CI wiring
• Docs: README “Privacy & Data Controls” section
• Rollout: feature flag + staged enablement

---