make `--upstream_sync` also sync the gap analysis graph progressively

Status

Maintainer-approved design — opportunistic gap-analysis (GA) fetch from production on access, replacing the prior idea of bulk-downloading the full ~9 GB GA graph via --upstream_sync. Implementation spec will land in a follow-up PR; this issue tracks the feature and acceptance criteria.

Supersedes bulk GA download as the primary local-dev path. Prateek's JIT Neo4j compute path remains an advanced fallback documented separately (not the default UI/REST path).

---

Problem

The gap-analysis cache is very large (~9 GB). Downloading it wholesale during --upstream_sync is unreliable and unnecessary for most local development: contributors typically need only the GA pairs they actually view.

Production (opencre.org) already serves precomputed GA rows from Postgres (cache-only; cache miss → 404). Local instances should reuse that upstream cache opportunistically instead of requiring a full bulk sync.

---

Proposed solution

When a user requests a GA pair (via UI or local REST GA API — same backend code path):

1. If the pair exists in the local GA store, serve it.
2. If not, and auto-fetch is enabled, fetch the full material row from prod (read-only HTTP client), upsert into the local GA store, then serve it.
3. While a fetch is in flight for a pair, return a "download in progress" state; concurrent requests for the same pair share a single in-flight fetch (frontend polls/waits).
4. On prod 404 (pair not in prod cache): show documented instructions for local backfill (links to opencre.org and/or GitHub docs — to be written as part of this work). Do not compute GA in the request path from the UI.
5. On prod unreachable or fetch failure: fall through to local cached row if present; otherwise show Try again (not local compute in UI).

GA auto-fetch is independent of --upstream_sync CRE graph sync. Locally cached pairs are preserved and not overwritten by a failed fetch.

---

Design decisions

1. Trigger: Both UI and local REST GA API use the same backend code path.
2. Prod URL: Configurable env var (default https://opencre.org). Prod down / miss behavior:
   • Prod unreachable (timeout, network error, exhausted retries): serve local cached row if it exists.
   • Not in local cache: show link to compute/backfill instructions (no UI compute).
   • Do not silently compute on prod-down in the UI/request path. Local compute remains CI/docs-only (advanced JIT Neo4j fallback), not a hidden fallback when prod is down.
3. Persist: Fetch the full material row from prod; upsert into the local GA store (gap_analysis_results / equivalent).
4. Concurrency: Single fetch per pair; backend returns "download in progress" while in-flight; frontend polls/waits.
5. 404 miss: Exact UX copy plus links to local backfill documentation (opencre.org and/or GitHub). Block local compute from UI for now.
6. Instructions: Link to docs to be written as part of this work — do not prescribe specific Makefile targets in this ticket.
7. Soft message: When both constituent standards exist locally and prod confirms they are valid GA participants, but the specific pair is missing on prod, show a soft CTA to file a ticket after the backfill instructions. Detection logic (all must hold):
   • Both standards exist locally in the CRE graph (nodes present for the pair's two standard names).
   • Both are valid GA participants — either:
     • the standards list endpoint includes both standards as GA-eligible, or
     • individual GET per standard succeeds against prod (or local graph metadata when prod standards list is unavailable).
   • GA request for that specific pair returns 404 or an error from prod (after retries exhausted).
   • When all three conditions hold, append the soft "file a ticket" CTA; otherwise show only the standard backfill instructions.
8. Refresh: Backend owns fetch lifecycle; frontend refresh sees the same state. Log when a local instance pulls a pair from prod.
9. Env disable: CRE_GA_NO_UPSTREAM=1 (or similar name) documented in .env.example — disables auto-fetch. On fetch failure: Try again button (not local calc in UI).
10. Independence: GA auto-fetch is independent of --upstream_sync CRE graph sync. Locally cached pairs are preserved (not overwritten by failed fetch).
11. Security: No auth on prod fetch; read-only HTTP client to prod GA endpoints; no writes to prod. State explicitly in implementation spec.
12. Configurable prod fetch (env vars): GA upstream HTTP client uses CRE_GA_UPSTREAM_* env vars, scoped parallel to the existing CRE_UPSTREAM_* pattern from upstream CRE graph sync (#951):
    • CRE_GA_UPSTREAM_TIMEOUT_SECONDS — default 30 (Heroku-typical; mirrors CRE_UPSTREAM_TIMEOUT_SECONDS)
    • CRE_GA_UPSTREAM_MAX_ATTEMPTS — default 4 (mirrors CRE_UPSTREAM_MAX_ATTEMPTS)
    • CRE_GA_UPSTREAM_RETRY_BACKOFF_SECONDS — default 2 (mirrors CRE_UPSTREAM_RETRY_BACKOFF_SECONDS; optional but recommended for parity)
    • Exhaust retries before surfacing Try again in UI.

Cache policy

• V1: Once cached locally, always serve from local cache — no automatic re-fetch from prod on subsequent requests.
• V2 (future): Add TTL-based re-fetch from prod (e.g. env-controlled staleness window). Document as Version 2; out of scope for v1 implementation.

Open items (TBD — do not block implementation)

Item	Notes
V2 TTL / force-refresh	Deferred to Version 2 (see Cache policy)

---

Out of scope

• Bulk download of the full GA graph via --upstream_sync as the primary path (superseded by this design).
• Computing GA on production (prod remains cache-only).
• Computing GA in the UI/request path on cache miss (blocked; backfill via docs/CLI only).
• Prescribing exact Makefile targets in this ticket (docs PR will cover operational steps).
• JIT Neo4j compute as the default path (advanced fallback in docs only).
• V2: TTL-based automatic re-fetch from prod for locally cached pairs (see Cache policy).

---

Acceptance criteria

• UI and local REST GA API share one backend fetch/cache code path.
• Configurable prod base URL (default opencre.org); read-only prod HTTP client (no auth, no writes).
• On cache miss with auto-fetch enabled: fetch full material row from prod and upsert locally.
• V1 cache policy: once a pair is cached locally, subsequent requests serve from local cache without automatic prod re-fetch.
• On prod unreachable (after CRE_GA_UPSTREAM_MAX_ATTEMPTS retries): serve local cached row if present; otherwise Try again with compute/backfill instructions link (no UI-triggered compute).
• On prod 404 for a pair not in local cache: documented backfill instructions (opencre.org + GitHub links); no local compute from UI.
• Single in-flight fetch per pair; concurrent requests get "download in progress"; frontend polls/waits.
• Soft CTA to file a ticket when soft-message detection (#7) conditions are met (both standards local + valid GA participants + prod pair 404/error).
• CRE_GA_NO_UPSTREAM=1 (or agreed name) disables auto-fetch; documented in .env.example.
• CRE_GA_UPSTREAM_TIMEOUT_SECONDS default 30; CRE_GA_UPSTREAM_MAX_ATTEMPTS default 4; documented in .env.example (and implementation spec).
• Failed fetch does not overwrite existing local cached row.
• Logging when local instance pulls a pair from prod.
• Tests: mocked prod HTTP, cache upsert, in-progress lock, 404 UX copy, env disable flag, read-only prod client behavior, logging, env defaults (CRE_GA_UPSTREAM_TIMEOUT_SECONDS, CRE_GA_UPSTREAM_MAX_ATTEMPTS), soft-message detection logic (positive/negative cases), V1 no re-fetch (cached pair served without second prod call).

---

Dependencies

• Production GA REST API serving precomputed material rows (existing on opencre.org).
• Local GA store schema (gap_analysis_results / equivalent) supporting upsert of material rows.
• Documentation PR: local backfill instructions, env vars, advanced JIT Neo4j fallback (not default).
• Detailed implementation spec PR (referenced from this issue).

---

Note

A detailed implementation spec will land in a follow-up PR linked from this issue. Volunteers should read this body (and the spec PR when available) before claiming the issue.