No repair lever for a README that failed to resolve on first ingest

Open
#161 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
43/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
typescript
Domain
backend

Research direction

Read apps/registry/src/services/mcp-registry/ingest.ts and readme.ts, then trace how versions store README data and how fetchGithubReadme is called. The issue proposes a targeted repair for versions with no README and a GitHub repository, plus run report counters; done means these can be handled without downloading bundles or changing ingest watermark behavior.

Written by the indexing model from the issue text.

Description

A mirrored version's README is resolved exactly once, at first write, and never revisited.

versionIsCurrent (apps/registry/src/services/mcp-registry/ingest.ts:732-745) compares only artifact digests, so once a version is stored, every later run skips it before the wire. --full does not bypass this — there is no full field in IngestOptions; the flag only widens the since window. deleteMirror is reachable solely from the upstream-removal path (ingest.ts:382), so an operator has no lever either.

That is correct and deliberate for the download — it is what makes a nightly run cheap. It is wrong for the README, because the README resolution can fail transiently in a way the download cannot:

  • fetchOne (readme.ts) collapses 404, 429, 5xx, timeout, and DNS failure into the same null. A throttled response is indistinguishable from a genuine absence.
  • The fallback issues up to 10 sequential unauthenticated requests per README-less server (5 filenames × 2 refs). Across a backfill of ~305 servers, roughly half of which ship no in-bundle README, that is exactly the shape that draws throttling.

So a full backfill can permanently blank a subset of package pages, and nothing in the system will ever notice or retry.

Not fixed inline, deliberately

Two in-PR remedies were considered and rejected in review:

  • Gate versionIsCurrent on readme != null — re-downloads every README-less bundle nightly, forever, to retry a fetch that will usually keep returning nothing. Trades a rare permanent gap for a permanent recurring cost.
  • Add a full/refreshMetadata option threaded to the skip — net-new surface on the hot path for a repair used maybe twice a year.

Suggested shape

A separate, targeted repair that does not touch the ingest hot path: select versions where readme IS NULL and the package has a githubRepo, and run only fetchGithubReadme against them. No bundle download, no watermark interaction, no change to versionIsCurrent. Runnable on demand after a backfill, and cheap enough to schedule occasionally.

Worth pairing with a counter in the run report (readmesResolved / readmeFetchFailed) so a throttled backfill is visible at the time rather than discovered later as blank pages.

Context: #158 (which introduced README ingestion), #159, #160.

Dominant language
TypeScript
Stars
14
Forks
13
Avg merge
16m
Merged PRs (30d)
4

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NimbleBrainInc/mpak

All issues in NimbleBrainInc/mpak

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.