Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Docs and operating-model requests 504: content hydration crawls 2,179 GitHub blobs serially inside a 30s Lambda

Open
  • Is something wrong?
#230 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
aws, github, typescript
Domain
backend, cloud, observability, performance

Research direction

Start with backend/src/docs/githubStore.ts and trace how ContentsApiGithubStore.sync() is called by operatingModel/loader.ts. Reproduce the direct docs endpoint timeout, then choose and scope a hydration approach that completes within the Lambda budget and makes GitHub requests fail fast. Also inspect seedRuntimeUsers for the existing-user role backfill; done means docs and operating-model endpoints recover, hangs return promptly, and missing roles cannot recur.

Written by the indexing model from the issue text.

Description

Symptom

The portal Today page shows "Process documents are unavailable — HTTP 504", an empty action queue, and My Plan / operating-model data missing. Work API calls were also failing (fixed separately, see below).

Diagnosis (verified against production 2026-09-26)

  • GET /docs/process-quality invoked directly on the backend Lambda returns Sandbox.Timedout after 30.00s; /health on the same container returns 200 in ~2s. CloudFront surfaces the timeout as 504.
  • Every docs/operating-model request calls ContentsApiGithubStore.sync() (backend/src/docs/githubStore.ts), which on a cold container hydrates the whole content/ tree one GitHub API call per blob, serially, with no fetch timeout.
  • DataTalksClub/dataops-knowledge content/ currently holds 2,179 blobs (443 md, 1,516 jpg, 220 png). At ~0.3–0.5s per blobs call that is ~10–15 minutes of work inside a 30s Lambda, so hydration never completes in one invocation. Each new container restarts the crawl (progress persists only in per-container /tmp), so no container ever becomes warm.
  • Blast radius: /docs/*, /search, /api/operating-model, /api/my-plan, and the Today page's process-quality call (operatingModel/loader.ts calls the same store.sync()).
  • Side damage: each doomed crawl burns up to ~2,200 GitHub requests; the knowledge token's rate limit already showed 1,301/5,000 used within one hour.

Work API 403s (fixed in data on 2026-09-26, needs a repo-level guard)

  • The #164 role gate denies users without a supported role. The three live user items predate the role attribute (created 2026-06-28, no role), and seedRuntimeUsers skips existing users as "unchanged", so the role was never backfilled. Every team read/work mutation returned 403 ("Team access requires an active admin or operator role").
  • Applied the declared seed state (role: 'admin' for grace/valeriia/alexey) directly to dataops-v1-users via UpdateItem; /api/cards and /api/tasks verified 200 after.
  • Follow-up needed: make seedRuntimeUsers update missing attributes on existing users (or migrate in place), so this cannot recur. Data migration was done by hand this time only because the gate shipped without it.

Fix directions (for grooming)

  1. Stop eager per-blob hydration. Candidates: hydrate only .md eagerly and fetch images lazily via ensureFile; or download a single tarball and extract content/; or sync the knowledge repo to S3 from CI and hydrate from S3 in one/few calls.
  2. Add an explicit timeout (AbortSignal.timeout) to request()/fetchImpl in githubStore.ts so a GitHub hang fails fast with 502 instead of eating the whole Lambda budget.
  3. Consider a startup/deploy-time hydration step (offline snapshot in the artifact) instead of on-demand crawling.

Related observability gap

Backend Lambda log streams stop on 2026-08-11 even though the function is actively invoked (CloudWatch metrics show hundreds of invocations/day; direct invokes today produced no streams). The execution role still has AWSLambdaBasicExecutionRole attached. Root cause unknown — needs its own investigation; until fixed, production debugging is blind (this diagnosis had to be reproduced by direct invokes).

Dominant language
TypeScript
Stars
2
Forks
0
No merged PRs in 30d

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DataTalksClub/dataops

All issues in DataTalksClub/dataops

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.