Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Docs and operating-model requests 504: content hydration crawls 2,179 GitHub blobs serially inside a 30s Lambda

未关闭
  • 哪里不对？
#230 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
aws, github, typescript
领域
backend, cloud, observability, performance

调研方向

Start with backend/src/docs/githubStore.ts and trace how ContentsApiGithubStore.sync() is called by operatingModel/loader.ts. Reproduce the direct docs endpoint timeout, then choose and scope a hydration approach that completes within the Lambda budget and makes GitHub requests fail fast. Also inspect seedRuntimeUsers for the existing-user role backfill; done means docs and operating-model endpoints recover, hangs return promptly, and missing roles cannot recur.

由索引模型根据 Issue 内容生成。

描述

Symptom

The portal Today page shows "Process documents are unavailable — HTTP 504", an empty action queue, and My Plan / operating-model data missing. Work API calls were also failing (fixed separately, see below).

Diagnosis (verified against production 2026-09-26)

  • GET /docs/process-quality invoked directly on the backend Lambda returns Sandbox.Timedout after 30.00s; /health on the same container returns 200 in ~2s. CloudFront surfaces the timeout as 504.
  • Every docs/operating-model request calls ContentsApiGithubStore.sync() (backend/src/docs/githubStore.ts), which on a cold container hydrates the whole content/ tree one GitHub API call per blob, serially, with no fetch timeout.
  • DataTalksClub/dataops-knowledge content/ currently holds 2,179 blobs (443 md, 1,516 jpg, 220 png). At ~0.3–0.5s per blobs call that is ~10–15 minutes of work inside a 30s Lambda, so hydration never completes in one invocation. Each new container restarts the crawl (progress persists only in per-container /tmp), so no container ever becomes warm.
  • Blast radius: /docs/*, /search, /api/operating-model, /api/my-plan, and the Today page's process-quality call (operatingModel/loader.ts calls the same store.sync()).
  • Side damage: each doomed crawl burns up to ~2,200 GitHub requests; the knowledge token's rate limit already showed 1,301/5,000 used within one hour.

Work API 403s (fixed in data on 2026-09-26, needs a repo-level guard)

  • The #164 role gate denies users without a supported role. The three live user items predate the role attribute (created 2026-06-28, no role), and seedRuntimeUsers skips existing users as "unchanged", so the role was never backfilled. Every team read/work mutation returned 403 ("Team access requires an active admin or operator role").
  • Applied the declared seed state (role: 'admin' for grace/valeriia/alexey) directly to dataops-v1-users via UpdateItem; /api/cards and /api/tasks verified 200 after.
  • Follow-up needed: make seedRuntimeUsers update missing attributes on existing users (or migrate in place), so this cannot recur. Data migration was done by hand this time only because the gate shipped without it.

Fix directions (for grooming)

  1. Stop eager per-blob hydration. Candidates: hydrate only .md eagerly and fetch images lazily via ensureFile; or download a single tarball and extract content/; or sync the knowledge repo to S3 from CI and hydrate from S3 in one/few calls.
  2. Add an explicit timeout (AbortSignal.timeout) to request()/fetchImpl in githubStore.ts so a GitHub hang fails fast with 502 instead of eating the whole Lambda budget.
  3. Consider a startup/deploy-time hydration step (offline snapshot in the artifact) instead of on-demand crawling.

Related observability gap

Backend Lambda log streams stop on 2026-08-11 even though the function is actively invoked (CloudWatch metrics show hundreds of invocations/day; direct invokes today produced no streams). The execution role still has AWSLambdaBasicExecutionRole attached. Root cause unknown — needs its own investigation; until fixed, production debugging is blind (this diagnosis had to be reproduced by direct invokes).

主要语言
TypeScript
星标
2
派生
0
30 天内没有已合并 PR

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

DataTalksClub/dataops 的其他 Issue

查看 DataTalksClub/dataops 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。