oss-crs-litellm healthcheck can fail before the CRS's litellm proxy finishes real-world startup, aborting the run
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 58/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- docker, docker-compose, python
- Domain
- ci-cd, devops, infrastructure
Research direction
Start in oss_crs/src/templates/run-crs-compose.docker-compose.yaml.j2 at the oss-crs-litellm healthcheck, then compare the offline branch and its LITELLM_LOCAL_MODEL_COST_MAP handling. Reproduce startup with a delayed network or equivalent slow initialization and verify that the Compose run remains healthy without masking genuine failures. Treat the separate Claude stderr observation as out of scope unless the issue is narrowed.
Written by the indexing model from the issue text.
Description
Summary
In crs-claude-code (and likely other CRSs using the internal litellm proxy), oss-crs-litellm's Docker healthcheck has a fixed budget of ~140 seconds (start_period: 90s + retries: 10 × interval: 5s) before Docker Compose marks it unhealthy and aborts the run with:
dependency failed to start: container oss-crs-litellm-1 is unhealthy
On a modest host (15GB RAM, otherwise idle), we repeatedly observed litellm genuinely still starting (not crashed, not resource-starved — confirmed via docker stats showing ample headroom) past that budget, because by default litellm attempts to fetch a remote model-cost map at startup (see the {% if offline %} branch in run-crs-compose.docker-compose.yaml.j2, which explicitly skips this fetch). When that fetch is slow (e.g. extra DNS/network hops in a Docker-outside-of-Docker CI setup), the healthcheck budget expires before litellm is actually ready, and the whole run fails.
Where
oss_crs/src/templates/run-crs-compose.docker-compose.yaml.j2, oss-crs-litellm service:
yaml
healthcheck:
test: ["CMD-SHELL", "python -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')""]
interval: 5s
timeout: 5s
retries: 10
start_period: 90s # ~140 Prisma migrations run on first boot
Evidence
Run 1 (no --offline): litellm container logs empty for the entire run; docker inspect ... .State.Health showed Connection refused continuously, FailingStreak climbing past 10, then "Status":"unhealthy" after ~4m13s — well beyond the ~140s budget. docker stats during this run showed litellm using only 1.17GB/8GB and the host had ~13GB free, ruling out resource starvation as the cause.
Run 2 (same host, oss-crs run --offline added): litellm started, generated its key, and served real POST /v1/messages calls to Anthropic successfully within the healthcheck budget. Run CRSs! completed in ~3 minutes instead of failing.
--offline's docstring ("Disables network access for CRS Compose operations") doesn't mention it also avoids this specific startup-time fetch, so this workaround isn't obvious from the CLI help.
Suggested fixes (any of)
Increase start_period/retries for oss-crs-litellm's healthcheck to tolerate slower networks (e.g. DooD-based CI runners).
Have litellm skip the remote model-cost-map fetch by default (or make LITELLM_LOCAL_MODEL_COST_MAP=True unconditional rather than gated behind --offline), since it appears to only affect cost reporting, not functionality.
Document that --offline is a practical workaround for slow-starting litellm in CI environments, until 1 or 2 lands.
Environment
Self-hosted GitLab CI, Docker executor, Docker-outside-of-Docker (shared host /var/run/docker.sock)
Host: 15GB RAM, otherwise idle during the run
oss-crs cloned from main at time of testing
Secondary observation (separate, undiagnosed)
After working around the above, we hit a second, different failure in the same pipeline: crs-claude-code's Claude Code agent process exits with code 1 ("Agent did not produce a patch"), but its stderr (/work/agent/claude_stderr.log) isn't surfaced to the container's stdout/the CLI's own logs, making it hard to diagnose from CI. Might be worth surfacing that file's contents (or at least its tail) in the CRS's own logging on failure.
- Dominant language
- Python
- Stars
- 187
- Forks
- 54
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 8
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ossf/oss-crs
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day
-
Docs: Add video presentation library to OSS-CRS documentationPossibly taken @subhechhamaiti716-byte claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
ossf/oss-crs#503 · 5 comments ·
Maintainers usually reply within 1 day
-
feat: validate target compatibility against CRS supported_target in crs.pyPossibly taken @SurbhiAgarwal1 claimed this 17 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
ossf/oss-crs#369 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
Integrate Google Mantis Skills as CRSsPossibly taken @nameetsinha7-hue claimed this 12 days ago. Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
ossf/oss-crs#337 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pearu/agent-sandbox#451 ·
Maintainers usually reply within 1 day
-
run_gate loads a custom tasks file from outside the working directoryPossibly taken @Ru0k3 claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
MakazhanAlpamys/Soup#3189 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
NYCPlanning/data-engineering#2996 ·
Maintainers usually reply within 1 day