oss-crs-litellm healthcheck can fail before the CRS's litellm proxy finishes real-world startup, aborting the run

Open
#367 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@shreyaabaranwal is already working on this.

Since Oct 27, 2026.

  • #441 by @shreyaabaranwal — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
58/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, docker-compose, python

Research direction

Start in oss_crs/src/templates/run-crs-compose.docker-compose.yaml.j2 at the oss-crs-litellm healthcheck, then compare the offline branch and its LITELLM_LOCAL_MODEL_COST_MAP handling. Reproduce startup with a delayed network or equivalent slow initialization and verify that the Compose run remains healthy without masking genuine failures. Treat the separate Claude stderr observation as out of scope unless the issue is narrowed.

Written by the indexing model from the issue text.

Description

Summary

In crs-claude-code (and likely other CRSs using the internal litellm proxy), oss-crs-litellm's Docker healthcheck has a fixed budget of ~140 seconds (start_period: 90s + retries: 10 × interval: 5s) before Docker Compose marks it unhealthy and aborts the run with:

dependency failed to start: container oss-crs-litellm-1 is unhealthy

On a modest host (15GB RAM, otherwise idle), we repeatedly observed litellm genuinely still starting (not crashed, not resource-starved — confirmed via docker stats showing ample headroom) past that budget, because by default litellm attempts to fetch a remote model-cost map at startup (see the {% if offline %} branch in run-crs-compose.docker-compose.yaml.j2, which explicitly skips this fetch). When that fetch is slow (e.g. extra DNS/network hops in a Docker-outside-of-Docker CI setup), the healthcheck budget expires before litellm is actually ready, and the whole run fails.

Where

oss_crs/src/templates/run-crs-compose.docker-compose.yaml.j2, oss-crs-litellm service:

yaml
healthcheck:
test: ["CMD-SHELL", "python -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')""]
interval: 5s
timeout: 5s
retries: 10
start_period: 90s # ~140 Prisma migrations run on first boot
Evidence
Run 1 (no --offline): litellm container logs empty for the entire run; docker inspect ... .State.Health showed Connection refused continuously, FailingStreak climbing past 10, then "Status":"unhealthy" after ~4m13s — well beyond the ~140s budget. docker stats during this run showed litellm using only 1.17GB/8GB and the host had ~13GB free, ruling out resource starvation as the cause.
Run 2 (same host, oss-crs run --offline added): litellm started, generated its key, and served real POST /v1/messages calls to Anthropic successfully within the healthcheck budget. Run CRSs! completed in ~3 minutes instead of failing.

--offline's docstring ("Disables network access for CRS Compose operations") doesn't mention it also avoids this specific startup-time fetch, so this workaround isn't obvious from the CLI help.

Suggested fixes (any of)
Increase start_period/retries for oss-crs-litellm's healthcheck to tolerate slower networks (e.g. DooD-based CI runners).
Have litellm skip the remote model-cost-map fetch by default (or make LITELLM_LOCAL_MODEL_COST_MAP=True unconditional rather than gated behind --offline), since it appears to only affect cost reporting, not functionality.
Document that --offline is a practical workaround for slow-starting litellm in CI environments, until 1 or 2 lands.
Environment
Self-hosted GitLab CI, Docker executor, Docker-outside-of-Docker (shared host /var/run/docker.sock)
Host: 15GB RAM, otherwise idle during the run
oss-crs cloned from main at time of testing
Secondary observation (separate, undiagnosed)

After working around the above, we hit a second, different failure in the same pipeline: crs-claude-code's Claude Code agent process exits with code 1 ("Agent did not produce a patch"), but its stderr (/work/agent/claude_stderr.log) isn't surfaced to the container's stdout/the CLI's own logs, making it hard to diagnose from CI. Might be worth surfacing that file's contents (or at least its tail) in the CRS's own logging on failure.

Dominant language
Python
Stars
187
Forks
54
Avg merge
4d 19h
Merged PRs (30d)
8

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ossf/oss-crs

All issues in ossf/oss-crs

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.