Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

A dead embedded Lemonade's leftover state file hijacks every client to a port with nothing on it

Open
  • Is something wrong?
#4,281 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai, backend

Research direction

Start in src/gaia/llm/lemonade_client.py at _embedded_lemonade_url() and _read_embedded_lemonade_state(), then compare the process checks in src/gaia/llm/lemonade_embedded.py, especially EmbeddedLemonade.status(). Add a regression test for a state.json with a dead pid and verify the workflow dials the URL prepared by its setup step.

Written by the indexing model from the issue text.

Description

bug

A dead embedded Lemonade keeps redirecting every GAIA client to its old port, so a perfectly healthy server on the default port looks unreachable.

Start gaia lemonade embedded, then have it die — killed, crashed, machine rebooted. The state file it left behind still names the port it once bound, and every client reads that file ahead of the default. From then on gaia llm, the Agent SDK, and the API server all dial a port with nothing on it and report Lemonade as down, while the real server answers on 13305. Nothing tells the user why; the only cure is knowing to delete a file they have never heard of.

This is currently red in CI. Test Agent SDK on Windows (Lemonade Integration) health-checks Lemonade on 13305, passes, and then every test in the run fails connecting to port 59939 — a stale embedded port on the persistent self-hosted runner. It has failed the same way on at least three unrelated branches (kalin/ui-design-language, claude/ui-tick-session-lock, claude/ui-fresh-cancel-event), so it is not any one PR's doing.

Suggested fix: don't let the state file win unless its process is actually alive. The embedded manager already knows how to decide this; the client just doesn't ask.

🔍 Technical details

_embedded_lemonade_url() (src/gaia/llm/lemonade_client.py:66) returns http://localhost:<port> straight from ~/.gaia/lemonade/state.json whenever the port parses as an integer. _read_embedded_lemonade_state() (:81) validates only that the port is an int in 1..65535 — no pid check, no health probe, no timestamp — and _get_lemonade_config() (:114) uses that in preference to DEFAULT_LEMONADE_URL whenever LEMONADE_BASE_URL is unset.

The asymmetry is the bug: state.json already records a pid (lemonade_embedded.py:964), and EmbeddedLemonade.status() (:765) does the full job — health probe, then _daemon_alive(pid), and _clear_state() when the state is incomplete. The client-side reader bypasses all of it.

Two candidate fixes:

  • Have _read_embedded_lemonade_state() require _daemon_alive(state["pid"]) before returning the state, falling back to DEFAULT_LEMONADE_URL otherwise.
  • Or have the embedded manager clear state.json on a non-graceful exit, which it cannot do for a SIGKILL/reboot — so the reader-side check is the one that actually closes the hole.

Evidence from run 36085092147:

Lemonade already healthy on port 13305 -- reusing persistent server.
⏳ Waiting for LLM server at http://localhost:13305...
✅ Server health check passed
...
LemonadeClientError: Failed to load model 'Gemma-4-E4B-it-GGUF' on http://localhost:59939/api/v1:
  [WinError 10061] No connection could be made because the target machine actively refused it

Unblocking CI in the meantime means deleting C:\Users\Administrator\.gaia\lemonade\state.json on actions-runner-01, or exporting LEMONADE_BASE_URL in the workflow so the stale file cannot win.

A regression test wants both halves: a unit test that a state.json naming a dead pid does not change the resolved base URL, and a workflow-level assertion that the URL the tests dial is the one the setup step readied.

Dominant language
Python
Stars
1.6k
Forks
169
Avg merge
2d 19h
Merged PRs (30d)
511

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from amd/gaia

All issues in amd/gaia

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.