bug(infinity-daemon): daemon restart leaves stale pid/socket files behind on failed startup; daemon never recovers without manual cleanup

Open
  • Is something wrong?
#190 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
backend, cli

Research direction

Start by locating the infinity-daemon startup and stop/restart entry points, then trace where TCP binding and daemon.log, daemon.sock, and daemon.pid changes occur. Reproduce the stated stale-pid scenario and inspect the existing daemon tests. Done means a failed startup leaves the running daemon’s management files and log intact, and stop/restart detect stale or mismatched daemon state without spawning a competing daemon.

Written by the indexing model from the issue text.

Description

Summary

When a daemon start fails partway (e.g. TCP port already bound), the failed run has already truncated daemon.log, replaced daemon.sock, and rewritten daemon.pid with its own (now dead) PID. From then on the installation is wedged: restart/stop trust the dead pidfile, new unix-socket clients get ECONNREFUSED, and the previously-running daemon is orphaned from management. Nothing heals itself; recovery today requires manually killing leftovers and deleting the files. The daemon should clean up after a failed start and verify liveness beyond the pidfile.

Environment

  • macOS (Darwin arm64)
  • infinity --version reports infinity-agent-cli 0.2.0
  • ~/.infinity/providers.json / ~/.infinity/rap.json configured with providers, sandbox, rap, and mcp

What I observed

  1. A running daemon (PID 79449, up since 08:12) with both model providers serving.
  2. ~/.infinity/daemon.pid contained 79814 — no such process.
  3. ~/.infinity/daemon.sock dated 08:32 (20 min after the daemon started). Raw connect() to it → ECONNREFUSED.
  4. infinity daemon restart → daemon is not running — starting a fresh one → Error: daemon exited during startup (exit status: 1) / Address already in use (os error 48) — the live daemon still held the TCP port.
  5. daemon.log was 0 bytes (truncated at 08:32), and stayed empty — all forensic evidence of whatever happened was wiped by the failed start.

Manual recovery (what I had to do): kill the leftover daemon + orphaned provider processes, delete the stale socket/pidfile, infinity daemon restart — after which pidfile matched the new PID, both providers served, unix socket connected, and websocket Welcome arrived with the connection staying open.

Repro (deterministic, no crash needed)

  infinity daemon restart          # healthy daemon A running, pidfile=A
  echo 999999 > ~/.infinity/daemon.pid   # simulate pid desync
  infinity daemon restart
  # → "daemon is not running — starting a fresh one"
  # → new daemon B truncates daemon.log, recreates daemon.sock,
  #   writes pidfile=B, then dies on TCP bind (A still holds it)
  # End state: socket file + pidfile belong to dead B; clients get
  # ECONNREFUSED; daemon A is orphaned from `stop`/`restart`;
  # pre-existing log content is gone.

Suggested behavior

  1. Bind first, mutate files second. On startup, attempt the TCP bind (and any other exclusive resource) before truncating daemon.log or replacing daemon.sock / daemon.pid. If the port is taken, abort with another daemon appears to be running and touch nothing.
  2. Don't leave corpses on failed startup. If the daemon exits during startup after creating files, remove the ones it created (at minimum a pidfile pointing at a PID that will never live again).
  3. Verify liveness, not just pid existence. stop/restart should confirm the pidfile PID is actually this daemon (e.g. kill(pid, 0) plus a command-line check, or better: probe the socket/port it should hold) and say plainly when management state and reality disagree, instead of "not running" → spawning a rival.
  4. Consider single-instance locking (e.g. flock on a lock file in ~/.infinity) so a second daemon can never clobber the first's files.

Related observation (separate, possibly its own issue)

daemon.log is truncated on every (re)start, including failed starts — which is what erased the evidence here. Log rotation (daemon.log.1) or append-only startup would preserve crash forensics. Happy to file separately if you prefer.

Dominant language
Rust
Stars
716
Forks
81
Avg merge
1d 18h
Merged PRs (30d)
11

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hydro-project/infinity

All issues in hydro-project/infinity

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.