Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

P4: the worker advertises lease slots its memory budget can no longer admit, so turns get an in-turn tool error instead of a 503

Open
  • Is something wrong?
#452 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, typescript
Domain
backend, infrastructure

Research direction

Start with remote-worker/internal/vmpool/errors.go and microvm-worker/main.go to trace how memory-budget refusals and worker capacity are handled; then inspect the harness path that turns exec status into an HTTP response. The issue identifies mapping budget refusals to saturation as the smallest user-visible fix. Done means a refusal produces a 503 with Retry-After instead of an in-turn tool error.

Written by the indexing model from the issue text.

Description

enhancement

Found during the P6.3 acceptance (#425, Task 18 step 1) on the shared KVM rig.

What happened

A P6 host ran container sandboxes plus one P4 microvm-worker (moca_microvm_0), with MICROVM_MAX_COMMITTED_MB=8192. With the unit's 4 GiB SH_MEMORY_RESERVE_MB, that leaves a 4 GiB VM budget, about 14 VMs at 288 MiB each. SH_WORKSPACE_IDLE=8h and SH_STANDBY_DEPTH=2.

After about 12 P4 sessions, every turn of the next three sessions failed. Each exec came back as a tool error:

memory-budget: committing 301989888 more bytes would reach 4529848320 against a budget of 4294967296
(MaxCommittedBytes 8589934592 less MemoryReserveBytes 4294967296)

The model received that as the bash tool's output. The turns did not get a 503 with Retry-After. The sessions stayed in their tier, which is correct per P6.3.

Why
  • The worker's presence record advertises capacityMax = WORKER_MAX_CONCURRENT (16): concurrent exec slots.
  • The supervisor's lease cap uses that number. remote-worker/internal/vmpool/errors.go calls it "primary admission control", with MaxRuns and the memory budget as backstops. The wire contract has no "busy" frame, so a backstop refusal can only surface as an exec error.
  • The boot-time budget check (microvm-worker/main.go, around L128) sizes the budget for maxConcurrent × (1 + StandbyDepth) VMs.
  • A session's idle, resident workspace VM holds memory for SH_WORKSPACE_IDLE (8 h here), yet it uses no lease slot. So the memory a worker has committed grows with the number of distinct recent sessions, not with concurrency. The worker keeps advertising free slots it can no longer admit.
  • P6.3's affinity keeps each session on its sandbox, so all of a session's turns reuse its resident VM. That is right for the workspace, but it means a busy P4 host reaches this state sooner.
What a user sees

New P4 sessions fail with a tool error in the transcript, on a host that reports spare capacity. Nothing retries, and nothing says "at capacity".

Options
  1. Report the real headroom. The worker lowers its effective capacity, or marks itself saturated, when the memory budget cannot admit one more VM. That needs a presence update; Hello is sent once, so this needs either a capacity update frame or a re-Hello.
  2. Make a refusal retryable. Map a budget refusal (RefuseMemoryBudget, or max-runs) to an exec status the harness classifies as saturated, so the turn answers 503 with Retry-After instead of handing the error text to the model.
  3. Evict under pressure. Reclaim the least-recently-used idle workspace VM (it is persisted on disk under SH_WORKSPACE_ROOT) before refusing.
  4. Document it. In P4-ON-P6.md, size MICROVM_MAX_COMMITTED_MB and SH_WORKSPACE_IDLE for the expected number of distinct sessions, not for concurrency.

(2) is the smallest user-visible fix. (3) stops a host's capacity from leaking away over time.

Repro

On a host with a small budget, create more distinct P4 sessions than the budget holds VMs, all within SH_WORKSPACE_IDLE. On the rig, the session after about 14 committed VMs failed. Raising the budget to 12288 MB let three more sessions pass.

Dominant language
TypeScript
Stars
4
Forks
9
Avg merge
9h 31m
Merged PRs (30d)
139

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from rossoctl/moca

All issues in rossoctl/moca

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.