RFC: hugepage/memory fit admission for best-of-K placement (minimal viable change)

Start in packages/api/internal/orchestrator/placement/placement_best_of_K.go, reading sample(), chooseNode(), the existing memoryScore accounting, and BestOfKConfig. Review the proposed placement unit cases for capacity filtering, no-pool nodes, all-full sets, and the K-sampling fallback; done means the agreed admission behavior, configuration and error handling are covered by tests, with the open rollout and widening questions resolved.

RFC: Hugepage/memory fit admission for best-of-K placement (minimal viable change)

Summary

Add a hard capacity (hugepage/memory) admission filter to the best-of-K placement sample() so that a node whose hugepage pool cannot hold the requested sandbox is never selected. Today hugepage headroom only influences ranking (memoryScore), never candidacy, so a full node can still win a sample and the request fails downstream on the orchestrator as 500: Failed to place sandbox. This RFC proposes the smallest change that closes that gap, gated behind a config flag with a shadow-then-enforce rollout.

This is the policy counterpart to #3553 (which asks to expose why placement failed) and the direct fix for the problem described in #3656.

Motivation

packages/api/internal/orchestrator/placement/placement_best_of_K.go --- sample() filters candidate nodes on exactly five predicates:

1. excluded set
2. CanAcceptNewRequests()
3. NodeSatisfiesCPU
4. NodeSatisfiesFeatures
5. label compatibility

None checks whether the node can physically hold the sandbox. Hugepage-pool load appears only as a soft term in memoryScore, used to rank survivors --- it never removes a node from candidacy. So under memory/hugepage pressure the placement engine can (a) pick a full node that then fails on the node side, or (b) when best-of-K samples only full nodes, fail the whole request. Observed on the dev cluster as batch runs where an entire concurrency level returned SandboxException: 500: Failed to place sandbox.

The failure is a placement-admission gap, not (always) genuine cluster exhaustion: the API had enough information locally to reject fast, but instead spent an API---orchestrator---placement round trip to fail slowly.

Goals

• Never forward a create to a node whose hugepage pool cannot hold it.
• Reuse the exact accounting memoryScore already uses, so ranking and admission never disagree.
• Zero-risk rollout: ship behind a flag, observe in shadow mode, then enforce.
• Preserve today's behavior for nodes that report no hugepage pool.

Non-Goals

• Not raising per-node density. Higher density needs memory sharing / reclamation (KSM, ballooning, proactive reclaim) --- out of scope here.
• Not changing CPU over-commit policy (R), which is a soft-score dimension and rarely a hard-failure source.
• Not the diagnostics surface --- that is #3553; this RFC only references it for the error-differentiation touchpoint.

Proposed design (minimal viable change)

All changes are confined to the placement package; no proto, no orchestrator, no node-agent changes. Node metrics already carry HugePagesTotal / HugePagesUsed / HugePagesReserved / HugePageSizeBytes.

1. Capacity helper (same accounting as memoryScore)

// nodeHasCapacity reports whether the node's hugepage pool can hold this
// sandbox: pool total minus (used + reserved + in-flight pending) must leave
// room for the request. A node reporting no pool (HugePagesTotal == 0) is
// treated as "unknown" and left to the soft score, preserving current behavior.
func nodeHasCapacity(n *nodemanager.Node, resources nodemanager.SandboxResources) bool {
	m := n.Metrics()
	if m.HugePagesTotal == 0 || m.HugePageSizeBytes == 0 {
		return true
	}
	var pendingMiB int64
	for _, r := range n.PlacementMetrics.InProgress() {
		pendingMiB += r.MiBMemory
	}
	need := hugePagesFor(resources.MiBMemory, m.HugePageSizeBytes) +
		hugePagesFor(pendingMiB, m.HugePageSizeBytes)
	committed := m.HugePagesUsed + m.HugePagesReserved
	if committed >= m.HugePagesTotal {
		return false
	}
	return m.HugePagesTotal-committed >= need
}

2. Wire it into sample()

sample() gains the resources nodemanager.SandboxResources parameter (already available in chooseNode, one-line call-site change), and after the label filter:

if config.HugepageFitFilter && !nodeHasCapacity(n, resources) {
	nodeFitRejectedCounter.Add(ctx, 1) // observability
	continue
}

3. Config flag (mirrors BEST_OF_K_HUGEPAGE_MEMORY)

type BestOfKConfig struct {
	R                 float64
	Alpha             float64
	K                 int
	ScoreHugepages    bool
	HugepageFitFilter bool // new; env BEST_OF_K_HUGEPAGE_FIT_FILTER, default false
}

4. K-sampling blind spot (the one non-trivial correctness point)

best-of-K samples only K nodes. With enforcement on, all K sampled nodes could be fit-rejected while free nodes exist but simply were not sampled. Mitigation: when the K-sample yields zero candidates and fit-filter is enabled, widen the search (one extra sampling round, or fall back to a full scan) before declaring failure. Without this, enforcement could fail placements that should succeed.

5. Error differentiation (ties into #3553)

When candidates end up empty, distinguish "no node with capacity" from "no compatible node (CPU/feature/label)". This lets operators tell scale up from misconfiguration --- exactly the ambiguity behind opaque Failed to place sandbox today.

Rollout plan

1. Shadow (HugepageFitFilter=false): helper runs, nodeFitRejectedCounter increments for would-be rejections, but no node is actually filtered. Measure how often a "would-be-rejected" node was in fact the one that later failed placement --- quantifies benefit with zero behavior change.
2. Enforce (HugepageFitFilter=true): flip per environment once the metric validates.

Alternatives considered

• Rank-only (status quo): soft memoryScore already exists but does not prevent selecting a full node --- this is the bug.
• Reject only at orchestrator: already happens; it is the slow round-trip failure we want to avoid.
• Global bin-packing / scheduler rewrite: much larger blast radius; unnecessary to close this specific gap.

Testing

• Unit: near-full node excluded from candidates; all-full node set --- empty candidates + distinct "no capacity" error; no-pool node unaffected; K-blind-spot fallback finds a free node outside the initial sample.
• A/B on a step-laddered replay (fit-filter off vs on): compare Failed to place rate and placement latency.
• Shadow-mode counter validated before enforcing.

Backwards compatibility

Default false --- behavior identical to today until explicitly enabled. Nodes without a reported pool are never hard-blocked. No API/proto/orchestrator changes.

Open questions

• Preferred widening strategy for the K blind spot: extra sampling round vs. full scan on empty result?
• Should the flag be global or per-cluster/per-pool?
• Should CPU over-commit (R) get a symmetric hard admission later, or stay soft?

---

Filed per CONTRIBUTING (issue-first for non-trivial changes). Happy to send the PR once the direction and the two open questions above are confirmed. Refs #3656 (problem), #3553 (diagnostics).