enhancement: The ledger frees a partition slot before the hardware does, so a request inside that window is refused

Start at DevicesReconciler.LivePhysicalOccupied and trace how reclaimMaxMisses, ledger remaining, and partitioned node keys affect placement before Allocate. Reproduce the allocated + remaining and reclaim-window partition e2e scenarios, then choose and implement a documented approach from the issue's options. Done means replacements no longer reach an unusable reclaimed instance, with tests covering the reclaim window and full-card behavior.

Summary

Deleting a partition-backed Pod frees the card in the ledger immediately, but the hardware
instance itself is only destroyed after the reclaimer's debounce (three consecutive misses). In
between, the node advertises room the card does not yet have, so a replacement scheduled into that
window is admitted by the scheduler and then refused at allocation time — surfacing on the Pod as
UnexpectedAdmissionError.

A workload that retries converges by itself. A workload that does not — restartPolicy: Never with no
owning controller — simply fails.

Mechanism

Occupancy for placement comes from DevicesReconciler.LivePhysicalOccupied, which builds the
occupied set from the allocation annotations of live Pods on the node. That set is the right one
for accounting, but it is not the physical truth during reclaim:

• the Pod object is gone, so its placement leaves LivePhysicalOccupied, and the ledger's
  remaining plus the node's *.partitioned* keys go back to the idle values;
• the vendor instance still exists, because reclaimMaxMisses passes must observe it unowned first;
• a replacement therefore picks a placement the card cannot actually host, and creating it fails.

Evidence

Reproduced on a single-card NVIDIA H100 with the partition e2e scenarios:

• The single-shot form fails. In the allocated + remaining scenario, the refill Pod submitted
  right after the saturating Pod was deleted never ran: Failed/UnexpectedAdmissionError. Everything
  else in that scenario passed — the freed count was advertised correctly (1 → 7), the room just was
  not usable yet.
• The retrying form self-heals. In the reclaim-window scenario, a same-profile replacement
  converged on its second attempt, 45 s after the predecessor was deleted, with exactly one
  terminal allocation failure along the way.

Also reproduced four times earlier on T-Head PPU hardware, so this is the shared reclaim path, not
one vendor's driver.

Reproduced again later on a node carrying eight accelerators with one of them partitioned, after the
stale-identity half described below was fixed: the same allocated + remaining scenario still fails
on its last check, and the same reclaim-window scenario now converges on its first attempt. The
window is therefore still open, and it is what remains of this issue.

Scope: what this issue is, and what does not close it

This issue is about one thing: for as long as the reclaimer has not destroyed the instance, the
node advertises room the card does not have, and a request that lands in that interval is refused
rather than held. Options 1 to 3 below each address that.

These do not close it, and a change that only does one of them should not be filed against this
issue:

• Making the refusal cleaner. A refusal is the correct outcome inside the window given the
  current design; the complaint is that the request should never have been admitted, or should have
  been queued rather than failed.
• Fixing the identity comparison in the reclaimer (see below). That removes a different failure
  that lives in the same interval. The window is untouched by it.
• Adding a retry at the call site. getAllocatingPodWithRetry exists and is used by only one of
  three call sites; closing that asymmetry removes one class of terminal rejection and is worth doing,
  but a request that names a placement the card genuinely cannot host is still refused.

A separate defect that lived in the same window, now fixed

An earlier revision of this issue stated that the replacement being handed the destroyed instance's
identity was already fixed by re-reading the card's partitions under its lock before destroying, and
that this made the failure inside the window clean — a refusal, never a dangling identity. That was
wrong, and it is worth recording why, because the reasoning looked sound.

The re-read is real and is on main: destroyMarkedInstancesOnCard re-reads the card's live
instances inside its lock and compares each against the marker with matchesMarker. But that
comparison is the GPU-instance id, the placement and the MIG-device identifier — and every one of
those is reproduced by a successor at the same slot. Placement selection is the lowest free slot and
therefore deterministic, and a MIG-device identifier is name-based, derived from the parent
accelerator and the instance's own identity, so destroying a partition and creating another at that
slot yields the same string. The re-read hands back an object whose every identity field equals the
predecessor's. It cannot separate them however often it runs.

So the window did not produce a clean refusal. It produced exactly the dangling identity the
re-read was believed to have removed: the reclaimer destroyed a partition an allocation had just
granted — measured at 0.32 s after the grant, on the same instance id, the same placement and the
same identifier — and the container then failed to start because its device could not be resolved.
Because that failed Pod left a marker of its own, the next allocation was condemned the same way, so
the first allocation after the markers were clear succeeded and every one after it did not.

Fixed by #499, which drops the liveness question entirely and asks the markers alone whether another
Pod claims the id. On hardware, three consecutive allocate-delete-allocate rounds at the same slot go
from 1/3 to 3/3, and the reclaim-window scenario's convergence cost drops from two attempts to one.

This changes nothing about the window itself. The refusal is now genuinely a refusal, which is
what the earlier revision claimed prematurely.

Options considered

1. Count the not-yet-destroyed instances as occupied — union LivePhysicalOccupied with the
   instances the durable markers still record but no live Pod owns. Placement then avoids them, which
   removes the window entirely wherever another placement is free, and turns the single-card-full case
   into an honest "no room" refusal (back-pressure the scheduler and Kueue can act on) rather than a
   failed create. Cheapest correct change; does not make the full-card refill succeed any sooner.
2. Reclaim synchronously during allocation — when the chosen placement is held by an unowned
   instance, destroy it in place and carry on. Closes the window completely, including the full-card
   case. Costs a vendor round trip inside Allocate, and needs the ownership test to be exactly the
   reclaimer's, or it risks destroying an instance whose Pod is still arriving.
3. Hold the ledger until the instance is really gone — release remaining and the node keys on
   destroy rather than on Pod deletion. Most truthful, and the scheduler stops being told about room
   that does not exist; makes every release visibly slower.
4. Leave it to callers — document the window and rely on retries. What happens today.

Worth noting that (1) also addresses the other half of the observation: an instance carved outside
GPUStack is never reflected in the node keys at all, and that never converges. Confirmed directly
since: an instance created by hand on a managed card is invisible to every per-profile key and to the
bare partition count for as long as it exists, and the count never catches up.