kube: single-node RolloutDiskToPVC stalls — longhorn scratch PVC can't schedule with default 3-replica + strict anti-affinity

Start in pkg/kube/lh-cfg-v1.9.1.yaml, specifically the data.default-setting.yaml block, and inspect the existing Longhorn settings. Use update_eve_image_cross_hv_with_volume.txt to reproduce the single-node upgrade path and verify that the CDI scratch PVC schedules and RolloutDiskToPVC no longer stalls in CREATING_VOLUME.

EVE-k single-node: RolloutDiskToPVC stalls indefinitely because longhorn's CDI scratch PVC fails to schedule

Summary

On a single-node EVE-k device, any flow that drives
kubeapi.RolloutDiskToPVC — most notably an EVE-kvm → EVE-k baseos
upgrade that carries a pre-existing Volume — stalls indefinitely.
Pillar's volumemgr stays at VolumeStatus.State=CREATING_VOLUME for
the lifetime of the device, periodically logging:

RolloutDiskToPVC: pvc:<uuid>-pvc-0 Failed after 600 seconds to
convert qcow to PVC ... client rate limiter Wait returned an error:
context deadline exceeded

Underneath: longhorn refuses to schedule replicas for the CDI scratch
PVC because EVE deploys longhorn with default-replica-count=3 and
upstream's replica-soft-anti-affinity=false, and a single-node
cluster cannot place 3 replicas without violating anti-affinity.
The scratch PVC stays Bound but its longhorn Volume is attached: no
forever; the CDI upload pod's FailedAttachVolume events repeat
every ~80 s.

Affected components

• pkg/pillar/kubeapi/vitoapiserver.go RolloutDiskToPVC (line 217)
• pkg/pillar/cmd/volumemgr/handlevolume.go (the caller that drives the
  state machine to CREATING_VOLUME)
• pkg/kube/lh-cfg-v1.9.1.yaml (EVE-shipped longhorn config; the
  default-setting ConfigMap sets only priority-class and
  disable-revision-counter, leaving replica-count and anti-affinity
  at upstream defaults).

Reproduction

Single-node EVE-k device with longhorn installed. Reproduces on every
upgrade from EVE-kvm carrying a pre-existing Volume. Eden-based recipe:

# Pre-staging on EVE-kvm
eden volume create -n xhv-pre-vol \
    docker://lfedge/eden-eclient:7a72275 --disk-size=200M
# wait for VolumeStatus.State=DELIVERED

# Push the kvm→k baseos upgrade
eden -t 10m controller edge-node eveimage-update \
    file://<rootfs-eve-k.squashfs> --os-version=<eve-k version> -m adam://

# After EVE-k boots and volumemgr unblocks from longhorn-wait, observe:
# /run/volumemgr/VolumeStatus/<uuid>#0.json reads
#   "State":109, "SubState":2, "Progress":100,
#   "Error":"Error converting /persist/vault/volumes/<uuid>-pvc-0.img
#            to PVC <uuid>-pvc-0: RolloutDiskToPVC ... attempts to
#            upload image failed"

# Confirm the underlying scheduling failure:
kubectl -n eve-kube-app describe pod cdi-upload-<uuid>-pvc-0
# Events show:
#   Warning FailedAttachVolume ... volume <pvc-X-scratch> is not ready
#                                  for workloads
#
# kubectl -n longhorn-system get volume.longhorn.io <pvc-X-scratch> -o yaml
# status.conditions:
#   - type: Scheduled
#     status: "False"
#     reason: ReplicaSchedulingFailure
#     message: 'disks are unavailable;precheck new replica failed'
#
# longhorn-manager logs:
#   "Unable to create new replica pvc-X-scratch-r-... error=
#    No available disk candidates to create a new replica of size N"

Root cause

Longhorn's upstream defaults:

Setting	Default	Effect on single-node
default-replica-count	3	every PVC wants 3 replicas
replica-soft-anti-affinity	false	replicas may not share a node
storage-minimal-available-percentage	25	disk must keep 25% free

With one node, the replica-count=3 + strict anti-affinity combination
mathematically can't be satisfied. Longhorn marks the volume as
Scheduled=False, reason=ReplicaSchedulingFailure. The CDI upload pod
that needs this PVC stays Pending; RolloutDiskToPVC polls until its
600s deadline, fails, and the volumemgr state machine never advances
out of CREATING_VOLUME.

EVE-shipped pkg/kube/lh-cfg-v1.9.1.yaml does not override either
setting, so it inherits upstream's multi-node-friendly defaults.

Proposed fix

Add the following to the data.default-setting.yaml block of
pkg/kube/lh-cfg-v1.9.1.yaml:

default-replica-count: 1
replica-soft-anti-affinity: true

Rationale:

• default-replica-count: 1: single-node EVE installations have only
  one disk to replicate to; the configured value is what's actually
  achievable. Multi-node clusters can override per-StorageClass or per-
  Volume.
• replica-soft-anti-affinity: true: allow longhorn to place replicas
  on the same node when no other node is available. On multi-node
  clusters longhorn still prefers different nodes; this only kicks in
  when forced.

These are the conventional single-node-cluster knobs documented in
longhorn's own deployment guide.

Optional follow-on: lower storage-minimal-available-percentage (25
is conservative for small /persist).

Workaround for operators today

Set timer.defer.content.delete=86400 via the controller, delete the
app/Volume on EVE-kvm (so no .img exists to migrate), push the
kvm→k upgrade, then recreate the same app/Volume on EVE-k. Pillar's
CAS finds the cached blobs (via the namespace-port + transitive-label
fixes in <my blob-reuse PR — link here>), no re-download, and longhorn
provisions a fresh PVC natively on eve-k — no .img → PVC conversion,
so this code path is sidestepped.

Out of scope

• The RolloutDiskToPVC path itself could also benefit from a longer
  / configurable deadline, but the underlying scheduling failure means
  no amount of waiting will unblock it on the current defaults.
• Operator-facing UI / controller validation that longhorn is
  configured appropriately for single-node deployments — that's a
  bigger conversation.

Environment

• EVE branch: master (reproduced on baseos-hv-check-volume-gate)
• HV: kvm → k cross-flavor upgrade
• ClusterType: REPLICATED_STORAGE (full longhorn install)
• Test: update_eve_image_cross_hv_with_volume.txt (eden draft;
  posted alongside the blob-reuse PR test plan)