Build custom CRC bundles with KubeVirt, without nested virtualization

Start from the CRCBundle API types and the existing download-source reconciliation in the controller, plus the ClusterInstance bootstrap path that consumes QCOW2PVCRef and the id_ecdsa Secret. Use SNC's release-4.22 branch as read-only reference for install and guest-sealing steps. Done means a KubeVirt/Job-based build produces a Ready bundle with golden PVC, key Secret and validation clone, with download behavior unchanged — far beyond a first contribution.

Problem

CRCBundle can prepare an existing upstream .crcbundle, but it cannot build a custom CRC bundle from an OpenShift release image. Custom builds are useful when a CRC cluster must include additional MachineConfigs or cluster resources.

The build must use KubeVirt APIs. It must not use SNC's host-side virt-install and virsh workflow, or require a hypervisor inside a build pod or guest.

Proposal

Add a fresh-release build source to CRCBundle. A build will create an OpenShift single-node cluster in a KubeVirt VM, apply CRC-specific and user-provided configuration, prepare the guest for cloning, then publish a ready-to-clone golden PVC and its matching SSH-key Secret.

This is an in-cluster build flow, not a portable bundle build or distribution system. Do not produce a .crcbundle archive or an OCI bundle image.

Use SNC's release-4.22 branch as a reference for the install steps and guest changes. Do not run SNC's host scripts directly. Replace their use of virt-install, virsh, libvirt storage, host DNS/network changes, and host guestfish/libguestfs with KubeVirt, CDI, Kubernetes Jobs, and guest-side operations.

API design

• Add optional CRCBundle.spec.build fields for the OpenShift release image, pull-secret reference, CRC build profile, custom installation and post-install manifest references, VM resources, node placement, and build deadline.
• Keep the current download-and-prepare source for existing CRCBundle objects. Validate that a resource selects exactly one source: download or build.
• For build sources, derive the actual OpenShift version from the release payload. Make spec.version conditional on the source.
• Treat the build recipe as immutable. Record hashes for referenced inputs when the build starts. Use a new CRCBundle for a changed recipe.
• Add ClusterTemplate.crcBundleRef so a pool or instance can select a named custom bundle. Keep crcVersion as the existing upstream download path. Reject a template that sets both.
• Resolve crcBundleRef consistently during pool creation, instance provisioning, and boot-key recovery.

Example:

apiVersion: guestcluster.opdev.io/v1alpha1
kind: CRCBundle
metadata:
  name: crc-4-22-custom-a
spec:
  arch: amd64
  storageClassName: fast-storage
  goldenVolumeSize: 80Gi
  build:
    releaseImage: quay.io/example/ocp-release@sha256:<digest>
    pullSecretRef:
      name: bundle-build-pull-secret
    profile: crc-kubevirt-v1
    installationManifestRefs:
      - name: custom-machineconfigs
    postInstallManifestRefs:
      - name: custom-cluster-resources
    vm:
      cores: 8
      memory: 24Gi
      diskSize: 64Gi

Keep referenced Secrets and ConfigMaps in the operator namespace. Do not store pull-secret data in CRCBundle status.

Build flow

1. A Kubernetes Job extracts the matching installer from the release image, creates bootstrap-in-place single-node Ignition and manifests, and embeds Ignition in the matching RHCOS live ISO. This step must not need /dev/kvm or host virtualization services.
2. The controller creates CDI storage for the ISO and root disk, then creates a KubeVirt VM to install the cluster. Use /dev/vda as the bootstrap installation disk. Remove the ISO from the boot configuration after install.
3. A build helper uses SSH and a guest API tunnel to wait for install completion, apply the CRC profile and user manifests, and wait for MachineConfig rollouts and required operators.
4. Prepare the guest for cloning. Port the required SNC guest steps, including certificate rotation, pull-secret removal and rollout, image-cache preparation, CRC authentication resources, and stopping/disabling kubelet. Control time inside the guest; do not change the host clock.
5. Shut down the VM. Wait for the VMI and launcher pod to release the disk. Copy the stopped raw disk into the final golden PVC. Do not make a failed or active build disk available as a clone source.
6. Boot a separate clone to validate the bundle. Never use the golden PVC as a writable validation disk.
7. Set CRCBundle Ready only after validation succeeds. Remove temporary Jobs, VMs, ISO storage, and credentials after they are no longer in use.

Use a versioned KubeVirt VM profile with explicit firmware and networking settings for both build and consumer VMs. The network design must provide the guest DNS names required by the CRC cluster (api.crc.testing, api-int.crc.testing, and *.apps-crc.testing) without storing a launcher-pod address in the installed cluster. Prove the network and certificate-rotation behavior in a small end-to-end test before building the full workflow.

Run disk cleanup inside the guest. Do not run libguestfs appliances or require nested virtualization. KubeVirt provides the VM abstraction on the management cluster.

Output contract

Produce only the resources needed by the existing ClusterInstance bootstrap path:

• Golden PVC containing a CDI-compatible raw disk at /disk.img.
• SSH private-key Secret with the existing id_ecdsa data key.
• Actual OpenShift version, release and build-input digests, disk virtual size, and VM profile recorded as metadata/status.

The current QCOW2PVCRef field points to a PVC containing raw /disk.img; its name is misleading. Keep it for compatibility in this change and document the raw format. Add a separate diskSHA256 field because a fresh build has no .crcbundle archive checksum. Update boot-key binding to use the disk digest for built bundles, while preserving checksum behavior for downloaded bundles. Ensure instance root disks have enough capacity for the golden disk's virtual size.

Reconciliation and cleanup

• Track preparation, installation, customization, sealing, and validation stages with status and conditions.
• Persist resource references and a build attempt identity so reconciliation can resume after an operator restart.
• Give Jobs deadlines and retry limits. Scope helper Jobs to the expected VMI UID, as the existing crc-agent does.
• On deletion, stop Jobs and VMs before deleting their storage.
• Add ownership checks and recovery tests for missing or replaced build resources.

Acceptance criteria

• Build a bundle from an OpenShift release image and custom manifests using KubeVirt VMs and Kubernetes Jobs only.
• The build does not run virt-install, virsh, a nested hypervisor, or a libguestfs appliance.
• A Ready bundle has a bootable raw golden PVC and matching SSH-key Secret.
• A separate validation clone passes the existing CRC startup path, including SSH, DNS, kubelet/API startup, pull-secret injection, and credential setup.
• Two independent ClusterInstance objects can use the same custom crcBundleRef and become Ready through the normal bootstrap path.
• Existing download-based CRCBundle and crcVersion behavior continues to work.

Suggested implementation order

1. Prove the KubeVirt install path, guest DNS, firmware, certificate rotation, and cold restart.
2. Add API fields, source validation, crcBundleRef, and disk identity.
3. Add build resource builders, reconciliation stages, and cleanup.
4. Extract shared guest operations for the build helper and crc-agent.
5. Add disk sealing and validation-clone support.
6. Update CRDs, RBAC, deployment images, samples, docs, and tests.
7. Run an OpenShift/KubeVirt end-to-end build and pool-bootstrap test.