[Workload]: migration-stress

Bắt đầu bằng cách xác định workload memory hiện có và định nghĩa workload của nó, sau đó kiểm tra cách các gói cloud-init và các dịch vụ systemd được biểu diễn. Triển khai workload migration-stress với lệnh stress-ng được tài liệu hóa và các tham số có thể cấu hình. Xác thực rằng một VM có workload này hiển thị hoạt động migration có thể đo lường được bằng virtctl migrate và oc get vmim.

Workload Name

migration-stress

Workload Description

High memory-dirty-rate workload purpose-built to stress KubeVirt live migration. Continuously writes to memory pages at a configurable rate, forcing the live migration pre-copy phase to iterate more rounds before convergence. Produces a sustained, measurable memory dirtying pattern that makes the VM actively hard to migrate — the exact scenario that storage, networking, and monitoring partners need to validate their products against during migration events.

This is uniquely KubeVirt/CNV — no equivalent exists in generic VM workload tooling. Live migration is a first-class operation in OpenShift Virtualization (triggered by node drain, maintenance, rebalancing), and partner products must continue functioning correctly while VMs migrate. The existing memory workload uses stress-ng to allocate and hold memory at 80% pressure — it tests memory capacity, not memory mutation rate. migration-stress targets a completely different axis: how fast pages are being dirtied, which directly determines migration difficulty and duration.

Tooling and Packages

• Tool: stress-ng with the vm stressor in aggressive write mode, or a custom mmap/write loop
• RPM packages: stress-ng
• systemd service command: stress-ng --vm 2 --vm-bytes 75% --vm-method write64 --vm-keep --aggressive
  • --vm 2: two VM stressor workers
  • --vm-bytes 75%: each worker maps 75% of available memory
  • --vm-method write64: write 64-bit values (high dirty rate)
  • --vm-keep: keep mappings (re-dirty same pages rather than remap)
  • --aggressive: maximize throughput
• Alternative: custom C/shell program using mmap(MAP_PRIVATE|MAP_ANONYMOUS) + sequential write loop for maximum dirty rate control
• Configurable parameters:
  • dirty-rate-target: target MB/s of memory dirtying (default: unbounded — as fast as possible)
  • vm-workers: number of stressor workers (default: 2)
  • vm-bytes-percent: percentage of memory to map per worker (default: 75)
  • vm-method: stress-ng vm method — write64 (fast), write1024 (faster), zero (fastest) (default: write64)

VM Count Model

Single VM (like cpu, memory, disk)

Required Resources

• Persistent storage (DataVolume)
• Kubernetes Service (for inter-VM communication)
• Kubernetes Secret (for credentials or config)
• Additional CPU/memory beyond defaults
• GPU or special device passthrough

Memory should be sized larger than defaults (e.g., 4Gi+) to produce meaningful migration difficulty. With only 1Gi of memory, even a high dirty rate converges quickly. At 4-8Gi with aggressive dirtying, migration takes noticeably longer and may require multiple pre-copy rounds.

Cloud-Init Details

packages:
  - stress-ng
write_files:
  - path: /etc/systemd/system/virtwork-migration-stress.service
    content: |
      [Unit]
      Description=Virtwork migration stress workload (high memory dirty rate)
      After=multi-user.target
      [Service]
      Type=simple
      ExecStart=/usr/bin/stress-ng \
        --vm 2 \
        --vm-bytes 75%% \
        --vm-method write64 \
        --vm-keep \
        --aggressive \
        --metrics-brief
      Restart=always
      RestartSec=5
      [Install]
      WantedBy=multi-user.target
runcmd:
  - systemctl enable --now virtwork-migration-stress.service

Use Case

• Storage partners (CSI drivers): During live migration, the storage layer must handle the VM's disk being accessed from a new node. A VM that is actively dirtying memory at high rates creates longer migration windows and more complex storage handoff scenarios. Partners need to validate that their CSI driver maintains I/O consistency during extended migration — not just the quick migrations that idle VMs produce.
• Network partners (CNI, SDN, OVN-Kubernetes): Live migration moves a VM's network identity (IP, MAC) to a new node. A VM with active memory writes produces a longer migration window during which network traffic must be correctly routed to both the source and destination. Partners need sustained migrations to validate network cutover correctness and downtime measurement.
• Monitoring/Observability partners: Partners need to validate that their agents correctly handle metric continuity during migration — no gaps, no duplicate data points, correct host/node attribution before and after migration. A VM that migrates quickly (idle VM, <1 second) doesn't exercise this. A VM that takes 30-60 seconds to migrate because of dirty memory exposes real-world monitoring gaps.
• Platform engineering / Node lifecycle: Validates that oc adm drain with live migration works correctly under realistic conditions. An idle cluster drains instantly; a cluster with migration-stress VMs reveals the real-world drain duration and any timeout issues.
• Red Hat CNV engineering: Provides a reproducible, tunable migration difficulty benchmark. Adjusting vm-bytes-percent and vm-method produces a spectrum from "easy migration" to "convergence-challenged migration" — useful for regression testing migration improvements.

Additional Context

• This workload is distinct from the existing memory workload in purpose and behavior:
  • memory: allocates and holds memory at target pressure — tests memory capacity and OOM behavior
  • migration-stress: continuously writes to memory pages — tests migration convergence and duration
  • A VM can run both simultaneously (memory pressure + high dirty rate), but they test different things.
• The key metric to observe is migration duration: virtctl migrate <vm-name> followed by watching oc get vmim (VirtualMachineInstanceMigration) for the migration to complete. With this workload active, migration should take measurably longer than an idle VM.
• Consider pairing this workload with a monitoring dashboard that shows: migration start time, dirty page rate during pre-copy iterations, convergence point, total migration duration, and post-migration workload resumption. This is the demo that sells the value to partners.
• stress-ng's --vm-method options provide a tunable dirty rate spectrum:
  • write64: moderate dirty rate (~2-5 GB/s depending on hardware)
  • write1024: higher dirty rate
  • zero: maximum dirty rate (memset to zero)
  • This lets users tune migration difficulty from "slightly harder than idle" to "convergence-challenged."
• The --vm-keep flag is critical — without it, stress-ng remaps memory on each iteration, which tests page fault handling rather than page dirtying. --vm-keep ensures the same pages are repeatedly dirtied, which is what stresses the migration pre-copy algorithm.
• Memory recommendation: 4Gi minimum, 8Gi for meaningful migration stress. At 2Gi with write64, migration typically converges in 1-2 pre-copy rounds. At 8Gi, it may take 5+ rounds or require post-copy migration fallback.