KubeVirt High Availability: What Happens When a Kubernetes Worker Fails?
What happens to KubeVirt virtual machines when a Kubernetes worker fails? Understand failure detection, VM restart policies, fencing, storage and recovery.
On this page
Kubernetes knows how to replace failed containers. But what happens when the failed workload is an entire virtual machine?
Imagine a Kubernetes worker hosting a VM with 32 GiB of memory, a persistent PostgreSQL database and a disk attached through CSI. The worker disappears. Kubernetes notices. The node becomes unreachable. Controllers begin reconciling the cluster.
But the VM does not simply reappear on another worker.
Before it can safely restart, the platform must answer a more difficult question: is the original virtual machine actually dead, or has it merely lost contact with the control plane?
Get that distinction wrong and you risk running two instances against the same persistent disk.
In Understanding KubeVirt Internals: Controllers, QEMU/KVM and VM Lifecycle , we examined how KubeVirt translates Kubernetes resources into running virtual machines.
Now we will follow the opposite path: what happens when the infrastructure underneath a running VM fails? No installation tutorial. No artificial benchmark. Just the failure mechanics that determine whether a KubeVirt platform is genuinely resilient.
Before the failure: one VM, several control loops
Consider a VM running on worker-01.
The Kubernetes API contains a VirtualMachine resource describing the desired state. KubeVirt maintains a corresponding VirtualMachineInstance (VMI), and Kubernetes schedules its virt-launcher pod onto a worker.
Inside that execution environment, QEMU/KVM runs the guest. The guest’s persistent disk is supplied through a Kubernetes volume, typically backed by a CSI storage system.
Several components participate in this arrangement:
- Kubernetes node controller: monitors node health and reachability.
- KubeVirt virt-controller: reconciles VM and VMI resources.
- KubeVirt virt-handler: manages node-local VM operations and reports virtualization-related state.
- Kubernetes scheduler: selects eligible nodes for new launcher pods.
- CSI components: coordinate volume provisioning, attachment and mounting according to the storage driver.
These components do not share a single failure detector or recovery mechanism. That matters because a worker can lose contact with Kubernetes while its QEMU process continues executing.
The API’s view of the world and the physical state of the guest can temporarily disagree.
The worker disappears: Kubernetes detects uncertainty
At 10:00, suppose worker-01 loses connectivity to the Kubernetes control plane. The kubelet can no longer update its Node Lease or report status.
Kubernetes normally uses lightweight Lease renewals alongside Node status updates to monitor worker health. After the configured node-monitoring grace period expires, the node controller can transition the Node’s Ready condition to Unknown.
The controller also applies the appropriate unreachable taint, preventing ordinary new workloads from being scheduled there. But Unknown means exactly that: the control plane cannot determine the node’s current state.
It does not mean the machine has powered off. The worker might have suffered a complete hardware failure. Alternatively, only its management network might be unavailable.
In the second scenario, the guest could still be running, writing to its disk and serving clients.
Kubernetes’ normal pod eviction behavior introduces another delay. Pods commonly receive default NoExecute tolerations of 300 seconds for unreachable or not-ready nodes, unless overridden by their controllers or specifications.
KubeVirt also maintains its own node-level heartbeat through virt-handler. Its node controller can mark a worker ineligible for new VM scheduling when that heartbeat becomes stale.
These mechanisms operate at different layers. Neither a Kubernetes node condition nor a stale KubeVirt heartbeat independently proves that QEMU has stopped.
The distinction is documented in Kubernetes Node Status and KubeVirt’s unresponsive-node handling .
Detecting an unreachable worker is only the first step. The following sequence shows how Kubernetes and KubeVirt progress from failure detection to a possible VM restart, including the safety checks that must be satisfied before replacement.
Who decides whether the VM should restart?
A KubeVirt VM has a desired running state, but that does not mean the platform can immediately create another instance.
The runStrategy field controls how KubeVirt responds when a VMI terminates.
Consider:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: critical-vm
spec:
runStrategy: Always
template:
spec:
# Guest configuration omittedWith Always, KubeVirt attempts to maintain a running guest by creating another VMI after the previous instance has terminated.
RerunOnFailure behaves differently. It restarts instances after certain uncontrolled failures, including infrastructure-related failures, but does not treat an intentional guest shutdown in the same way.
Neither strategy means that KubeVirt immediately launches a duplicate VM when its worker becomes unreachable. The existing VMI must first be resolved through the relevant lifecycle and cleanup mechanisms.
This is a critical protection.
Imagine that the original VM continues running on an isolated worker while the control plane creates a second VM against the same disk. Both operating systems might believe they exclusively own the filesystem.
A restart policy is not a fencing mechanism. KubeVirt’s run strategy documentation explains the intended behavior of each strategy.
The fencing problem: unreachable is not dead
This is where VM high availability becomes more complicated than ordinary stateless container recovery.
Consider two possible failures.
Scenario A: Complete power loss
The worker loses power. QEMU stops executing. The guest is no longer writing to its disk. Once the failure has been established and the old instance’s ownership and storage state have been resolved, recovery can proceed on an eligible worker.
Scenario B: Network partition
The worker loses access to the Kubernetes API but retains power and storage connectivity. The VM continues running. From the control plane’s perspective, both scenarios can initially look similar. Restarting the VM elsewhere in Scenario B could create a split-brain situation.
For this reason, a production recovery design needs a reliable way to establish that the original workload can no longer modify its storage. That is the purpose of fencing.
Depending on the infrastructure, fencing can involve powering off a physical host through an out-of-band management controller, terminating a cloud instance through an infrastructure API, or using a storage-level mechanism that reliably excludes the previous writer.
These mechanisms are not interchangeable. Their safety depends on what they actually guarantee. Deleting a Kubernetes Node object does not power off the machine.
Force-deleting a pod removes API state without proving that its process has stopped on an unreachable worker. And deleting a VolumeAttachment object is not evidence that the old host has released the underlying device.
KubeVirt documents special handling for unresponsive nodes, including cases where administrators must establish that the node is dead before force-removing stuck resources.
The governing rule is simple: never confuse removing an object from the Kubernetes API with terminating the workload it represents.
Storage decides whether recovery can complete
Assume the original VM is now safely stopped or fenced. KubeVirt can reconcile the terminated instance, and Kubernetes can attempt to schedule a replacement launcher pod. But that pod still needs its disk.
For a VM using a persistent volume, the storage system may need to release an existing attachment, attach the volume to another worker, expose the device and mount it into the new execution environment. The exact sequence depends on the CSI driver and storage backend.
A block volume with ReadWriteOnce access may be suitable for cold restart on another worker, provided the driver supports safe detachment and reattachment. However, ReadWriteOnce does not itself guarantee single-writer fencing at the physical storage layer. It is a Kubernetes access mode, not a distributed storage consensus protocol.
In the same way, a ReadWriteMany volume can support access from multiple nodes, but that does not make arbitrary concurrent writes to a VM filesystem safe. Storage availability, attachment semantics, filesystem behavior and fencing must all be considered.
A failed volume detach can delay VM recovery even when the destination worker has sufficient CPU and memory. The reverse is also possible: the volume becomes available, but the VM cannot start because the destination lacks compatible CPU features, required devices or network attachments.
High availability is therefore constrained by the intersection of compute, storage and networking capabilities. Not by the number of Ready workers alone.
Reconstructing the virtual machine
Once the old instance is safely resolved and the desired running state still requires a guest, KubeVirt can create a replacement VMI. Kubernetes then schedules a new virt-launcher pod onto an eligible worker.
The destination must satisfy the VM’s resource requests, placement constraints, architecture and virtualization requirements. The persistent volume must become usable on that worker. QEMU starts a new guest process, and the guest operating system boots. This is not live migration.
The VM’s volatile memory state is lost. Processes restart. TCP connections disappear. Applications must recover from the interruption. For a database workload, recovery may involve replaying a write-ahead log, checking filesystem consistency and reconnecting dependent services.
Even after the VMI reaches Running, the application might not yet be ready to accept traffic. KubeVirt supports readiness and liveness probes, and guest watchdog mechanisms can help detect certain classes of guest or application failure.
But these are distinct from worker failure detection. A guest operating system can hang while Kubernetes continues reporting the worker as Ready. By contrast, a VM can remain functional while the control plane temporarily loses contact with its host.
That is why VM availability and application availability must be measured separately.
Live migration is not failure recovery
Live migration and high-availability restart solve different problems. During planned maintenance, the source worker is still operational.
KubeVirt can coordinate a live migration by preparing a destination instance and transferring guest execution state, including memory, while the source remains available. With pre-copy migration, memory pages are transferred while the guest continues running. A final handover transfers execution to the destination.
This can reduce guest interruption, but migration depends on storage access, CPU compatibility, network configuration and available resources.
Some device configurations cannot migrate. Workloads with high memory dirty rates may also struggle to complete migration within acceptable limits. When a worker suddenly loses power, there is no functioning source from which to transfer the VM’s live memory state.
The platform must instead perform failure detection and a safe restart, potentially followed by guest and application recovery.
| Scenario | Typical mechanism | Guest memory preserved? |
|---|---|---|
| Planned worker drain | Live migration, if supported | Yes, subject to migration success |
| Planned shutdown without migration | Graceful guest shutdown and restart | No |
| Worker power loss | Safe restart after failure resolution | No |
| Network partition | Establish safety/fencing before replacement | Not through cold restart |
| Guest application failure | Application recovery, probes or watchdog | Depends on recovery action |
The practical implication is important: a VM that supports live migration is not automatically protected against unplanned worker failure.
See the upstream KubeVirt live migration documentation and node maintenance guidance .
What does KubeVirt high availability actually guarantee?
There is no universal recovery time guarantee simply because a VM runs under KubeVirt.
Recovery time depends on several sequential operations:
- Detecting the infrastructure failure.
- Establishing that the previous instance is no longer active or can no longer access its storage.
- Resolving the old VMI and launcher lifecycle.
- Scheduling a replacement onto an eligible worker.
- Making persistent storage available.
- Booting the guest operating system.
- Recovering the application.
Each stage can introduce delay or fail independently. This is why Recovery Time Objective (RTO) and Recovery Point Objective (RPO) should be defined at the application level.
A persistent volume does not automatically provide an RPO of zero. Data may remain in volatile caches, storage may fail, and asynchronous replication may lag. Likewise, a VM that successfully restarts after five minutes does not meet a five-minute application RTO if the database requires another ten minutes to recover.
For critical workloads, the design should include explicit evidence for failure detection, fencing, storage recovery, guest startup and application readiness. If the workload requires near-continuous availability, application-level replication or clustering may be more appropriate than relying solely on VM-level restart.
When KubeVirt HA is the wrong abstraction
KubeVirt can be a good fit for organizations that want to operate virtual machines alongside container workloads under Kubernetes native management. But that does not make it the right availability mechanism for every application.
Consider a legacy database that requires strict storage fencing, dedicated device passthrough, predictable recovery behavior and a tightly controlled maintenance process. KubeVirt may support the workload, but the required infrastructure and operational complexity can exceed the benefits of consolidating management under Kubernetes.
The same applies to workloads whose availability requirements cannot tolerate the time needed for failure detection, safe storage recovery and guest reboot.
In these cases, the alternatives are not limited to choosing another hypervisor. The application may need native replication, clustering, an external database service or a different fault-tolerance architecture.
The relevant question is not whether KubeVirt can restart a VM. It is whether the complete system can meet the workload’s recovery objectives under the failures that matter.
To summarize - High Availability begins with uncertainty
A Kubernetes worker becoming unreachable is not proof that the virtual machines running on it have stopped. That uncertainty drives the entire KubeVirt recovery problem.
Kubernetes detects node health changes. KubeVirt reconciles VM state. Infrastructure and storage mechanisms must establish safe ownership before replacement execution can proceed. Only then can a guest restart and its application begin recovering.
Understanding these boundaries is essential when designing a virtualization platform on Kubernetes.
High availability is not the ability to start another VM. It is the ability to restore service without creating a second failure while recovering from the first.
For teams evaluating KubeVirt as a production virtualization platform, ape factory provides Kubernetes and platform architecture assessments covering failure domains, storage architecture and operational readiness.
