Kubernetes Pod Evictions: Diagnosing Resource Pressure
Diagnose Kubernetes pod evictions caused by memory, disk, and PID pressure, and build safer capacity, observability, and incident-response practices.

On this page
A Kubernetes cluster can have healthy Deployments and still lose application pods when a node runs short of memory, local storage, or process IDs. These are node-pressure evictions: the kubelet attempts to reclaim resources to keep the node operational. They are different from application crashes, container out-of-memory kills, and voluntary evictions initiated through the Eviction API.
For platform teams, the objective is not to eliminate every eviction. It is to understand the signals, make resource allocation predictable, and ensure that applications recover when node capacity becomes constrained.
Distinguish the failure modes
| Event | Typical trigger | What happens | Primary investigation |
|---|---|---|---|
| Node-pressure eviction | Memory, filesystem, inode, or PID scarcity | Kubelet terminates a selected pod; its status may show Evicted | Node conditions, kubelet events, resource usage |
| Container OOM kill | Container exceeds its memory cgroup limit | Container is killed, often restarted within the same pod | lastState.terminated.reason, memory limit |
| API-initiated eviction | Drain or other client uses the Eviction API | Eviction honors applicable PodDisruptionBudgets | Drain logs, PDB status |
| Scheduler preemption | Higher-priority pending pod needs capacity | Lower-priority pods may be terminated | Pod priority and scheduler events |
| Node failure | Host or kubelet becomes unavailable | Pods may be replaced after node-failure handling | Node readiness and infrastructure health |
Important: PodDisruptionBudgets are not a guarantee against kubelet node-pressure evictions. A PDB primarily constrains voluntary disruptions performed through the Eviction API.
Understand node pressure signals
The kubelet monitors resources such as:
memory.available: memory the kubelet estimates can be reclaimed or used.nodefs.availableandnodefs.inodesFree: free space and inodes on the node filesystem.imagefs.availableandimagefs.inodesFree: image filesystem capacity, when separate.containerfs.*: container writable-layer filesystem signals on supported layouts and versions.pid.available: available process identifiers.
The node can report MemoryPressure, DiskPressure, or PIDPressure. Kubelet thresholds, filesystem layout, and reclaim behavior depend on the Kubernetes distribution and version.
kubectl get nodes
kubectl describe node worker-1
kubectl get node worker-1 -o json | jq '.status.conditions[] | select(.type | endswith("Pressure"))'A True pressure condition is actionable, but do not rely on node conditions alone: brief resource spikes and eviction decisions can occur before a monitoring scrape catches a condition transition.
Investigate an evicted pod
Start with the terminated pod object, if it is still present:
kubectl get pods -A --field-selector=status.phase=Failed
kubectl get pods -A -o json | jq -r '
.items[]
| select(.status.reason == "Evicted")
| [.metadata.namespace, .metadata.name, .spec.nodeName, .status.message]
| @tsv'
kubectl describe pod <pod-name> -n <namespace>
kubectl get events -A --sort-by=.lastTimestampThe pod status message often identifies the resource under pressure and may include the container’s observed usage relative to its request. Retain this information in incident records before deleting failed pod objects.
Next, correlate the affected pod with its node:
kubectl top node worker-1
kubectl top pods -A --sort-by=memory
kubectl describe node worker-1kubectl top provides a current snapshot, not historical proof of what caused an earlier eviction. Use Prometheus history, node logs, and kubelet metrics for the incident timeline.
Memory pressure: requests, limits, and QoS
The scheduler uses requests to place pods. The kubelet evaluates actual resource pressure on the node. A memory limit constrains a container’s consumption, but it does not reserve memory on the node.
Kubernetes assigns pods to Guaranteed, Burstable, or BestEffort QoS classes based on CPU and memory requests and limits. QoS is a useful operational signal, not a complete eviction-order specification.
Under memory pressure, kubelet ranking considers whether pod usage exceeds requests, pod priority, and relative usage. A higher-priority pod can receive stronger protection even when it is not Guaranteed.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: payments-api
template:
metadata:
labels:
app: payments-api
spec:
containers:
- name: api
image: registry.example.com/payments-api:1.4.0
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
memory: 1GiThis is an illustrative Burstable configuration, not a universal sizing recommendation. CPU limits are deliberately omitted: some latency-sensitive services prefer CPU requests without throttling-prone CPU limits, subject to platform policy. Memory requests and limits should be based on workload measurements and failure tolerance.
To inspect QoS:
kubectl get pods -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass,NODE:.spec.nodeName'An OOMKilled container should be investigated separately:
kubectl get pod <pod-name> -n <namespace> -o json | jq '
.status.containerStatuses[]? |
{name, restartCount, lastTermination: .lastState.terminated}'Disk pressure is not just a full root filesystem
Node storage pressure may come from container writable layers, images, logs, local ephemeral volumes, or inode exhaustion. Depending on the runtime and filesystem layout, these consume different filesystems.
On a node you are authorized to inspect:
df -h
df -i
sudo du -xhd1 /var/lib/kubelet 2>/dev/null
sudo du -xhd1 /var/log 2>/dev/nullUse the container runtime appropriate to the platform (crictl, podman, or a distribution-specific diagnostic tool). Do not indiscriminately delete files beneath kubelet or container-runtime directories: manual deletion can corrupt runtime state or remove data needed by running workloads.
Explicit ephemeral-storage requests and limits help make local disk consumption visible:
resources:
requests:
cpu: 100m
memory: 256Mi
ephemeral-storage: 1Gi
limits:
memory: 512Mi
ephemeral-storage: 3GiRemember that ephemeral-storage accounting and enforcement depend on node filesystem configuration. An ephemeral-storage limit is not a replacement for log retention, application cleanup, or node capacity planning.
PID pressure and runaway process creation
A process-spawning bug, an aggressive job, or a compromised workload can exhaust the node’s process capacity. PID pressure may affect workloads unrelated to the original offender.
kubectl get node worker-1 -o jsonpath='{.status.conditions[?(@.type=="PIDPressure")].status}{"\n"}'On the host, inspect process and thread counts with approved diagnostic access:
ps -eLf | wc -l
cat /proc/sys/kernel/pid_maxConsider runtime PID limits, workload process behavior, and dedicated node pools for workloads that create many processes. Do not assume CPU and memory quotas alone prevent PID exhaustion.
Eviction thresholds and node reservations
Kubelet supports hard and soft eviction thresholds. Hard thresholds trigger reclaim and eviction without a configured soft-threshold grace period; soft thresholds require a sustained breach before eviction. The kubelet can first reclaim node-level resources, such as unused images, before selecting pods.
Example for a self-managed Kubernetes cluster:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
memory.available: "200Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
evictionSoft:
memory.available: "500Mi"
evictionSoftGracePeriod:
memory.available: "2m"
evictionMaxPodGracePeriod: 60This is not a drop-in production configuration. Kubelet settings must be evaluated against the complete set of default thresholds: supplying custom thresholds may change how unspecified defaults are handled, depending on the kubelet configuration and MergeDefaultEvictionSettings behavior.
Reserve resources for the operating system and Kubernetes components using the distribution’s supported mechanisms. systemReserved and kubeReserved affect node allocatable accounting; enforcement requires careful cgroup configuration and validation.
OpenShift-specific guidance
OpenShift manages kubelet configuration through operators and supported cluster resources. Avoid editing kubelet files directly on Red Hat CoreOS nodes. Review the supported KubeletConfig and MachineConfigPool workflow for your OpenShift release, and test changes on a dedicated pool before broad rollout.
A change to node reservations or eviction thresholds can affect scheduling capacity, MachineConfigPool updates, and cluster maintenance. Treat it as a platform change, not a quick incident workaround.
Use namespace policy to prevent unsafe defaults
A LimitRange can inject default resource requests and limits into containers that omit them:
apiVersion: v1
kind: LimitRange
metadata:
name: workload-defaults
namespace: production
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 256Mi
default:
memory: 512Mi
min:
memory: 64Mi
max:
memory: 4GiA ResourceQuota constrains aggregate namespace allocations:
apiVersion: v1
kind: ResourceQuota
metadata:
name: workload-budget
namespace: production
spec:
hard:
requests.cpu: "12"
requests.memory: 24Gi
limits.memory: 48Gi
pods: "80"Neither resource quotas nor limit ranges guarantee physical node capacity. A namespace can remain within quota while a node experiences pressure. Combine admission policy with right-sizing, placement controls, and cluster-level capacity management.
Monitor before users notice
Start with a small set of actionable signals:
| Signal | Why it matters |
|---|---|
kube_node_status_condition | Reports node pressure conditions via kube-state-metrics |
kube_pod_status_reason{reason="Evicted"} | Indicates retained pod objects reporting eviction, if exposed by your kube-state-metrics version |
node_memory_MemAvailable_bytes | Host memory headroom; not identical to kubelet’s eviction calculation |
node_filesystem_avail_bytes | Filesystem capacity, with correct mountpoint/device selection |
kubelet_evictions | Kubelet eviction counters, where available and scraped |
| Container memory working set and OOM counters | Workload-level memory behavior |
Example PromQL for a node pressure condition:
kube_node_status_condition{
condition=~"MemoryPressure|DiskPressure|PIDPressure",
status="true"
} == 1Example for available host memory ratio:
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytesUse this as a capacity indicator, not as an exact kubelet eviction-threshold calculation. For disk alerts, select the actual node, image, and container filesystems; checking only / can miss the filesystem responsible for eviction.
For OpenShift, prefer the built-in cluster monitoring stack and existing platform alerts before deploying duplicate node-level rules. Ensure custom alerts use metrics actually available through your monitoring endpoint.
Incident response: stabilize, diagnose, recover
When evictions are active, follow a controlled sequence:
- Identify the affected node and pressure signal. Collect node conditions, pod status messages, events, and relevant metric history.
- Protect critical capacity. Verify which applications and system services are affected. Avoid unplanned deletion of healthy pods.
- Find the resource consumer. Distinguish sustained workload growth from a sudden leak, runaway job, log explosion, or storage configuration issue.
- Apply the smallest safe mitigation. Examples include scaling out the node pool, reducing a noncritical workload, fixing log retention, or correcting resource requests.
- Verify recovery. Confirm pressure conditions clear, replacements become Ready, and workload-level service indicators recover.
- Capture follow-up actions. Update capacity forecasts, admission defaults, alerts, and runbooks.
Avoid treating the following as routine fixes:
# DANGEROUS as a blanket cleanup action:
# kubectl delete pods -A --field-selector=status.phase=Failed
# DANGEROUS on a live node without runtime-aware procedures:
# find /var/log/containers -name '*.log' -deleteDeleting all failed pods destroys diagnostic evidence and may affect unrelated workloads. Deleting runtime-managed log files can disrupt log collection and is not a substitute for supported rotation settings.
Design for resilience, not just eviction avoidance
Requests and limits are one layer of reliability. Production workloads also need sufficient replicas, topology-aware scheduling, readiness probes, and realistic spare capacity.
A PodDisruptionBudget is valuable during planned node drains but does not stop node-pressure eviction. Priority classes influence eviction and scheduling behavior, but assigning every workload a high priority defeats their purpose.
For stateful applications, examine storage locality, recovery time, quorum, and whether the workload can tolerate a node disappearing unexpectedly. A restarted pod is not necessarily an available service.
Operational checklist
[ ] Workload requests reflect observed steady-state and peak usage.
[ ] Memory limits are tested against application behavior and OOM risk.
[ ] Ephemeral-storage growth and log retention are controlled.
[ ] Node filesystem and inode capacity are monitored separately.
[ ] Critical workloads have appropriate priority and placement policies.
[ ] Namespace quotas and limit ranges are managed through GitOps.
[ ] Pressure alerts are validated against actual cluster metrics.
[ ] Incident runbooks preserve evidence before cleanup.
[ ] Kubelet configuration changes follow the platform-supported workflow.
[ ] Capacity plans account for maintenance and node failures.
Conclusion
Pod eviction is a symptom of a resource-management decision made under pressure. The durable response is to connect application resource profiles, node capacity, scheduling, and observability into a single operational model. When platform teams can explain why a pod was evicted and how the service recovered, eviction handling becomes a manageable reliability practice rather than an unpredictable production incident.


