Technical Insight How-to Guide

Kubernetes Pod Evictions: Diagnosing Resource Pressure

Diagnose Kubernetes pod evictions caused by memory, disk, and PID pressure, and build safer capacity, observability, and incident-response practices.

Kubernetes Pod Evictions: Diagnosing Resource Pressure
On this page

A Kubernetes cluster can have healthy Deployments and still lose application pods when a node runs short of memory, local storage, or process IDs. These are node-pressure evictions: the kubelet attempts to reclaim resources to keep the node operational. They are different from application crashes, container out-of-memory kills, and voluntary evictions initiated through the Eviction API.

For platform teams, the objective is not to eliminate every eviction. It is to understand the signals, make resource allocation predictable, and ensure that applications recover when node capacity becomes constrained.

Distinguish the failure modes

EventTypical triggerWhat happensPrimary investigation
Node-pressure evictionMemory, filesystem, inode, or PID scarcityKubelet terminates a selected pod; its status may show EvictedNode conditions, kubelet events, resource usage
Container OOM killContainer exceeds its memory cgroup limitContainer is killed, often restarted within the same podlastState.terminated.reason, memory limit
API-initiated evictionDrain or other client uses the Eviction APIEviction honors applicable PodDisruptionBudgetsDrain logs, PDB status
Scheduler preemptionHigher-priority pending pod needs capacityLower-priority pods may be terminatedPod priority and scheduler events
Node failureHost or kubelet becomes unavailablePods may be replaced after node-failure handlingNode readiness and infrastructure health

Important: PodDisruptionBudgets are not a guarantee against kubelet node-pressure evictions. A PDB primarily constrains voluntary disruptions performed through the Eviction API.

Understand node pressure signals

The kubelet monitors resources such as:

  • memory.available: memory the kubelet estimates can be reclaimed or used.
  • nodefs.available and nodefs.inodesFree: free space and inodes on the node filesystem.
  • imagefs.available and imagefs.inodesFree: image filesystem capacity, when separate.
  • containerfs.*: container writable-layer filesystem signals on supported layouts and versions.
  • pid.available: available process identifiers.

The node can report MemoryPressure, DiskPressure, or PIDPressure. Kubelet thresholds, filesystem layout, and reclaim behavior depend on the Kubernetes distribution and version.

kubectl get nodes
kubectl describe node worker-1
kubectl get node worker-1 -o json | jq '.status.conditions[] | select(.type | endswith("Pressure"))'

A True pressure condition is actionable, but do not rely on node conditions alone: brief resource spikes and eviction decisions can occur before a monitoring scrape catches a condition transition.

Investigate an evicted pod

Start with the terminated pod object, if it is still present:

kubectl get pods -A --field-selector=status.phase=Failed

kubectl get pods -A -o json | jq -r '
  .items[]
  | select(.status.reason == "Evicted")
  | [.metadata.namespace, .metadata.name, .spec.nodeName, .status.message]
  | @tsv'

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -A --sort-by=.lastTimestamp

The pod status message often identifies the resource under pressure and may include the container’s observed usage relative to its request. Retain this information in incident records before deleting failed pod objects.

Next, correlate the affected pod with its node:

kubectl top node worker-1
kubectl top pods -A --sort-by=memory
kubectl describe node worker-1

kubectl top provides a current snapshot, not historical proof of what caused an earlier eviction. Use Prometheus history, node logs, and kubelet metrics for the incident timeline.

Memory pressure: requests, limits, and QoS

The scheduler uses requests to place pods. The kubelet evaluates actual resource pressure on the node. A memory limit constrains a container’s consumption, but it does not reserve memory on the node.

Kubernetes assigns pods to Guaranteed, Burstable, or BestEffort QoS classes based on CPU and memory requests and limits. QoS is a useful operational signal, not a complete eviction-order specification.

Under memory pressure, kubelet ranking considers whether pod usage exceeds requests, pod priority, and relative usage. A higher-priority pod can receive stronger protection even when it is not Guaranteed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api:1.4.0
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              memory: 1Gi

This is an illustrative Burstable configuration, not a universal sizing recommendation. CPU limits are deliberately omitted: some latency-sensitive services prefer CPU requests without throttling-prone CPU limits, subject to platform policy. Memory requests and limits should be based on workload measurements and failure tolerance.

To inspect QoS:

kubectl get pods -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass,NODE:.spec.nodeName'

An OOMKilled container should be investigated separately:

kubectl get pod <pod-name> -n <namespace> -o json | jq '
  .status.containerStatuses[]? |
  {name, restartCount, lastTermination: .lastState.terminated}'

Disk pressure is not just a full root filesystem

Node storage pressure may come from container writable layers, images, logs, local ephemeral volumes, or inode exhaustion. Depending on the runtime and filesystem layout, these consume different filesystems.

On a node you are authorized to inspect:

df -h
df -i
sudo du -xhd1 /var/lib/kubelet 2>/dev/null
sudo du -xhd1 /var/log 2>/dev/null

Use the container runtime appropriate to the platform (crictl, podman, or a distribution-specific diagnostic tool). Do not indiscriminately delete files beneath kubelet or container-runtime directories: manual deletion can corrupt runtime state or remove data needed by running workloads.

Explicit ephemeral-storage requests and limits help make local disk consumption visible:

resources:
  requests:
    cpu: 100m
    memory: 256Mi
    ephemeral-storage: 1Gi
  limits:
    memory: 512Mi
    ephemeral-storage: 3Gi

Remember that ephemeral-storage accounting and enforcement depend on node filesystem configuration. An ephemeral-storage limit is not a replacement for log retention, application cleanup, or node capacity planning.

PID pressure and runaway process creation

A process-spawning bug, an aggressive job, or a compromised workload can exhaust the node’s process capacity. PID pressure may affect workloads unrelated to the original offender.

kubectl get node worker-1 -o jsonpath='{.status.conditions[?(@.type=="PIDPressure")].status}{"\n"}'

On the host, inspect process and thread counts with approved diagnostic access:

ps -eLf | wc -l
cat /proc/sys/kernel/pid_max

Consider runtime PID limits, workload process behavior, and dedicated node pools for workloads that create many processes. Do not assume CPU and memory quotas alone prevent PID exhaustion.

Eviction thresholds and node reservations

Kubelet supports hard and soft eviction thresholds. Hard thresholds trigger reclaim and eviction without a configured soft-threshold grace period; soft thresholds require a sustained breach before eviction. The kubelet can first reclaim node-level resources, such as unused images, before selecting pods.

Example for a self-managed Kubernetes cluster:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: "200Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
evictionSoft:
  memory.available: "500Mi"
evictionSoftGracePeriod:
  memory.available: "2m"
evictionMaxPodGracePeriod: 60

This is not a drop-in production configuration. Kubelet settings must be evaluated against the complete set of default thresholds: supplying custom thresholds may change how unspecified defaults are handled, depending on the kubelet configuration and MergeDefaultEvictionSettings behavior.

Reserve resources for the operating system and Kubernetes components using the distribution’s supported mechanisms. systemReserved and kubeReserved affect node allocatable accounting; enforcement requires careful cgroup configuration and validation.

OpenShift-specific guidance

OpenShift manages kubelet configuration through operators and supported cluster resources. Avoid editing kubelet files directly on Red Hat CoreOS nodes. Review the supported KubeletConfig and MachineConfigPool workflow for your OpenShift release, and test changes on a dedicated pool before broad rollout.

A change to node reservations or eviction thresholds can affect scheduling capacity, MachineConfigPool updates, and cluster maintenance. Treat it as a platform change, not a quick incident workaround.

Use namespace policy to prevent unsafe defaults

A LimitRange can inject default resource requests and limits into containers that omit them:

apiVersion: v1
kind: LimitRange
metadata:
  name: workload-defaults
  namespace: production
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 256Mi
      default:
        memory: 512Mi
      min:
        memory: 64Mi
      max:
        memory: 4Gi

A ResourceQuota constrains aggregate namespace allocations:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: workload-budget
  namespace: production
spec:
  hard:
    requests.cpu: "12"
    requests.memory: 24Gi
    limits.memory: 48Gi
    pods: "80"

Neither resource quotas nor limit ranges guarantee physical node capacity. A namespace can remain within quota while a node experiences pressure. Combine admission policy with right-sizing, placement controls, and cluster-level capacity management.

Monitor before users notice

Start with a small set of actionable signals:

SignalWhy it matters
kube_node_status_conditionReports node pressure conditions via kube-state-metrics
kube_pod_status_reason{reason="Evicted"}Indicates retained pod objects reporting eviction, if exposed by your kube-state-metrics version
node_memory_MemAvailable_bytesHost memory headroom; not identical to kubelet’s eviction calculation
node_filesystem_avail_bytesFilesystem capacity, with correct mountpoint/device selection
kubelet_evictionsKubelet eviction counters, where available and scraped
Container memory working set and OOM countersWorkload-level memory behavior

Example PromQL for a node pressure condition:

kube_node_status_condition{
  condition=~"MemoryPressure|DiskPressure|PIDPressure",
  status="true"
} == 1

Example for available host memory ratio:

node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes

Use this as a capacity indicator, not as an exact kubelet eviction-threshold calculation. For disk alerts, select the actual node, image, and container filesystems; checking only / can miss the filesystem responsible for eviction.

For OpenShift, prefer the built-in cluster monitoring stack and existing platform alerts before deploying duplicate node-level rules. Ensure custom alerts use metrics actually available through your monitoring endpoint.

Incident response: stabilize, diagnose, recover

When evictions are active, follow a controlled sequence:

  1. Identify the affected node and pressure signal. Collect node conditions, pod status messages, events, and relevant metric history.
  2. Protect critical capacity. Verify which applications and system services are affected. Avoid unplanned deletion of healthy pods.
  3. Find the resource consumer. Distinguish sustained workload growth from a sudden leak, runaway job, log explosion, or storage configuration issue.
  4. Apply the smallest safe mitigation. Examples include scaling out the node pool, reducing a noncritical workload, fixing log retention, or correcting resource requests.
  5. Verify recovery. Confirm pressure conditions clear, replacements become Ready, and workload-level service indicators recover.
  6. Capture follow-up actions. Update capacity forecasts, admission defaults, alerts, and runbooks.

Avoid treating the following as routine fixes:

# DANGEROUS as a blanket cleanup action:
# kubectl delete pods -A --field-selector=status.phase=Failed

# DANGEROUS on a live node without runtime-aware procedures:
# find /var/log/containers -name '*.log' -delete

Deleting all failed pods destroys diagnostic evidence and may affect unrelated workloads. Deleting runtime-managed log files can disrupt log collection and is not a substitute for supported rotation settings.

Design for resilience, not just eviction avoidance

Requests and limits are one layer of reliability. Production workloads also need sufficient replicas, topology-aware scheduling, readiness probes, and realistic spare capacity.

A PodDisruptionBudget is valuable during planned node drains but does not stop node-pressure eviction. Priority classes influence eviction and scheduling behavior, but assigning every workload a high priority defeats their purpose.

For stateful applications, examine storage locality, recovery time, quorum, and whether the workload can tolerate a node disappearing unexpectedly. A restarted pod is not necessarily an available service.

Operational checklist

[ ] Workload requests reflect observed steady-state and peak usage.
[ ] Memory limits are tested against application behavior and OOM risk.
[ ] Ephemeral-storage growth and log retention are controlled.
[ ] Node filesystem and inode capacity are monitored separately.
[ ] Critical workloads have appropriate priority and placement policies.
[ ] Namespace quotas and limit ranges are managed through GitOps.
[ ] Pressure alerts are validated against actual cluster metrics.
[ ] Incident runbooks preserve evidence before cleanup.
[ ] Kubelet configuration changes follow the platform-supported workflow.
[ ] Capacity plans account for maintenance and node failures.

Conclusion

Pod eviction is a symptom of a resource-management decision made under pressure. The durable response is to connect application resource profiles, node capacity, scheduling, and observability into a single operational model. When platform teams can explain why a pod was evicted and how the service recovered, eviction handling becomes a manageable reliability practice rather than an unpredictable production incident.

Related Articles

Stay ahead

Get the latest insights.

Insights on cloud, AI and open source — delivered to your inbox.

Email subscriptions are coming soon.