Technical Insight Explanation

Karpenter on EKS: Designing NodePools for Production

Learn how to design production-ready Karpenter NodePools on Amazon EKS, balancing capacity, Spot availability, workload disruption and cost.

Karpenter on EKS: Designing NodePools for Production
On this page

Kubernetes autoscaling is not simply about adding nodes when a cluster runs out of capacity. In production, it is about balancing workload requirements, infrastructure availability, cost efficiency, and operational stability.

Karpenter changes how Amazon EKS provisions compute capacity. Instead of scaling predefined node groups, it evaluates unschedulable workloads and selects suitable EC2 instances based on their scheduling requirements.

That flexibility is valuable. It also introduces architectural decisions that platform teams need to understand before enabling automated node provisioning and consolidation.

From Node Groups to Workload-Driven Provisioning

Traditional Cluster Autoscaler implementations typically scale existing node groups. Instance families, capacity types, and scaling boundaries are defined through those groups.

Karpenter approaches the problem differently. When pods cannot be scheduled, Karpenter evaluates their resource requests, node selectors, affinity rules, topology constraints, and tolerations. It then determines which available instance types could accommodate the workloads.

The provisioning process involves three important resources:

  • NodePool: Defines scheduling constraints, capacity preferences, resource limits, and disruption policies.
  • EC2NodeClass: Specifies AWS infrastructure details, including AMI selection, networking, instance roles, and storage.
  • NodeClaim: Represents an individual capacity request and tracks its provisioning lifecycle.

This separation allows platform teams to define workload policies independently of the underlying EC2 infrastructure configuration. Karpenter does not replace the Kubernetes scheduler: it provisions capacity, while the scheduler binds pods to nodes.

Designing NodePools Around Workloads

One common mistake is reproducing traditional node-group configurations as Karpenter NodePools. Restricting a NodePool to a handful of EC2 instance types may appear predictable, but reduces Karpenter’s ability to select available and cost-effective capacity.

A more flexible configuration uses instance categories, generations, architectures, and capacity types:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: general-purpose
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gte
          values: ["6"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
  limits:
    cpu: "500"
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "10%"

This illustrative NodePool allows several instance families and generations rather than hard-coding a short list. The general-purpose EC2NodeClass must be created separately with valid IAM, subnet, security group, and AMI configuration. This manifest is not a complete Karpenter installation.

The 10% disruption budget is an example, not a universal production recommendation. It constrains applicable voluntary disruptions, but does not prevent all node terminations. Review the upstream disruption documentation for budget rounding, overlapping budgets, and exceptions.

The objective is not unrestricted provisioning: define the smallest set of constraints that accurately represents workload requirements. CPU-intensive, memory-intensive, and latency-sensitive services may require different policies.

Spot Capacity Is a Scheduling Strategy

Spot instances can reduce infrastructure expenditure, but Spot is interruptible capacity. Applications must tolerate node replacement, termination, and potentially temporary capacity shortages.

A production Spot strategy should account for diversification across instance families and availability zones, replica distribution, graceful shutdown, PodDisruptionBudgets, and appropriate On-Demand capacity for workloads that cannot tolerate interruption.

When both Spot and On-Demand are eligible, Karpenter generally prefers Spot, subject to availability and its capacity selection logic. This does not guarantee a specific price or an immediate On-Demand fallback in every scheduling situation. Weighted NodePools likewise express preferences, not an unconditional failover contract.

Interruption handling requires infrastructure configuration. In an AWS deployment, Karpenter’s interruption processing uses an SQS queue receiving relevant EventBridge events. Simply adding spot to a NodePool does not configure that pipeline. Validate queue configuration and IAM permissions as part of installation.

Consolidation: Where Cost Optimization Meets Availability

Karpenter can remove empty nodes and replace or remove underutilized capacity when it can reschedule workloads. Infrastructure savings may be meaningful, but consolidation can also trigger pod termination, application restarts, cache invalidation, and temporary reductions in service capacity.

PodDisruptionBudgets constrain voluntary evictions. NodePool disruption budgets constrain certain Karpenter-initiated disruptions. The karpenter.sh/do-not-disrupt annotation can provide additional protection against voluntary disruption, subject to documented exceptions.

None of these mechanisms guarantees protection from EC2 failures, Spot interruptions, or every type of node termination. In particular, distinguish voluntary consolidation from expiration, drift, and involuntary disruption when designing policy.

Start with conservative consolidation delays and budgets, then adjust them using measured application behavior. A development environment and a stateful production workload should rarely share identical disruption policies.

Availability Zones and Topology Constraints

Karpenter provisions capacity according to the scheduling constraints expressed by workloads. Kubernetes topology spread constraints are therefore central to high availability.

Distributing replicas across availability zones can reduce exposure to a zone-level failure. However, strict topology constraints can leave pods pending if suitable capacity is unavailable in a particular zone. Validate both normal scale-out and capacity-shortage scenarios rather than assuming that zone diversity will always be achievable.

Operating Karpenter in Production

A working NodePool is only the beginning. Production operations require visibility into provisioning decisions, node lifecycle events, disruption attempts, and scheduling failures.

kubectl get nodepools
kubectl get ec2nodeclasses
kubectl get nodeclaims
kubectl describe nodepool general-purpose
kubectl get nodes -L karpenter.sh/nodepool,karpenter.sh/capacity-type
kubectl logs -n kube-system \
  -l app.kubernetes.io/name=karpenter \
  -c controller --tail=100

Monitoring should answer operational questions: How long do pods remain pending? How frequently are nodes replaced? Is consolidation causing excessive restarts? Are NodePool limits, EC2 quotas, subnet IP capacity, or availability-zone shortages blocking provisioning?

Establish an explicit AMI upgrade strategy. Avoid automatically tracking @latest in production without a controlled validation and rollout process; a newly resolved AMI can induce node drift and replacement. Ensure Karpenter’s controller has reliable capacity independent of the nodes it manages.

Common Production Failure Modes

  • Overly restrictive instance selection. Narrow lists reduce provisioning flexibility and may increase pending pod time.

  • Aggressive consolidation. Very short delays can generate unnecessary infrastructure churn for variable workloads.

  • Incomplete disruption policies. Applications without replication, termination handling, or appropriate budgets can experience avoidable outages.

  • Uncontrolled AMI changes. Automatically selecting newly published AMIs without validation increases operational uncertainty.

  • Insufficient capacity planning. NodePool limits, EC2 service quotas, subnet IP availability, and zone constraints can block scale-out even with otherwise valid configuration.

Karpenter is part of the platform’s capacity-management architecture, not merely another autoscaling controller.

Validation Before Production

Validate the example against the installed Karpenter CRDs and an existing EC2NodeClass in a non-production EKS cluster:

kubectl apply --dry-run=server -f nodepool.yaml
kubectl get ec2nodeclass general-purpose
kubectl apply -f nodepool.yaml
kubectl get nodepool general-purpose
kubectl describe nodepool general-purpose

Then deploy a workload requiring additional capacity, inspect its NodeClaim and node assignment, and test consolidation and interruption handling. These runtime tests have not been performed for this article. The example should not be presented as empirically validated until they are completed.

Conclusion

Karpenter gives EKS platform teams greater control over how compute capacity is selected, provisioned, and replaced. Its value comes from combining flexible instance selection with workload-aware scheduling and controlled disruption.

Production-ready NodePools should reflect application requirements, failure tolerance, and operational objectives—not merely infrastructure cost targets. Achieving that requires deliberate configuration, realistic failure testing, and continuous operational validation.

Building a more efficient Kubernetes platform? ape factory helps engineering teams design, operate, and optimize Kubernetes platforms, including workload scheduling, infrastructure automation, reliability, and cost management.

References

Related Articles

Stay ahead

Get the latest insights.

Insights on cloud, AI and open source — delivered to your inbox.

Email subscriptions are coming soon.