Technical Insight Explanation

Kubernetes Topology-Aware Routing: Performance, Costs, and Failure Modes

Configure Kubernetes Service trafficDistribution, compare zone preference with strict node-local routing, and validate locality, fallback, and failure behavior.

Kubernetes Topology-Aware Routing: Performance, Costs, and Failure Modes
On this page

Kubernetes can distribute Service traffic across healthy endpoints in different availability zones. That is useful for resilience, but it can also add network hops, latency, and cross-zone data-transfer charges.

Topology-aware routing is a preference, not a guarantee. It can keep traffic near its source, but its effectiveness depends on endpoint placement, client distribution, and the Service proxy implementation.

This guide targets Kubernetes v1.34 or later with a Service proxy that supports trafficDistribution. Verify support in your CNI or replacement Service dataplane before relying on the behavior.

Choose the right traffic policy

Three mechanisms are often confused:

MechanismBehaviorFallback
trafficDistribution: PreferSameZonePrefer ready endpoints in the client’s zoneCluster-wide when no same-zone endpoint exists
trafficDistribution: PreferSameNodePrefer ready endpoints on the client’s nodeSame zone, then cluster-wide
internalTrafficPolicy: LocalUse only node-local endpoints for internal trafficNo fallback; traffic fails without a local endpoint

PreferSameZone and PreferSameNode were introduced as beta values in Kubernetes v1.34, enabled by default through the PreferSameTrafficDistribution feature gate. PreferClose is the older same-zone alias and should be treated as legacy terminology. Check the target release and Service dataplane before relying on either preference.

A strict Local traffic policy takes precedence over a locality preference for internal traffic. Do not combine them expecting zone fallback.

How endpoint locality works

The EndpointSlice controller associates endpoints with topology information and publishes routing hints. For PreferSameZone, hints identify an endpoint’s own zone rather than using the proportional allocatable-CPU heuristic of the older topology-mode: Auto mode. PreferSameNode adds node-level preference, with same-zone and then cluster-wide fallback when local endpoints are unavailable. A compatible Service proxy must understand the relevant hints; support is implementation- and version-specific.

Diagram source
flowchart LR
    A[Client Pod in Zone A] --> B[Service virtual IP]
    B --> C{Ready endpoint in Zone A?}
    C -->|Yes| D[Prefer Zone A endpoint]
    C -->|No| E[Use ready endpoint in another zone]

This diagram describes PreferSameZone, not strict traffic isolation. It also abstracts away implementation details: packet forwarding and connection selection depend on the cluster dataplane.

Deploy a zone preferring Service

Start with a backend whose Pods are distributed across zones. The following example assumes the production namespace exists, a container image listening on port 8080, and worker nodes labeled with topology.kubernetes.io/zone.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-backend
  namespace: production
spec:
  replicas: 6
  selector:
    matchLabels:
      app: api-backend
  template:
    metadata:
      labels:
        app: api-backend
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: api-backend
      containers:
        - name: api
          image: registry.example.com/api-backend:v1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: api-backend
  namespace: production
spec:
  selector:
    app: api-backend
  ports:
    - name: http
      port: 80
      targetPort: 8080
  trafficDistribution: PreferSameZone

Implementation note: registry.example.com/api-backend:v1 is a placeholder, not a public runnable image. Replace it with your own backend exposing /ready on port 8080. ScheduleAnyway encourages spreading but does not guarantee one endpoint per zone; scheduling capacity and other constraints still apply.

trafficDistribution influences Service traffic, not Pod scheduling. Use topology spread constraints or zone-specific Deployments to place the backends appropriately.

Prefer the same node or require it?

For supported clusters, PreferSameNode tries node-local endpoints first, then zone-local endpoints, then other endpoints. It suits latency-sensitive services where availability matters more than strict locality:

apiVersion: v1
kind: Service
metadata:
  name: cache-preferred
  namespace: production
spec:
  selector:
    app: cache
  ports:
    - port: 6379
      targetPort: 6379
  trafficDistribution: PreferSameNode

By contrast, internalTrafficPolicy: Local is a hard constraint:

apiVersion: v1
kind: Service
metadata:
  name: cache-local-only
  namespace: production
spec:
  selector:
    app: cache
  ports:
    - port: 6379
      targetPort: 6379
  internalTrafficPolicy: Local

A client on a node without a ready local cache endpoint cannot reach this Service through its ClusterIP, even when healthy cache Pods exist elsewhere. Use it only when every client node is expected to host a backend, for example through an appropriately configured DaemonSet.

Understand fallback and overload

PreferSameZone falls back to cluster-wide routing when the client’s zone has no ready endpoint. It does not automatically spill traffic to other zones merely because local endpoints are busy. A traffic hotspot in Zone A can therefore overload Zone A’s backends while spare capacity exists in Zone B.

The older service.kubernetes.io/topology-mode: Auto annotation behaves differently: it uses CPU-capacity-based endpoint allocation and may disable hints when safeguards cannot be satisfied. If both the annotation and trafficDistribution are configured, topology-mode: Auto takes precedence, according to the Kubernetes v1.34 documentation. Avoid mixing them during migration; remove the legacy annotation when you intend trafficDistribution to control routing.

The removed topologyKeys Service field is not a modern alternative; it was removed in Kubernetes v1.22.

Validate the behavior, not just the YAML

First inspect the topology and placement:

kubectl get nodes -L topology.kubernetes.io/zone
kubectl -n production get pods -l app=api-backend -o wide
kubectl -n production get svc api-backend -o yaml

Then inspect the EndpointSlices and any published hints:

kubectl -n production get endpointslices \
  -l kubernetes.io/service-name=api-backend -o yaml

Check endpoint zone, nodeName, conditions.ready, and hints. With PreferSameZone, look for hints.forZones; with PreferSameNode, verify the node-level hints supported by your target release. The dataplane must honor those hints: Kubernetes API acceptance alone does not establish routing behavior.

For a meaningful test, run client Pods pinned to nodes in at least two zones. Send repeated requests to the Service and record the backend Pod and zone serving each request. Compare with trafficDistribution unset, then repeat after making all backends in one zone unready. Use new connections for each observation; persistent connections can conceal changes in backend selection.

Test these cases explicitly:

  1. Same-zone endpoints available: measure the proportion of requests served locally.
  2. No ready same-zone endpoint: verify cluster-wide fallback for PreferSameZone.
  3. Uneven client load: watch backend saturation, latency, and error rates by zone.
  4. internalTrafficPolicy: Local without a node-local endpoint: verify the expected failure.
  5. Unsupported or alternate dataplane: establish whether it honors the configured preference.

Hints and configuration alone do not prove reduced cross-zone traffic. Confirm the actual traffic path using application telemetry or network flow data, then compare cloud transfer charges before and after rollout.

Production trade-offs

Locality can reduce inter-zone data transfer, but it can also concentrate traffic on a smaller backend pool. Plan for autoscaling delays, uneven demand, Pod disruptions, and changing zone capacity. Keep readiness probes meaningful: they determine whether an endpoint should receive traffic, not whether it has spare capacity.

For critical applications, roll out locality preferences gradually and retain a rollback path. Track per-zone request volume, p95/p99 latency, errors, backend utilization, and cross-zone bytes. Use strict Local policies only when their availability trade-off is deliberate.

At ape factory, we treat traffic locality as a measurable platform-engineering optimization—not a YAML-only cost-saving feature. Start with the failure mode you can tolerate, then measure whether locality improves the system.

Further reading

Related Articles

Stay ahead

Get the latest insights.

Insights on cloud, AI and open source — delivered to your inbox.

Email subscriptions are coming soon.