Kubernetes Leader Election: Reliable Active-Passive Workloads

Running three replicas of a controller improves availability, but it does not automatically make its actions safe. If all three replicas reconcile the same external system, schedule the same payment, or execute the same maintenance task, replication can create duplicate work rather than resilience.

Leader election coordinates those replicas so that one normally performs leader-only operations while others remain ready to take over. Kubernetes provides a native coordination primitive for this: the Lease resource in coordination.k8s.io/v1.

The important distinction is that leader election is a coordination mechanism, not an exactly-once execution guarantee. A robust design also handles stale leaders, partial failures, idempotency, and recovery.

Identify workloads that need a leader

Leader election is useful for controllers, reconciliation loops, cluster-wide schedulers, and singleton workers whose work cannot safely be performed concurrently. It is not automatically necessary for ordinary stateless HTTP services: those should generally remain active-active behind a Service.

Before adding an election protocol, ask whether the operation can instead be made idempotent or distributed safely across workers. For scheduled jobs, consider whether a Kubernetes CronJob with concurrencyPolicy meets the requirement; this controls overlapping Jobs from the same CronJob, but does not provide a universal exactly-once guarantee.

How Kubernetes Leases work

A Lease records the current holder, renewal timestamp, lease duration, and transition count. Candidates try to acquire or renew the same namespaced object through the Kubernetes API. Optimistic concurrency prevents simultaneous successful updates to the same resource version.

kubectl get leases -n platform-system
kubectl get lease controller-leader -n platform-system -o yaml

Relevant fields include spec.holderIdentity, spec.renewTime, spec.leaseDurationSeconds, and spec.leaseTransitions. A Lease is not a network lock enforced on the old process. When an isolated leader cannot renew, another replica may eventually acquire the Lease while the original process is still running. This is why leadership loss must terminate or fence leader-only work.

A useful mental model:

flowchart LR
  A[Replica A] --> L[(Kubernetes Lease)]
  B[Replica B] --> L
  C[Replica C] --> L
  L --> W[Current leader executes work]
  W --> E[External system]

Implement election with client-go

For Go applications, the maintained client-go/tools/leaderelection package handles Lease coordination. This implementation uses resourcelock.New, signal-aware cancellation, a bounded reconciliation cycle, and a fail-closed response to leadership loss. Pin compatible k8s.io/* module versions in go.mod and build the image through your normal CI pipeline.

Language-independent leader election: The example uses Go, but the Kubernetes coordination.k8s.io/v1 Lease API is not Go-specific. Python, Java, TypeScript, Rust, and other applications can implement the same coordination protocol. The application must still stop leader-only work safely and protect external side effects against stale leaders.

package main

import (
    "context"
    "errors"
    "log"
    "os"
    "os/signal"
    "syscall"
    "time"

    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/leaderelection"
    "k8s.io/client-go/tools/leaderelection/resourcelock"
)

const electionName = "controller-leader"

func main() {
    namespace, pod := os.Getenv("POD_NAMESPACE"), os.Getenv("POD_NAME")
    uid := os.Getenv("POD_UID")
    if namespace == "" || pod == "" || uid == "" {
        log.Fatal("POD_NAMESPACE, POD_NAME, and POD_UID are required")
    }

    // A recreated Pod must never reuse the identity of an older process.
    identity := pod + "_" + uid
    ctx, stop := signal.NotifyContext(
        context.Background(), syscall.SIGTERM, syscall.SIGINT,
    )
    defer stop()

    config, err := rest.InClusterConfig()
    if err != nil {
        log.Fatal(err)
    }
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        log.Fatal(err)
    }

    lock, err := resourcelock.New(
        resourcelock.LeasesResourceLock,
        namespace,
        electionName,
        clientset.CoreV1(),
        clientset.CoordinationV1(),
        resourcelock.ResourceLockConfig{Identity: identity},
    )
    if err != nil {
        log.Fatal(err)
    }

    leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
        Lock:            lock,
        LeaseDuration:   25 * time.Second,
        RenewDeadline:   18 * time.Second,
        RetryPeriod:     4 * time.Second,
        ReleaseOnCancel: false,
        Callbacks: leaderelection.LeaderCallbacks{
            OnStartedLeading: func(leaderCtx context.Context) {
                if err := serveAsLeader(leaderCtx); err != nil &&
                    !errors.Is(err, context.Canceled) {
                    log.Printf("leader work failed: %v", err)
                    // A real controller should escalate persistent errors.
                }
            },
            OnStoppedLeading: func() {
                // Leadership was lost: exit instead of risking stale writes.
                log.Print("leadership lost; stopping process")
                os.Exit(1)
            },
        },
    })
}

func serveAsLeader(ctx context.Context) error {
    // Reconcile once immediately, then periodically. Work must be
    // cancellable and safe to retry after a leader transition.
    timer := time.NewTicker(30 * time.Second)
    defer timer.Stop()

    for {
        if err := reconcile(ctx); err != nil {
            if ctx.Err() != nil {
                return ctx.Err()
            }
            log.Printf("reconciliation failed: %v", err)
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
        }
    }
}

func reconcile(ctx context.Context) error {
    // Replace with a bounded, idempotent operation. Pass ctx to every
    // downstream request; use fencing/transactions for non-idempotent writes.
    if err := ctx.Err(); err != nil {
        return err
    }
    log.Print("leader reconciliation requested")
    return nil
}

The Deployment must expose POD_UID via the Downward API, alongside POD_NAME and POD_NAMESPACE. The unique identity prevents a newly created Pod from being confused with a previous incarnation that had the same name.

- name: POD_UID
  valueFrom:
    fieldRef:
      fieldPath: metadata.uid

The sample intentionally leaves business reconciliation as a clearly marked integration point. Logging alone does not perform useful work; replace reconcile with a context-aware, idempotent operation and enforce fencing or transactional ownership where side effects cannot safely overlap.

LeaseDuration must exceed RenewDeadline, which must exceed RetryPeriod. The values shown are illustrative and require tuning against API latency and expected failover behavior. Client-go leader election does not provide fencing or exactly-once execution.

ReleaseOnCancel remains false: cancellation does not prove that every external side effect has stopped. Enabling it is appropriate only when the application can guarantee that leader-only work has terminated before the Lease is released.

Deploy replicas with least-privilege RBAC

The election needs a namespaced ServiceAccount with permission to create and maintain its specific Lease. Creating a new Lease requires collection-level create; get and update can be restricted to the named object.

apiVersion: v1
kind: Namespace
metadata:
  name: platform-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: controller
  namespace: platform-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: controller-election
  namespace: platform-system
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["create"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    resourceNames: ["controller-leader"]
    verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: controller-election
  namespace: platform-system
subjects:
  - kind: ServiceAccount
    name: controller
    namespace: platform-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: controller-election
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller
  namespace: platform-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: controller
  template:
    metadata:
      labels:
        app: controller
    spec:
      serviceAccountName: controller
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: controller
      containers:
        - name: controller
          image: registry.example.com/platform/controller:v1.0.0
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true

The image is illustrative and must contain the compiled controller. For real deployments, add probes that reflect the application’s actual health model, an appropriate seccomp profile, and image provenance controls. A follower is not unhealthy merely because it is not the leader; readiness should reflect whether the replica can serve its intended role.

Verify permissions before rollout:

kubectl auth can-i create leases -n platform-system \
  --as=system:serviceaccount:platform-system:controller
kubectl auth can-i update lease/controller-leader -n platform-system \
  --as=system:serviceaccount:platform-system:controller

Protect against split-brain and duplicate side effects

The most important production failure mode is a stale leader. Consider this sequence: replica A acquires the Lease, loses access to the API server, and pauses. Replica B acquires the expired Lease and starts processing. Replica A resumes and attempts to finish an external transaction. Both may now act on the same resource.

Use several layers of protection:

1. Cancellation: propagate the leader context to background workers, database calls, and reconciliation loops.
2. Process termination: stop promptly when leadership is lost; do not continue on stale cached state.
3. Idempotency: assign stable operation IDs so retries cannot create duplicate external effects.
4. Fencing: where the external system supports it, reject operations with an older leadership generation or fencing token.
5. Transactional ownership: use database constraints or conditional updates when the actual business invariant requires stronger guarantees.

Do not assume the Kubernetes Lease resourceVersion is a ready-made globally ordered fencing token for an external database. If you need strict fencing, design an explicit monotonic token or conditional-write protocol in the system that performs the protected operation.

Choose the right integration pattern

For Kubernetes operators using controller-runtime, prefer its built-in manager election rather than writing separate election code:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    LeaderElection:          true,
    LeaderElectionID:        "example-operator.platform.example.com",
    LeaderElectionNamespace: "platform-system",
})
if err != nil {
    return err
}
return mgr.Start(ctrl.SetupSignalHandler())

The snippet belongs inside an application function that returns error; import sigs.k8s.io/controller-runtime as ctrl. The exact Options structure and defaults depend on the pinned controller-runtime version. Check the generated operator’s RBAC for Lease permissions.

A sidecar election process is possible for applications that cannot integrate with the Kubernetes client directly, but it introduces another coordination boundary: the application must react correctly to leadership loss. Avoid obsolete example images or polling endpoints that report a leader without providing a reliable cancellation mechanism.

For Python or other languages, use a maintained library that supports coordination.k8s.io/v1 Leases and documents its failure behavior. Do not assume older ConfigMap-lock examples are compatible with current Kubernetes client libraries.

High availability is more than three replicas

Leader election does not create capacity. Place replicas across failure domains where available, provide enough CPU and memory headroom for a standby to become active, and plan for control-plane API interruptions.

A PodDisruptionBudget can reduce voluntary disruption during maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: controller
  namespace: platform-system
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: controller

This budget does not protect against node failures or guarantee an active leader. In particular, all three replicas might be healthy from Kubernetes’ perspective while none can renew the Lease because the API server is unavailable. Set availability objectives around time to resume useful work, not merely Pod readiness.

Observe the election, not just the Pods

At minimum, capture the current leader identity, leadership acquisition and loss events, election attempts, and the duration of leaderless intervals. Instrument the actual business operation as well: a stable leader that stops making progress is still an outage.

Useful operational commands:

kubectl get lease controller-leader -n platform-system \
  -o jsonpath='{.spec.holderIdentity}{"\n"}'
kubectl describe lease controller-leader -n platform-system
kubectl get pods -n platform-system -l app=controller -o wide
kubectl get events -n platform-system --sort-by=.lastTimestamp

If your application exports leader_election_is_leader{app="controller",pod="..."}, a diagnostic query can check the number of active claimants:

sum by (app) (leader_election_is_leader{app="controller"})

Expected steady-state value is one. An observed value of zero may indicate a leaderless interval, but missing metrics can also remove the series entirely. Alert separately on target availability (up) and use explicit application inventory or absent-series handling. A value above one is a serious investigation signal, though metric scrape timing can briefly overlap during handover.

For controller-runtime and client-go, inspect the metrics your pinned version actually exports rather than assuming a universal metric name. Record lease transitions and successful reconciliations as separate signals.

Test failover deliberately

A production readiness exercise should cover more than deleting the leader Pod:

Run tests in staging first, with external side effects directed to a controlled test system. Measure failover time and the number of duplicated or abandoned operations. These are more meaningful than confirming only that holderIdentity changes.

Kubernetes and OpenShift platform considerations

On OpenShift, use the standard Lease API and namespaced RBAC for application-level leader election. Do not modify control-plane component election settings to solve an application coordination problem. Operators managed by the platform may already implement leader election through their controller framework.

For restricted or disconnected environments, mirror application images into the approved registry and package the Deployment, ServiceAccount, Role, RoleBinding, and PodDisruptionBudget as a GitOps-managed unit. Keep leader-election authorization separate from the controller’s broader reconciliation permissions, and review which teams may alter the Lease or impersonate its ServiceAccount.

A namespace-scoped Lease is also an ownership boundary: different applications should use different Lease names, even if they run in the same namespace.

Conclusion

Kubernetes Leases provide a practical foundation for active-passive controllers and singleton workloads. client-go and controller-runtime handle much of the coordination machinery, but correctness still depends on what happens after leadership is lost.

For production systems, pair election with cancellable work, idempotent operations, fencing where necessary, least-privilege RBAC, failure-domain placement, and observable failover tests. The objective is not merely to elect a leader; it is to ensure the platform remains correct when the leader changes.