Technical Insight How-to Guide

Monitoring Kubernetes etcd: Control Plane Health and Reliability

A practical guide to monitoring Kubernetes etcd health, Raft consensus, disk latency, database growth, and Prometheus alerts across self-managed and OpenShift clusters.

Monitoring Kubernetes etcd: Control Plane Health and Reliability
On this page

A healthy Kubernetes control plane depends on a healthy etcd cluster. etcd stores the API server’s persistent state, including workload definitions, configuration and Secrets. When its storage stalls, quorum becomes unstable, or the database approaches its backend quota, the symptoms often appear elsewhere: slow API requests, controllers falling behind, and deployments that fail to progress.

Effective etcd monitoring therefore means more than checking whether three processes are running. Platform teams need visibility into consensus, disk latency, database growth, client traffic, and operational recovery.

This guide builds a practical monitoring baseline for self-managed Kubernetes and explains the differences for OpenShift and managed Kubernetes services.

How etcd fits into Kubernetes

The Kubernetes API server is the primary client of etcd. Controllers and schedulers normally interact with the API server, not directly with etcd. etcd members use the Raft consensus protocol to agree on changes to cluster state.

A three-member cluster requires a quorum of two voting members. Losing one member may be tolerated; losing two means the cluster cannot commit new writes. A member can also be running but unhealthy because of slow storage or networking.

Diagram source
flowchart LR
    C[Controllers and scheduler] --> A[Kubernetes API server]
    U[Clients and operators] --> A
    A --> E1[etcd member 1]
    E1 <--> E2[etcd member 2]
    E1 <--> E3[etcd member 3]
    E2 <--> E3

The leader handles Raft proposal coordination, but clients can connect to other healthy members; the topology above is conceptual rather than a strict network routing diagram.

The metrics that matter

SignalRepresentative metricWhat it reveals
Leader availabilityetcd_server_has_leaderWhether a member sees a leader
Leadership churnetcd_server_leader_changes_seen_totalConsensus instability
WAL durability latencyetcd_disk_wal_fsync_duration_seconds_bucketSlow synchronous log writes
Backend commit latencyetcd_disk_backend_commit_duration_seconds_bucketSlow database commits
Peer latencyetcd_network_peer_round_trip_time_seconds_bucketCommunication delays between members
Backend sizeetcd_mvcc_db_total_size_in_bytesPhysical backend size including free pages
In-use database sizeetcd_mvcc_db_total_size_in_use_in_bytesData still occupying live pages
Backend quotaetcd_server_quota_backend_bytesConfigured database quota
Failed proposalsetcd_server_proposals_failed_totalRaft proposal failures
Pending proposalsetcd_server_proposals_pendingConsensus backlog

Metric availability and labels depend on the etcd and Kubernetes distribution versions. Check the live /metrics endpoint or Prometheus series before deploying alerts.

Establish the monitoring architecture

For a self-managed kubeadm cluster, etcd may run as a static pod on control-plane nodes. Prometheus can scrape its metrics endpoint using a correctly configured ServiceMonitor, PodMonitor, or static targets. The network path, endpoint binding, and TLS requirements must all match the actual deployment.

Do not expose etcd’s metrics endpoint on 0.0.0.0 simply to simplify scraping. Depending on the endpoint and version, metrics can disclose sensitive operational information. Prefer a private, access-controlled interface and use TLS where supported and required by your platform.

For OpenShift, etcd and its monitoring are platform-managed. Use the built-in cluster monitoring stack and supported observability interfaces rather than editing etcd static pod manifests or installing an overlapping scraper. The etcd operator owns lifecycle and configuration.

For GKE, EKS, and other managed control planes, direct etcd access may be unavailable. Use provider-exposed control-plane metrics, API server latency, audit events, and provider health indicators. Do not assume that self-managed etcd metrics or maintenance commands are supported.

Check the Prometheus series first

Before writing alerts, verify that the expected series exist:

etcd_server_has_leader
etcd_server_quota_backend_bytes
etcd_disk_wal_fsync_duration_seconds_bucket

Use the actual labels to isolate a specific cluster and job. In a central Thanos or federated Prometheus deployment, an unscoped query can accidentally combine multiple Kubernetes clusters.

Build actionable PromQL queries

WAL fsync latency, p99

histogram_quantile(
  0.99,
  sum by (cluster, instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

Backend commit latency, p99

histogram_quantile(
  0.99,
  sum by (cluster, instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

If your metrics do not carry a cluster label, remove it from the grouping; use the actual external labels instead.

Leadership changes in the last hour

increase(etcd_server_leader_changes_seen_total[1h])

Evaluate leadership changes by member and cluster. A cluster-wide sum may count the same election from multiple members, so it is not necessarily the number of distinct elections.

Database quota utilization

100 * (
  etcd_mvcc_db_total_size_in_bytes
  /
  etcd_server_quota_backend_bytes
)

Pending proposals

etcd_server_proposals_pending

Sustained pending proposals, rising disk latency, and frequent leader changes together are a stronger incident signal than any one metric in isolation.

Example Prometheus alerts

The following examples assume that the relevant metrics are scraped. Thresholds are illustrative starting points, not universal service-level objectives. Tune them to your cluster topology, baseline latency, and operational requirements.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: etcd-health-alerts
  namespace: monitoring
spec:
  groups:
    - name: etcd-health
      rules:
        - alert: EtcdMemberHasNoLeader
          expr: etcd_server_has_leader == 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "etcd member {{ $labels.instance }} has no leader"

        - alert: EtcdWALFsyncSlow
          expr: |
            histogram_quantile(
              0.99,
              sum by (instance, le) (
                rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
              )
            ) > 0.1
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High etcd WAL fsync latency on {{ $labels.instance }}"

        - alert: EtcdBackendQuotaNearLimit
          expr: |
            etcd_mvcc_db_total_size_in_bytes
              / etcd_server_quota_backend_bytes > 0.8
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "etcd backend exceeds 80% of quota on {{ $labels.instance }}"

        - alert: EtcdFailedProposals
          expr: increase(etcd_server_proposals_failed_total[15m]) > 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "etcd Raft proposals are failing on {{ $labels.instance }}"

For OpenShift, create user-defined alert rules only through supported monitoring configuration and avoid duplicating existing platform-managed alerts. Check the actual namespace, rule selection, and Prometheus access model first.

Design a useful Grafana dashboard

A production dashboard should answer five questions immediately:

  1. Does every member see a leader? Display etcd_server_has_leader per instance.
  2. Is consensus stable? Show leader changes, failed proposals, and pending proposals.
  3. Is storage keeping up? Plot WAL fsync and backend commit p99 by member.
  4. Is the backend approaching quota? Show total database size, in-use size, and quota utilization.
  5. Are members communicating normally? Plot peer round-trip time and network traffic.

Avoid reducing leader health to a single sum. A sum of three can obscure member identity, while a sum of two might mean either a missing target or a member reporting no leader. Include scrape health (up) and target inventory alongside etcd-specific signals.

If Grafana uses a shared Thanos datasource, scope dashboards by cluster and environment. Make sure the selected label names match your Prometheus external labels.

Validate health with etcdctl

For self-managed clusters, etcdctl provides a direct view of endpoints, members, and alarms. Use the appropriate TLS client certificates and follow your distribution’s documented access method.

export ETCDCTL_API=3
export ETCD_ENDPOINTS="https://127.0.0.1:2379"

etcdctl \
  --endpoints="$ETCD_ENDPOINTS" \
  --cacert=/path/to/ca.crt \
  --cert=/path/to/client.crt \
  --key=/path/to/client.key \
  endpoint health

etcdctl \
  --endpoints="$ETCD_ENDPOINTS" \
  --cacert=/path/to/ca.crt \
  --cert=/path/to/client.crt \
  --key=/path/to/client.key \
  endpoint status --write-out=table

etcdctl \
  --endpoints="$ETCD_ENDPOINTS" \
  --cacert=/path/to/ca.crt \
  --cert=/path/to/client.crt \
  --key=/path/to/client.key \
  alarm list

The localhost endpoint is appropriate only when the command executes on a host or in a container that can actually reach that member. For a complete health assessment, query all intended members using valid endpoints and credentials.

On OpenShift, prefer supported operator status, cluster alerts, and vendor-documented diagnostic commands. Avoid creating arbitrary privileged CronJobs that mount control-plane certificates from the host.

Troubleshoot symptoms systematically

SymptomInvestigate first
Frequent leader changesPeer latency, packet loss, node restarts, CPU starvation, slow fsync
High WAL fsync latencyStorage IOPS, write latency, filesystem, contention, host health
Database approaching quotaObject churn, stale resources, compaction state, backend size
API requests slowAPI server metrics, etcd request latency, network, control-plane load
Missing etcd metricsScrape target, TLS, authorization, endpoint binding, metric name
Failed proposalsQuorum, leadership stability, storage and peer connectivity

Correlate events across members and components. A high API server request latency is not automatically an etcd problem, and an etcd leader change is not necessarily a user-visible outage.

Compaction and defragmentation are maintenance operations

etcd retains historical revisions until compaction. Compaction makes older revisions eligible for reclamation; defragmentation can reclaim space in the physical backend file.

These operations are not interchangeable, and neither should be executed automatically just because a metric crosses a threshold.

Before intervention:

  • Confirm that the cluster has quorum and a healthy leader.
  • Check whether the Kubernetes distribution already manages compaction and defragmentation.
  • Verify that a current, restorable etcd backup exists and that recovery procedures are documented.
  • Follow the distribution’s supported maintenance runbook.
  • Schedule and sequence defragmentation carefully; it can temporarily block operations on the member being defragmented.

Do not run an indiscriminate etcdctl defrag against all members during a live incident. On OpenShift, use Red Hat-supported procedures rather than direct maintenance commands from a generic Kubernetes tutorial.

Monitoring does not replace recovery readiness

Even comprehensive observability cannot prevent every control-plane failure. A reliable operational model also needs:

  • Scheduled, protected etcd backups appropriate to the Kubernetes distribution.
  • Periodic restore exercises in a non-production environment.
  • Clear incident ownership and escalation paths.
  • Documented quorum-loss and member-replacement procedures.
  • Storage performance and capacity planning for control-plane nodes.
  • Alert routing that distinguishes transient anomalies from sustained incidents.

A snapshot that has never been restored is not a validated recovery strategy.

A practical platform engineering baseline

For production clusters, begin with the following controls:

AreaMinimum operational expectation
ConsensusMonitor leader visibility, leadership churn, failed and pending proposals
StorageTrack WAL fsync and backend commit latency per member
CapacityAlert on database quota utilization and abnormal growth
NetworkingTrack peer round-trip latency and connectivity
ScrapingMonitor missing targets and metrics gaps
OperationsValidate health before and after control-plane maintenance
RecoveryMaintain tested backups and distribution-specific runbooks

Review thresholds against actual baselines. A fixed latency limit copied from another environment may be too lenient for a latency-sensitive cluster or unnecessarily noisy for a different infrastructure profile.

Conclusion

etcd is a foundational dependency of Kubernetes reliability. Its failures are often first noticed as API latency, stalled controllers, or failed workloads rather than an obvious database outage.

A useful monitoring strategy combines Raft consensus signals, storage latency, database capacity, endpoint health, and API server behavior. Just as importantly, it respects the operational boundaries of the Kubernetes distribution: what is appropriate for kubeadm may be unsupported on OpenShift or a managed control plane.

For platform engineering teams, the objective is not merely to collect more etcd metrics. It is to detect degradation early, identify the failing layer, and recover safely without turning routine maintenance into a control-plane incident.

Related Articles

Stay ahead

Get the latest insights.

Insights on cloud, AI and open source — delivered to your inbox.

Email subscriptions are coming soon.