Monitoring Kubernetes etcd: Control Plane Health and Reliability
A practical guide to monitoring Kubernetes etcd health, Raft consensus, disk latency, database growth, and Prometheus alerts across self-managed and OpenShift clusters.

On this page
A healthy Kubernetes control plane depends on a healthy etcd cluster. etcd stores the API server’s persistent state, including workload definitions, configuration and Secrets. When its storage stalls, quorum becomes unstable, or the database approaches its backend quota, the symptoms often appear elsewhere: slow API requests, controllers falling behind, and deployments that fail to progress.
Effective etcd monitoring therefore means more than checking whether three processes are running. Platform teams need visibility into consensus, disk latency, database growth, client traffic, and operational recovery.
This guide builds a practical monitoring baseline for self-managed Kubernetes and explains the differences for OpenShift and managed Kubernetes services.
How etcd fits into Kubernetes
The Kubernetes API server is the primary client of etcd. Controllers and schedulers normally interact with the API server, not directly with etcd. etcd members use the Raft consensus protocol to agree on changes to cluster state.
A three-member cluster requires a quorum of two voting members. Losing one member may be tolerated; losing two means the cluster cannot commit new writes. A member can also be running but unhealthy because of slow storage or networking.
The diagram could not be displayed. Its source is available below.
Diagram source
flowchart LR
C[Controllers and scheduler] --> A[Kubernetes API server]
U[Clients and operators] --> A
A --> E1[etcd member 1]
E1 <--> E2[etcd member 2]
E1 <--> E3[etcd member 3]
E2 <--> E3The leader handles Raft proposal coordination, but clients can connect to other healthy members; the topology above is conceptual rather than a strict network routing diagram.
The metrics that matter
| Signal | Representative metric | What it reveals |
|---|---|---|
| Leader availability | etcd_server_has_leader | Whether a member sees a leader |
| Leadership churn | etcd_server_leader_changes_seen_total | Consensus instability |
| WAL durability latency | etcd_disk_wal_fsync_duration_seconds_bucket | Slow synchronous log writes |
| Backend commit latency | etcd_disk_backend_commit_duration_seconds_bucket | Slow database commits |
| Peer latency | etcd_network_peer_round_trip_time_seconds_bucket | Communication delays between members |
| Backend size | etcd_mvcc_db_total_size_in_bytes | Physical backend size including free pages |
| In-use database size | etcd_mvcc_db_total_size_in_use_in_bytes | Data still occupying live pages |
| Backend quota | etcd_server_quota_backend_bytes | Configured database quota |
| Failed proposals | etcd_server_proposals_failed_total | Raft proposal failures |
| Pending proposals | etcd_server_proposals_pending | Consensus backlog |
Metric availability and labels depend on the etcd and Kubernetes distribution versions. Check the live /metrics endpoint or Prometheus series before deploying alerts.
Establish the monitoring architecture
For a self-managed kubeadm cluster, etcd may run as a static pod on control-plane nodes. Prometheus can scrape its metrics endpoint using a correctly configured ServiceMonitor, PodMonitor, or static targets. The network path, endpoint binding, and TLS requirements must all match the actual deployment.
Do not expose etcd’s metrics endpoint on 0.0.0.0 simply to simplify scraping. Depending on the endpoint and version, metrics can disclose sensitive operational information. Prefer a private, access-controlled interface and use TLS where supported and required by your platform.
For OpenShift, etcd and its monitoring are platform-managed. Use the built-in cluster monitoring stack and supported observability interfaces rather than editing etcd static pod manifests or installing an overlapping scraper. The etcd operator owns lifecycle and configuration.
For GKE, EKS, and other managed control planes, direct etcd access may be unavailable. Use provider-exposed control-plane metrics, API server latency, audit events, and provider health indicators. Do not assume that self-managed etcd metrics or maintenance commands are supported.
Check the Prometheus series first
Before writing alerts, verify that the expected series exist:
etcd_server_has_leaderetcd_server_quota_backend_bytesetcd_disk_wal_fsync_duration_seconds_bucketUse the actual labels to isolate a specific cluster and job. In a central Thanos or federated Prometheus deployment, an unscoped query can accidentally combine multiple Kubernetes clusters.
Build actionable PromQL queries
WAL fsync latency, p99
histogram_quantile(
0.99,
sum by (cluster, instance, le) (
rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
)
)Backend commit latency, p99
histogram_quantile(
0.99,
sum by (cluster, instance, le) (
rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
)
)If your metrics do not carry a cluster label, remove it from the grouping; use the actual external labels instead.
Leadership changes in the last hour
increase(etcd_server_leader_changes_seen_total[1h])Evaluate leadership changes by member and cluster. A cluster-wide sum may count the same election from multiple members, so it is not necessarily the number of distinct elections.
Database quota utilization
100 * (
etcd_mvcc_db_total_size_in_bytes
/
etcd_server_quota_backend_bytes
)Pending proposals
etcd_server_proposals_pendingSustained pending proposals, rising disk latency, and frequent leader changes together are a stronger incident signal than any one metric in isolation.
Example Prometheus alerts
The following examples assume that the relevant metrics are scraped. Thresholds are illustrative starting points, not universal service-level objectives. Tune them to your cluster topology, baseline latency, and operational requirements.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: etcd-health-alerts
namespace: monitoring
spec:
groups:
- name: etcd-health
rules:
- alert: EtcdMemberHasNoLeader
expr: etcd_server_has_leader == 0
for: 1m
labels:
severity: critical
annotations:
summary: "etcd member {{ $labels.instance }} has no leader"
- alert: EtcdWALFsyncSlow
expr: |
histogram_quantile(
0.99,
sum by (instance, le) (
rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
)
) > 0.1
for: 10m
labels:
severity: warning
annotations:
summary: "High etcd WAL fsync latency on {{ $labels.instance }}"
- alert: EtcdBackendQuotaNearLimit
expr: |
etcd_mvcc_db_total_size_in_bytes
/ etcd_server_quota_backend_bytes > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "etcd backend exceeds 80% of quota on {{ $labels.instance }}"
- alert: EtcdFailedProposals
expr: increase(etcd_server_proposals_failed_total[15m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "etcd Raft proposals are failing on {{ $labels.instance }}"For OpenShift, create user-defined alert rules only through supported monitoring configuration and avoid duplicating existing platform-managed alerts. Check the actual namespace, rule selection, and Prometheus access model first.
Design a useful Grafana dashboard
A production dashboard should answer five questions immediately:
- Does every member see a leader? Display
etcd_server_has_leaderper instance. - Is consensus stable? Show leader changes, failed proposals, and pending proposals.
- Is storage keeping up? Plot WAL fsync and backend commit p99 by member.
- Is the backend approaching quota? Show total database size, in-use size, and quota utilization.
- Are members communicating normally? Plot peer round-trip time and network traffic.
Avoid reducing leader health to a single sum. A sum of three can obscure member identity, while a sum of two might mean either a missing target or a member reporting no leader. Include scrape health (up) and target inventory alongside etcd-specific signals.
If Grafana uses a shared Thanos datasource, scope dashboards by cluster and environment. Make sure the selected label names match your Prometheus external labels.
Validate health with etcdctl
For self-managed clusters, etcdctl provides a direct view of endpoints, members, and alarms. Use the appropriate TLS client certificates and follow your distribution’s documented access method.
export ETCDCTL_API=3
export ETCD_ENDPOINTS="https://127.0.0.1:2379"
etcdctl \
--endpoints="$ETCD_ENDPOINTS" \
--cacert=/path/to/ca.crt \
--cert=/path/to/client.crt \
--key=/path/to/client.key \
endpoint health
etcdctl \
--endpoints="$ETCD_ENDPOINTS" \
--cacert=/path/to/ca.crt \
--cert=/path/to/client.crt \
--key=/path/to/client.key \
endpoint status --write-out=table
etcdctl \
--endpoints="$ETCD_ENDPOINTS" \
--cacert=/path/to/ca.crt \
--cert=/path/to/client.crt \
--key=/path/to/client.key \
alarm listThe localhost endpoint is appropriate only when the command executes on a host or in a container that can actually reach that member. For a complete health assessment, query all intended members using valid endpoints and credentials.
On OpenShift, prefer supported operator status, cluster alerts, and vendor-documented diagnostic commands. Avoid creating arbitrary privileged CronJobs that mount control-plane certificates from the host.
Troubleshoot symptoms systematically
| Symptom | Investigate first |
|---|---|
| Frequent leader changes | Peer latency, packet loss, node restarts, CPU starvation, slow fsync |
| High WAL fsync latency | Storage IOPS, write latency, filesystem, contention, host health |
| Database approaching quota | Object churn, stale resources, compaction state, backend size |
| API requests slow | API server metrics, etcd request latency, network, control-plane load |
| Missing etcd metrics | Scrape target, TLS, authorization, endpoint binding, metric name |
| Failed proposals | Quorum, leadership stability, storage and peer connectivity |
Correlate events across members and components. A high API server request latency is not automatically an etcd problem, and an etcd leader change is not necessarily a user-visible outage.
Compaction and defragmentation are maintenance operations
etcd retains historical revisions until compaction. Compaction makes older revisions eligible for reclamation; defragmentation can reclaim space in the physical backend file.
These operations are not interchangeable, and neither should be executed automatically just because a metric crosses a threshold.
Before intervention:
- Confirm that the cluster has quorum and a healthy leader.
- Check whether the Kubernetes distribution already manages compaction and defragmentation.
- Verify that a current, restorable etcd backup exists and that recovery procedures are documented.
- Follow the distribution’s supported maintenance runbook.
- Schedule and sequence defragmentation carefully; it can temporarily block operations on the member being defragmented.
Do not run an indiscriminate etcdctl defrag against all members during a live incident. On OpenShift, use Red Hat-supported procedures rather than direct maintenance commands from a generic Kubernetes tutorial.
Monitoring does not replace recovery readiness
Even comprehensive observability cannot prevent every control-plane failure. A reliable operational model also needs:
- Scheduled, protected etcd backups appropriate to the Kubernetes distribution.
- Periodic restore exercises in a non-production environment.
- Clear incident ownership and escalation paths.
- Documented quorum-loss and member-replacement procedures.
- Storage performance and capacity planning for control-plane nodes.
- Alert routing that distinguishes transient anomalies from sustained incidents.
A snapshot that has never been restored is not a validated recovery strategy.
A practical platform engineering baseline
For production clusters, begin with the following controls:
| Area | Minimum operational expectation |
|---|---|
| Consensus | Monitor leader visibility, leadership churn, failed and pending proposals |
| Storage | Track WAL fsync and backend commit latency per member |
| Capacity | Alert on database quota utilization and abnormal growth |
| Networking | Track peer round-trip latency and connectivity |
| Scraping | Monitor missing targets and metrics gaps |
| Operations | Validate health before and after control-plane maintenance |
| Recovery | Maintain tested backups and distribution-specific runbooks |
Review thresholds against actual baselines. A fixed latency limit copied from another environment may be too lenient for a latency-sensitive cluster or unnecessarily noisy for a different infrastructure profile.
Conclusion
etcd is a foundational dependency of Kubernetes reliability. Its failures are often first noticed as API latency, stalled controllers, or failed workloads rather than an obvious database outage.
A useful monitoring strategy combines Raft consensus signals, storage latency, database capacity, endpoint health, and API server behavior. Just as importantly, it respects the operational boundaries of the Kubernetes distribution: what is appropriate for kubeadm may be unsupported on OpenShift or a managed control plane.
For platform engineering teams, the objective is not merely to collect more etcd metrics. It is to detect degradation early, identify the failing layer, and recover safely without turning routine maintenance into a control-plane incident.


