Kubernetes Secrets Management: Production Security Patterns
Design production Kubernetes secrets management with encryption at rest, workload identity, External Secrets Operator, least privilege, GitOps and reliable rotation.

On this page
A Kubernetes Secret is an API object, not a complete secrets-management system. It provides a standard interface for delivering credentials to workloads, but does not establish who should own those credentials, how they are generated, how they rotate, or how they are recovered after an incident.
Production platforms need to answer a more demanding question: how can a workload obtain the credentials it needs without turning the cluster, deployment repository, or CI pipeline into a long-lived credential repository?
This article examines the boundaries between Kubernetes Secrets, external secret managers, workload identity, GitOps, and application lifecycle management. The objective is a defensible operating model, not a collection of YAML recipes.
Start with the threat model
Kubernetes stores Secret values as base64-encoded fields in its API representation. Base64 is encoding, not encryption. Whether the underlying values are encrypted in etcd depends on the cluster’s encryption-at-rest configuration and managed-service behavior.
Encryption at rest addresses one threat: unauthorized access to stored data or backups. It does not prevent an authorized Kubernetes API client from reading a Secret, a privileged workload from extracting credentials, or an application from logging them. Kubernetes administrators and principals with sufficiently broad workload-creation privileges can often obtain secrets indirectly.
For production design, distinguish at least four exposure paths:
- Control-plane storage: etcd, snapshots, encryption keys and backup systems.
- Kubernetes API authorization: direct Secret reads and indirect access through workload creation or modification.
- Workload runtime: mounted files, environment variables, process memory, logs and debugging interfaces.
- Delivery pipeline: CI variables, Git history, deployment templates and controller credentials.
Controls should match these paths. A secret encrypted in Git may still be readable in the cluster; an external secret manager may still synchronize plaintext into a Kubernetes Secret.
Choose the right delivery model
There is no single mechanism that fits every workload. Three common patterns solve different problems.
| Pattern | What it provides | Main operational trade-off |
|---|---|---|
| Kubernetes Secret | Native API object, compatible with standard workloads | Secret data exists inside the cluster and must be protected there |
| External Secrets Operator (ESO) | Synchronizes values from an external provider into Kubernetes Secrets | Adds a controller and synchronization dependency; target Secrets still exist in Kubernetes |
| Secrets Store CSI Driver | Mounts provider-managed secrets into pods through a CSI volume | Depends on driver/provider integration and application file-consumption behavior; optional sync creates Kubernetes Secrets |
Sealed Secrets and SOPS address a different boundary: how encrypted configuration is stored and reviewed in Git. They are not substitutes for authorization, rotation, or runtime protection. Sealed Secrets also requires careful backup of the controller’s private keys; losing those keys can make existing sealed manifests undecryptable.
For a platform with an established cloud secret manager, ESO is often a pragmatic default when applications already expect Kubernetes Secret objects. For applications that support external identity or direct provider access, avoiding static credentials entirely may be preferable.
An External Secrets architecture for EKS
Consider an EKS workload whose database credentials are stored in AWS Secrets Manager. ESO reads the remote secret and materializes a namespaced Kubernetes Secret consumed by the application.
The diagram could not be displayed. Its source is available below.
Diagram source
flowchart LR
A[Application Pod] -->|Reads mounted Secret| K[Kubernetes Secret]
E[External Secrets Operator] -->|Reconciles| K
E -->|Assumes role via SA JWT| S_STS[AWS STS]
S_STS -->|Temporary credentials| E
E -->|Authorized read| S[AWS Secrets Manager]
S -->|Secret value| EThe security boundary is important: the IAM role grants the controller access to the remote value; Kubernetes RBAC and namespace isolation govern the resulting Secret. These are separate authorization systems.
Prerequisites: a supported ESO installation and CRDs; an EKS identity integration (for example IRSA, configured for the intended ServiceAccount); an IAM role limited to the required Secrets Manager resources; and network access to the AWS API. The following is an illustrative SecretStore and ExternalSecret, not a complete IAM or ESO installation guide.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: application-secrets
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: eu-central-1
auth:
jwt:
serviceAccountRef:
name: eso-reader
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: database-credentials
namespace: production
spec:
refreshPolicy: Periodic
refreshInterval: 15m
secretStoreRef:
name: application-secrets
kind: SecretStore
target:
name: database-credentials
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: production/application/database
property: passwordThe eso-reader ServiceAccount must exist in the production namespace and be configured for IRSA with an annotated AWS IAM role ARN. By setting auth.jwt.serviceAccountRef, the controller retrieves the projected OIDC token of eso-reader and calls sts:AssumeRoleWithWebIdentity to fetch temporary credentials scoped strictly to that tenant. This prevents the ESO controller itself from needing cluster-wide AWS access. The example assumes that the remote value is a JSON object with a password property; IAM trust policies and secret structures must be validated in the target environment.
Prefer a namespaced SecretStore when a team only needs access to its own secrets. A ClusterSecretStore can be appropriate for shared infrastructure, but increases the importance of explicit access boundaries and admission controls.
Workload identity instead of long-lived cloud keys
Putting an AWS access key and secret access key into a Kubernetes Secret to let ESO retrieve other secrets creates a bootstrap credential that also needs protection and rotation.
On EKS, use an appropriate workload identity mechanism according to the controller’s supported authentication configuration. While both IAM Roles for Service Accounts (IRSA) and EKS Pod Identity avoid long-lived IAM user keys, they differ under the hood: IRSA relies on OIDC-projected ServiceAccount tokens exchanged via sts:AssumeRoleWithWebIdentity (which matches ESO’s auth.jwt.serviceAccountRef), whereas EKS Pod Identity provides credentials via a local node agent endpoint. Ensure the SecretStore auth block matches the identity model implemented on the cluster.
Workload identity does not eliminate authorization design. Bind the identity to the intended controller ServiceAccount, restrict allowed secret ARNs and actions, and monitor remote secret access. Avoid a single highly privileged identity shared by unrelated namespaces.
On OpenShift or self-managed Kubernetes, the available federation and provider integration will differ. Do not copy an EKS-specific identity manifest into another platform and assume equivalent behavior.
Encryption at rest is necessary but insufficient
For self-managed Kubernetes, encryption at rest is configured at the API-server layer using an EncryptionConfiguration. Provider choice, key generation, distribution across API servers, backups, and key rotation require an explicit operating procedure. Never commit live encryption keys to a repository or use example keys in production.
For managed Kubernetes, first establish what the provider encrypts by default, whether customer-managed keys are supported, and which controls are exposed. EKS and OpenShift do not use the same operational procedure as a self-managed kubeadm control plane. Editing a static kube-apiserver manifest is therefore not a portable recommendation.
Encryption keys and etcd backups belong to the same recovery design. If keys are lost, encrypted data may become unrecoverable. If backups are accessible without adequate protection, encryption may provide less isolation than expected.
RBAC: protect indirect access as well
The obvious rule is to restrict get, list, and watch on Secrets. But Kubernetes access control is not limited to direct Secret reads. A user who can create a pod in a namespace may be able to mount a Secret into that pod and extract its contents, even without get permission on Secrets.
Consequently, least privilege requires more than a narrowly scoped Secret-reader Role:
- Restrict who can create or modify pods, Deployments, Jobs, and other workload resources in sensitive namespaces.
- Use namespace boundaries that correspond to meaningful trust boundaries; namespaces alone are not strong isolation against cluster administrators.
- Apply admission controls and workload security policies to reduce privileged execution and credential exposure.
- Disable automatic service-account token mounting when workloads do not need Kubernetes API access.
- Audit sensitive API operations without logging Secret values in request or response bodies.
Most applications do not need Kubernetes API permission to read a Secret merely to consume it through secretKeyRef or a Secret volume. The kubelet handles that delivery. Avoid granting application ServiceAccounts unnecessary Secret-reader roles.
Rotation is an application workflow
A remote secret manager can rotate a password; ESO can synchronize the updated value; Kubernetes can update a mounted Secret volume. None of those steps guarantees that the application has started using the new credential.
The propagation path depends on consumption mode:
- Environment variables: values are fixed for the lifetime of the container and require a rollout. When environment variables are unavoidable, teams typically pair ESO with automated rollout controllers (such as Stakater Reloader) or inject config checksum annotations into deployment templates to trigger rolling updates upon Secret mutation.
- Secret volumes: updates are eventually reflected in mounted files, subject to kubelet propagation and mount behavior. Applications must reload or reopen the files. subPath mounts do not receive normal automatic updates.
- Direct provider access: the application must refresh its own credential cache and handle authentication failures appropriately.
For database credentials, rotation must coordinate the database, secret manager, synchronization controller, and application. Where supported, a staged rotation with overlapping valid credentials can reduce downtime. The application must actually support that overlap; merely storing old and new passwords in a Secret does not implement zero-downtime rotation.
A production rotation runbook should define the owner, maximum propagation delay, rollback procedure, failure alerts, and evidence that the old credential was revoked.
GitOps without plaintext credentials
GitOps repositories should contain the declaration of secret dependencies, not raw credentials. For ESO, this typically means committing SecretStore and ExternalSecret manifests while keeping the actual values in the external provider.
When encrypted values must be stored in Git, SOPS or Sealed Secrets can be appropriate, provided the team has a key-management and disaster-recovery plan. Encryption is only one part of the design: reviewers still need to understand who can decrypt, how keys rotate, and what happens during cluster recovery.
Avoid examples that place plaintext passwords in stringData and describe the resulting manifest as GitOps-safe. stringData simplifies API submission; it does not make the value confidential in source control, CI logs, or shell history.
Verify the system, not just the manifest
A production readiness check should exercise the complete delivery and recovery path:
# Confirm the external resource and its synchronization status.
kubectl -n production get externalsecret database-credentials
kubectl -n production describe externalsecret database-credentials
# Confirm the target exists without printing secret values.
kubectl -n production get secret database-credentials
# Inspect controller errors; review output before sharing logs.
kubectl -n external-secrets logs \
-l app.kubernetes.io/name=external-secrets \
--tail=100Then test a controlled remote credential change, measure how long the new value takes to reach the workload, and verify the application can reconnect. Test a failed synchronization and a temporarily unavailable provider. Finally, exercise restoration from backups and confirm the required decryption keys and identity bindings are available.
Monitoring should cover reconciliation failures, stale credentials, unexpected secret access, rotation failures, and certificate or credential expiry. Avoid exporting secret contents as labels, log fields, or diagnostic artifacts.
Conclusion
Production secrets management is a lifecycle problem, not a YAML problem. Encryption at rest protects stored data; workload identity reduces long-lived cloud credentials; external secret managers centralize ownership; and Kubernetes controls how applications receive values. Each mechanism addresses a different failure mode.
A sound platform design connects these layers with least-privilege authorization, explicit rotation behavior, observability, and tested recovery. The goal is not merely to keep passwords out of Git. It is to ensure that credentials remain controlled from creation to revocation—even when workloads, nodes, controllers, or clusters fail.
ape factory helps teams design and operate Kubernetes platforms with security, identity, GitOps, and day-2 operations built into the platform architecture.
References
- Kubernetes: Secrets
- Kubernetes: Encrypting Secret Data at Rest
- Kubernetes: Good Practices for Kubernetes Secrets
- Kubernetes: Distribute Credentials Securely Using Secrets
- External Secrets Operator: API and SecretStore documentation
- AWS EKS: IAM roles for service accounts
- Secrets Store CSI Driver documentation
- SOPS
- Sealed Secrets


