Kubernetes 1.29 (Mandala): Features, Graduations, and Upgrade Impact

Kubernetes 1.29, Mandala (The Universe), was released on December 13, 2023 as the final Kubernetes release of that year. It delivered 49 enhancements: 11 graduating to Stable, 19 entering Beta, and 19 reaching Alpha, with a strong focus on storage safety, API scalability, security, scheduling efficiency, and new networking capabilities.

The official Mandala theme represents the Kubernetes community as an interconnected universe of contributors. Its release logo symbolizes both the Kubernetes project and the people who collectively sustain it, as described in the official Kubernetes 1.29 release announcement .

Key Highlights and Graduations

The sections below distinguish generally available capabilities from features that still require additional production validation. The release blog presents selected Beta and Alpha enhancements rather than the complete set; consult the official Kubernetes 1.29 changelog for the full release record.

Storage

ReadWriteOncePod access mode — Stable

ReadWriteOncePod is now generally available for CSI volumes.

The existing ReadWriteOnce access mode restricts a volume to a single node, but multiple Pods on that node can potentially access it. ReadWriteOncePod instead ensures that only one Pod across the cluster can read from or write to the associated PersistentVolumeClaim.

Production impact:

• Workloads that require a cluster-wide single-writer guarantee can use a stable Kubernetes access mode.
• Stateful applications can avoid the ambiguity of multiple Pods accessing a ReadWriteOnce volume from the same node.
• Storage platforms must support the relevant CSI behavior; verify compatibility with the CSI driver before changing existing claims.

CSI node volume expansion secrets — Stable

CSI node expansion secret support is generally available. Kubernetes can provide an optional secret to a CSI driver as part of NodeExpandVolumeRequest.

This supports storage systems that require credentials during node-side expansion, including:

• Encrypted block storage that needs a passphrase to resize the device.
• Storage drivers that must authenticate to a backend system during expansion.
• Drivers that perform credential-backed validation before resizing a filesystem or volume.

Production impact: CSI drivers can perform authenticated node-side expansion without relying on non-standard credential delivery mechanisms. Operators should confirm that their CSI driver supports this capability before depending on it.

Security and Authentication

KMS v2 encryption at rest — Stable

KMS v2 is now stable and is the recommended mechanism for integrating Kubernetes API data encryption with an external key service. The release highlights improvements in:

• Performance
• Key rotation
• Health checks and status
• Observability

KMS v1 is disabled by default in Kubernetes 1.29 and requires an explicit opt-in to continue using it.

Production impact: Clusters using KMS v1 should treat KMS configuration as an upgrade-blocking review item. Validate provider compatibility, key rotation procedures, failure handling, and recovery before moving production API data encryption to KMS v2.

The supplied release excerpts do not identify the exact KMS v1 feature-gate name or provide a migration command. Verify those details against the Kubernetes documentation and the full release notes rather than assuming configuration equivalence.

Legacy ServiceAccount token cleanup — Beta

The LegacyServiceAccountTokenCleanUp feature reduces exposure from old auto-generated Secret-based ServiceAccount tokens.

In Kubernetes 1.29, legacy auto-generated tokens are:

1. Marked invalid after a long period without use—one year by default.
2. Automatically removed if they remain unused for an additional long period—another year by default.

This follows earlier changes that introduced time-limited, Pod-bound ServiceAccount tokens, stopped automatically generating legacy Secret-based tokens, and began tracking their last-used date.

Production impact: Inventory applications, automation, and external integrations that still rely on legacy ServiceAccount token Secrets. Long-dormant credentials may become invalid and later be removed, potentially exposing previously hidden dependencies during disaster recovery or infrequent maintenance workflows.

API Architecture and Control Plane

Kubernetes 1.29 includes several control-plane and API-related graduations to Stable:

• Priority and Fairness for API server requests
• Paged LIST queries from the Kubernetes API
• CRD Validation Expression Language
• Kubernetes component health SLIs
• Removal of transient node predicates from the KCCM service controller

Production impact:

• API Priority and Fairness provides a stable foundation for controlling API server request concurrency.
• Paged LIST support can improve the handling of large API result sets.
• Stable CRD validation expressions provide a more mature mechanism for enforcing rules on custom resources.
• Component health SLIs improve the foundation for control-plane health monitoring.

The supplied excerpts list these graduations but do not provide their exact configuration fields, metrics, defaults, or migration procedures. Operators should verify implementation details in the official changelog and Kubernetes documentation before changing production configurations.

Node and Workload Operations

The following capabilities also graduated to Stable:

• Kubelet resource metrics endpoint
• Tracking Ready Pods in Job status

Production impact:

• The stable kubelet resource metrics endpoint provides a supported basis for node-level resource telemetry.
• Ready Pod tracking in Job status can improve the operational visibility of Job-managed workloads.

The release excerpts do not specify metric names, endpoint configuration, or Job status field semantics. Those details require verification in the documentation.

Node lifecycle separated from taint management — Beta

Node lifecycle handling and taint-based eviction are separated into two controllers:

• NodeLifecycleController adds taints to unhealthy nodes.
• TaintManager deletes Pods from nodes carrying a NoExecute taint.

Production impact: This separation clarifies controller responsibilities around unhealthy nodes and Pod eviction. Because the feature is Beta, operators should test failure, recovery, and eviction behavior under their own node outage scenarios before relying on it at scale.

QueueingHint scheduler optimization — Beta

The QueueingHint feature aims to reduce unnecessary scheduling retries by improving decisions about when unschedulable Pods should be requeued.

Production impact: Large or scheduling-intensive clusters may benefit from reduced useless retry activity and better scheduler throughput. Validate scheduler behavior and relevant observability signals before broad production adoption.

Networking

Static and dynamic NodePort allocation ranges — Stable

The ability to reserve NodePort ranges for dynamic and static allocation graduated to Stable.

Production impact: This provides a stable mechanism for reducing collisions between explicitly selected NodePorts and dynamically allocated NodePorts. The supplied release excerpts do not define configuration details, so operators should verify the required API and component settings before implementation.

nftables backend for kube-proxy — Alpha

Kubernetes 1.29 introduces an Alpha kube-proxy backend based on nftables. The existing Linux implementation is based on iptables, while nftables is intended to address limitations and performance issues associated with iptables.

Production impact: This is strategically important for Linux distributions moving away from iptables, but it remains Alpha in 1.29. Treat it as an evaluation feature rather than an established production replacement, and verify host operating system and networking compatibility.

APIs for managing Service IP ranges — Alpha

Kubernetes 1.29 introduces two API objects:

• ServiceCIDR
• IPAddress

These APIs support a new allocation model intended to let operators add, remove, or resize Service IP ranges without restarting the API server. One intended use is expanding available Service IP capacity by creating additional ServiceCIDR objects.

Production impact: The feature could help address Service IP exhaustion and renumbering, but it is Alpha. Do not assume API stability, upgrade compatibility, or production support without further validation.

Windows and Runtime Integration

Image pulls based on RuntimeClass — Alpha

The RuntimeClassInImageCriApi feature adds support for pulling container images according to the Pod’s RuntimeClass. It is disabled by default in Kubernetes 1.29.

The feature changes image references from being based only on an image name or image ID to a tuple of:

• imageID
• runtimeClass

This is particularly relevant when the runtime must choose a different image manifest than the host-default platform selection, such as for Windows Hyper-V containers.

Container runtimes may choose whether to support this capability. If the runtime does not support it, kubelet retains its pre-1.29 behavior.

Production impact: Enabling the feature gate alone is not sufficient; runtime support must also be confirmed. Test image selection and fallback behavior before using it for heterogeneous or VM-based container workloads.

In-place Pod resource updates for Windows — Alpha

The Alpha capability for modifying Pod resource requests and limits without restarting the Pod now supports Windows containers.

Production impact: This can reduce disruption when adjusting resources for Windows workloads, but Alpha status means it should first be evaluated in non-production environments.

Scheduling

Pod affinity and anti-affinity with matchLabelKeys — Alpha

Kubernetes 1.29 introduces Alpha support for defining Pod affinity or anti-affinity using matchLabelKeys. The stated goal is to improve scheduling calculation accuracy during rolling updates.

Production impact: This may help operators express placement relationships that follow changing workload labels during rollouts. Because the feature is Alpha, API behavior and operational edge cases require careful testing.

Complete Stable Graduation Summary

The official release announcement lists 11 enhancements promoted to Stable:

1. Removal of transient node predicates from the KCCM service controller
2. Reserved NodePort ranges for dynamic and static allocation
3. Priority and Fairness for API server requests
4. KMS v2 improvements
5. Paged LIST queries from the Kubernetes API
6. ReadWriteOncePod PersistentVolume access mode
7. Kubernetes component health SLIs
8. CRD Validation Expression Language
9. nodeExpandSecret in the CSI PersistentVolume source
10. Ready Pod tracking in Job status
11. Kubelet resource metrics endpoint

This is the authoritative Stable list supplied by the Kubernetes 1.29 release page . Not every item is described in enough detail in the supplied excerpts to prescribe configuration changes.

Deprecations, Removals, and Breaking Changes

Removal of flowcontrol.apiserver.k8s.io/v1beta2

Kubernetes 1.29 no longer serves the deprecated flowcontrol.apiserver.k8s.io/v1beta2 API version for:

• FlowSchema
• PriorityLevelConfiguration

This is a direct upgrade compatibility issue. Manifests and client software that still request this API version must be changed before upgrading the API server to Kubernetes 1.29.

The supplied excerpts do not state the exact replacement API version. Determine the supported replacement from the deprecated API migration guidance referenced by the official release page before modifying resources or generated clients.

In-tree cloud provider integrations disabled by default

Kubernetes 1.29 defaults to running without built-in cloud provider integrations. This affects clusters relying on the legacy in-tree integrations for:

• Azure
• GCE
• vSphere

The recommended migration is to deploy the equivalent external cloud controller manager. Doing so requires:

• Running a suitable cloud controller manager in the control plane.
• Setting --cloud-provider=external for kubelet on every relevant node.
• Setting --cloud-provider=external for kube-apiserver.
• Setting --cloud-provider=external for kube-controller-manager.

A temporary legacy opt-in remains available by setting these feature gates to false:

• DisableCloudProviders
• DisableKubeletCloudCredentialProviders

Operator action: Do not begin a 1.29 upgrade until the cluster’s cloud integration model is known. Validate external controller deployment, component flags, cloud credentials, and node initialization behavior. Treat the legacy feature-gate override as an explicit compatibility choice, not the preferred end state.

Deprecation of Node.status.nodeInfo.kubeProxyVersion

The .status.nodeInfo.kubeProxyVersion field on Node objects is deprecated and is proposed for removal in a future release.

The field has historically been managed by kubelet, but kubelet does not reliably know:

• Which kube-proxy version is running.
• Whether kube-proxy is running at all.

Operator action: Remove dependencies on this field from inventory systems, policy engines, dashboards, health checks, and custom controllers. It should no longer be treated as an authoritative source of kube-proxy version information.

The supplied sources do not define a replacement field or API.

Legacy package repositories

The legacy repositories were deprecated in August 2023:

• apt.kubernetes.io
• yum.kubernetes.io

They were frozen in September 2023 and were scheduled to disappear in January 2024. The replacement community-owned package repository is:

• https://pkgs.k8s.io

This repository transition was not introduced specifically by Kubernetes 1.29, but it directly affects installation and upgrade automation.

Operator action: Review package manager configuration, bootstrap scripts, machine images, configuration management, and CI/CD pipelines for references to the legacy repositories. Migrate those references before depending on package-based Kubernetes 1.29 installation.

Upgrade Impact and Operational Notes

Before upgrading

Use the following review as an upgrade gate:

• Identify all manifests and clients that use flowcontrol.apiserver.k8s.io/v1beta2.
• Determine whether the cluster uses an in-tree Azure, GCE, or vSphere cloud provider.
• Review control-plane and kubelet --cloud-provider settings.
• Inventory use of Node.status.nodeInfo.kubeProxyVersion.
• Check package configuration for apt.kubernetes.io and yum.kubernetes.io.
• Review KMS v1 usage and validate a KMS v2 migration plan.
• Identify workloads or automation using legacy Secret-based ServiceAccount tokens.
• Verify CSI driver compatibility with storage features you intend to adopt.
• Validate the container runtime before evaluating RuntimeClass-aware image pulls.

No upgrade commands are included here because the supplied source excerpts do not provide a verified control-plane or worker upgrade command sequence.

Control-plane considerations

The control plane requires particular attention to three changes:

1. Removed flow-control APIs: Update affected manifests and clients before the new API server is introduced.
2. Cloud provider behavior: External cloud controller managers require --cloud-provider=external across the API server, controller manager, and relevant kubelets.
3. Encryption configuration: KMS v2 is stable and recommended, while KMS v1 is disabled by default.

The supplied sources do not define a specific control-plane upgrade order, rollback procedure, or supported version-skew sequence. Follow the applicable installation tooling and Kubernetes documentation rather than deriving a procedure from this feature summary.

Worker-node considerations

Before upgrading Linux or Windows workers:

• Verify kubelet cloud-provider configuration.
• Confirm CSI driver support for any storage capabilities being adopted.
• Test Alpha runtime features only with a compatible container runtime.
• Remove operational dependencies on Node.status.nodeInfo.kubeProxyVersion.
• Validate scheduling, taint eviction, and Service networking after node rollout.

For Windows nodes, the 1.29 patch history is especially important. The official changelog reports that kubelet versions through 1.29.12 were affected by CVE-2024-9042, a command-injection vulnerability involving the nodes/*/logs/query API, and identifies 1.29.13 as the fixed 1.29 version. The supplied changelog extends through 1.29.15, so the original v1.29.0 tag should be treated as a historical release reference rather than an automatically appropriate deployment target.

Review the complete 1.29 changelog for subsequent security and regression fixes before selecting a patch release.

Dependency verification

The supplied excerpts do not establish new minimum operating system, container runtime, CSI, CNI, or external dependency versions for the original Kubernetes 1.29 release. Later patch entries mention changes to the Go version used to build Kubernetes, but those build-tool updates should not be interpreted as runtime dependency requirements.

Any platform-specific minimum versions therefore require manual verification against the relevant component documentation and the full release notes.

Conclusion

Kubernetes 1.29 “Mandala” combines mature storage and control-plane capabilities with forward-looking work in nftables networking, dynamic Service address management, RuntimeClass-aware image pulls, and Windows resource management. The most important upgrade work is not feature enablement: it is removing flowcontrol.apiserver.k8s.io/v1beta2 dependencies, addressing the shift away from in-tree cloud providers, reviewing KMS configuration, migrating legacy package repositories, and selecting an appropriately patched 1.29 release.

Official Resources

• Kubernetes 1.29: Mandala release announcement
• Official Kubernetes 1.29 changelog and release notes
• Kubernetes v1.29.0 GitHub release tag
• Kubernetes documentation