Kubernetes 1.28 - (Planternetes): What changes for production

Kubernetes 1.28, Planternetes, is the second Kubernetes release of 2023 and includes 45 enhancements: 12 graduating to Stable, 14 to Beta, and 19 entering Alpha. Its official garden-inspired theme celebrates the care, collaboration, and continuous growth behind each Kubernetes release, while the technical focus spans more flexible version skew, stronger failure recovery, admission control, storage automation, and new workload APIs (release announcement ).

Key Highlights and Graduations

The following sections cover selected highlights from the release announcement. They are not an exhaustive list of all 45 enhancements; consult the complete Kubernetes 1.28 changelog for the full release record.

Architecture and API Machinery

Stable: expanded control plane-to-node version skew

Kubernetes 1.28 expands the supported skew between core node components and control plane components from n-2 to n-3. Specifically, kubelet and kube-proxy from the oldest supported minor version can work with kube-apiserver, kube-scheduler, kube-controller-manager, and cloud-controller-manager from the newest supported minor version.

This gives operators more flexibility when scheduling node maintenance. Because a kubelet minor-version upgrade includes draining the node—and therefore disrupting Pods that were running there—the expanded skew can help organizations with long-running workloads reduce the frequency of worker-node maintenance. The release announcement describes an annual upgrade model in which operators can perform three sequential control plane minor-version upgrades while upgrading nodes only once per calendar year and remaining within upstream-supported skew.

The expanded skew is an option, not a requirement. Clusters can still upgrade nodes more frequently.

Beta: improved CRD validation rules

Common Expression Language, or CEL, validation lets CustomResourceDefinition authors implement non-trivial validation directly in a CRD schema rather than relying on a separate admission webhook.

Kubernetes 1.28 adds two optional fields for validation failures:

• reason
• fieldPath

These fields let API authors provide a more specific failure reason and identify the field associated with a failed validation rule. Operationally, this can improve API usability while reducing the infrastructure and lifecycle burden associated with validation webhooks.

Alpha: mixed-version API server proxy

During a control plane upgrade, downgrade, or runtime-config rollout, different API server instances may support different combinations of built-in API groups, versions, and resources. The new mixed-version proxy can detect when the local API server does not recognize a request and transparently forward that request to a compatible peer API server.

This feature operates within the API server aggregation layer and is intended to hide temporary control plane version skew from clients. Because the capability is Alpha in 1.28, operators should treat it as an evaluation feature rather than assume it is appropriate for general production enablement.

Control plane source-code reorganization

Kubernetes contributors began reorganizing kube-apiserver code around a new staging repository that consumes k/apiserver while exposing a larger, carefully selected subset of reusable API server functionality. This is a gradual internal reorganization rather than a user-facing API graduation, but it establishes a path toward a separate repository for generic API server functionality.

Security and Admission Control

Beta: ValidatingAdmissionPolicy

ValidatingAdmissionPolicy provides in-process, CEL-based validation of Kubernetes API requests as an alternative to validating admission webhooks. The goal is to lower the infrastructure barrier for custom admission policies while providing primitives for consistent Kubernetes and extension-policy practices.

In Kubernetes 1.28, using this Beta feature requires both:

• The admissionregistration.k8s.io/v1beta1 API group
• The ValidatingAdmissionPolicy feature gate in the control plane

Operators evaluating this feature should verify that both are enabled consistently across control plane instances.

Beta: admission webhook match conditions

The matchConditions field for ValidatingWebhookConfiguration and MutatingWebhookConfiguration graduates to Beta and is enabled by default in Kubernetes 1.28.

Each match condition is a CEL expression that must evaluate to true before Kubernetes sends the admission request to the remote webhook. This allows platform teams to narrow webhook scope, avoiding remote calls for requests that do not require a particular policy or mutation.

Stable: self-user attributes API

The authentication API for retrieving self-user attributes graduates to Stable. The supplied release excerpt lists the graduation but does not provide its exact resource schema or operator migration requirements; verify those details against the official documentation before adopting it.

Storage and Stateful Workloads

Stable: recovery from non-graceful node shutdown

Recovery from non-graceful node shutdown is now generally available.

When a node fails without kubelet-mediated shutdown handling—because of hardware failure, an unresponsive operating system, Windows behavior, a non-systemd Linux init system, or incorrect graceful-shutdown configuration—StatefulSet Pods can remain indefinitely in Terminating. Their VolumeAttachment objects can also remain associated with the failed node, preventing PersistentVolumes from attaching to a healthy replacement node.

The stable recovery mechanism enables cleanup so stateful workloads can fail over to another node. This is particularly important for immutable infrastructure and hardware-failure scenarios in which the original node may never return.

Operators should review their documented non-graceful shutdown recovery procedure before an incident occurs. The release excerpt does not include the exact cleanup commands, so they should be verified against the current Kubernetes documentation .

Stable: retroactive default StorageClass assignment

Kubernetes automatically assigns a storageClassName to a PersistentVolumeClaim when one is not provided. The control plane can also assign the default StorageClass to existing PVCs that do not have storageClassName defined.

Although earlier Kubernetes versions had this behavior, it becomes automatic, always active, and Stable in Kubernetes 1.28. Storage administrators should review the cluster’s default StorageClass and identify PVCs without an explicitly defined storage class before upgrading, especially where retroactive assignment could affect provisioning expectations.

Removal: CSI migration for GCE PD

Kubernetes 1.28 removes CSI migration for GCE PD. The supplied release excerpt does not identify exact API objects, feature-gate flags, or a migration sequence, so those details require manual verification in the v1.28.0 changelog and current storage documentation .

Before upgrading, operators should inventory clusters that use GCE PD storage and confirm that their storage configuration no longer depends on the removed migration behavior.

Node and Runtime

Beta: controlled Linux swap support

Support for swap space on Linux nodes advances to Beta. The feature is intended to provide controlled and predictable swap behavior for two overlapping audiences:

• Node administrators performing node-level performance tuning or addressing stability and noisy-neighbor concerns
• Application developers whose workloads can benefit from swap memory

The supplied release material does not specify the exact kubelet configuration, feature-gate state, or supported runtime prerequisites. Those settings must be confirmed manually before enabling swap in a production environment.

Alpha: API-aware sidecar containers

Kubernetes 1.28 introduces an Alpha restartPolicy field for init containers, allowing the API to distinguish a sidecar container from a traditional init container.

An init container with restartPolicy: Always is treated as a sidecar:

• It starts in the declared init-container order.
• The kubelet waits for it to start, rather than waiting for it to complete, before starting regular containers.
• Startup is considered complete after the postStart handler finishes and, when configured, the startup probe succeeds.
• Its state is represented through the Started field in ContainerStatus.
• It restarts independently even when the Pod-level restartPolicy is Never or OnFailure.
• It does not block Pod completion after all regular containers have completed.
• It can restart during Pod termination.

Omitting the init-container restartPolicy preserves traditional run-to-completion init-container behavior. As an Alpha API capability, sidecar-aware behavior should be tested against application startup, shutdown, probe, and termination semantics before use.

Alpha: CDI device injection

Container Device Interface support allows complex devices—those requiring more than a single /dev node—to be injected into containers using a standardized mechanism.

Kubernetes 1.28 enables device-plugin developers to use the CDIDevices field added to the Container Runtime Interface in 1.27 and pass CDI devices directly to CDI-enabled runtimes. The release announcement identifies recent containerd and CRI-O releases as examples of CDI-enabled runtimes, but does not specify minimum runtime versions. Runtime compatibility therefore requires manual verification.

Stable: node and device resource reporting

The following node-related enhancements graduate to Stable:

• Support for third-party device monitoring plugins
• The kubelet Pod resources endpoint
• Reporting allocatable resources through the podresources API

The supplied release excerpt lists these graduations without configuration details. Operators and device-plugin vendors should consult the complete changelog and documentation for exact endpoint and integration behavior.

Workloads and Jobs

Alpha: delayed replacement Pods for Jobs

Kubernetes 1.28 adds a Job API field that lets users choose when replacement Pods are created:

• As soon as the previous Pod begins terminating, preserving existing behavior
• Only after the previous Pod has fully terminated, using the new optional behavior

Delayed replacement is relevant to indexed machine-learning and batch workloads that require one unique Pod per index. It can also prevent temporary resource overlap, unwanted scheduling contention, or undesired Cluster Autoscaler scale-ups while an old Pod is still terminating.

The exact field name and manifest schema are not present in the supplied excerpt and must be verified in the official documentation before use.

Alpha: per-index Job retry backoff

Indexed Jobs can now apply a backoff limit per index and continue processing other indexes even when one index fails.

Previously, all indexes shared a single backoff limit. Reaching that limit caused the controller to mark the entire Job as failed and clean up its resources, including indexes that had not completed. The new model is better suited to independently executable workloads such as large integration-test suites or other embarrassingly parallel jobs.

Networking

Kubernetes 1.28 includes several Stable networking graduations:

• Proxy terminating endpoints
• Expanded DNS configuration
• Cleanup of iptables chain ownership
• Reduced iptables-restore input size
• Moving the EndpointSlice reconciler into staging

These graduations improve the maturity of service routing, DNS configuration, and iptables management. The supplied release excerpt does not provide exact configuration changes, flags, or migration requirements for these items; operators using custom networking components should review the full changelog for implementation-specific impact.

Command-Line Experience

kubectl events graduates to Stable in Kubernetes 1.28. The release excerpt does not document its command syntax or output guarantees, so consult the current kubectl documentation before incorporating it into operational automation.

Deprecations, Removals, and Breaking Changes

Removed

• CSI migration for GCE PD

Clusters that depend on this migration behavior require review before upgrade. The supplied source does not provide the exact removed flags, APIs, or migration path; these must be checked manually in the official changelog .

Deprecated

• Ceph RBD in-tree volume plugin
• CephFS in-tree volume plugin

The release announcement identifies both in-tree plugins as deprecated but does not state their removal release, exact replacement procedure, feature flags, or manifest migration steps. Operators should inventory PersistentVolumes, StorageClasses, and workload specifications that reference these plugins and verify the supported migration path in the current Kubernetes documentation .

Corrected release-announcement item

The release team explicitly corrected the original announcement to state that CRI container and Pod statistics without cAdvisor did not make the Kubernetes 1.28 release. Upgrade plans must not assume this feature is available in 1.28.

Items requiring manual verification

The supplied source excerpts do not provide complete v1.28.0 details for:

• Removed or deprecated command-line flags
• Exact API field names for the new Job replacement policy
• Feature-gate names or defaults for Linux swap, mixed-version proxy, CDI injection, or sidecar containers
• GCE PD and Ceph storage migration procedures
• Minimum container runtime, operating system, or external component versions
• Dependency version changes associated specifically with the initial v1.28.0 release

Do not infer these values from feature status alone. Verify them against the v1.28.0 section of the official changelog and the documentation for your installation method and infrastructure provider.

Upgrade Impact and Operational Notes

Control plane planning

Before changing control plane versions:

1. Review the v1.28.0 Urgent Upgrade Notes, deprecations, API changes, and dependency sections in the official changelog .
2. Inventory usage of GCE PD CSI migration and the deprecated Ceph RBD and CephFS in-tree plugins.
3. Review admission configuration, especially if enabling ValidatingAdmissionPolicy or depending on webhook matchConditions.
4. Verify that all control plane instances use consistent API and feature-gate settings during a rollout.
5. Test CRDs that use CEL validation, including the new reason and fieldPath behavior.
6. Treat the Alpha mixed-version proxy as optional; the broader n-3 node skew does not require enabling it.

The mixed-version proxy may reduce client-visible API differences during control plane version skew, but it remains Alpha and should not replace normal upgrade validation.

Worker-node planning

A kubelet minor-version upgrade uses a node-drain process and can disrupt Pods running on that node. Kubernetes 1.28’s n-3 skew provides more scheduling flexibility for worker upgrades, but operators must still ensure that every kubelet and kube-proxy version remains within the supported range relative to the newest control plane version.

Before worker upgrades:

• Validate the version of every kubelet and kube-proxy.
• Review disruption implications for long-running and stateful workloads.
• Test non-graceful node shutdown recovery for workloads using PersistentVolumes.
• Verify Linux swap settings before relying on Beta swap support.
• Confirm CDI support in the deployed container runtime before testing CDI device injection.
• Validate Alpha sidecar startup, restart, and termination behavior in a non-production environment.

Storage checks

Storage deserves explicit pre-upgrade attention because Kubernetes 1.28 combines a removal, two deprecations, and changed defaulting maturity:

• Confirm that GCE PD environments no longer rely on removed CSI migration behavior.
• Locate Ceph RBD and CephFS in-tree plugin usage.
• Confirm which StorageClass is marked as the cluster default.
• Review PVCs that omit storageClassName, because retroactive default StorageClass assignment is now stable and always active.
• Document the response process for a failed node that leaves StatefulSet Pods and volume attachments associated with an unavailable machine.

Patch-release and security considerations

The supplied changelog extends beyond v1.28.0 through v1.28.15, so operators should review the entire 1.28 patch history rather than evaluating only the initial tag. In particular, the changelog records CVE-2024-5321, affecting Windows kubelets through 1.28.11, with the fix included in 1.28.12.

Choose the intended 1.28 patch release only after reviewing its security notices, bug fixes, regressions, and dependency sections in the official changelog.

Conclusion

Kubernetes 1.28 Planternetes balances operational maturity with forward-looking API work. Stable non-graceful node shutdown recovery, retroactive StorageClass assignment, and expanded n-3 node skew offer immediate operational value, while Beta admission controls and Alpha sidecar, device, Job, and mixed-version capabilities provide a path for controlled evaluation.

The most important upgrade actions are to review the GCE PD CSI migration removal, inventory deprecated Ceph in-tree storage plugins, validate worker-version skew, and test stateful failure recovery. Because several exact flags, schemas, dependency requirements, and migration procedures are not included in the supplied release excerpt, verify those details directly from the official resources before changing production clusters.

Official Resources

• Kubernetes 1.28: Planternetes release announcement
• Official Kubernetes 1.28 changelog and release notes
• Kubernetes v1.28.0 GitHub release tag
• Kubernetes documentation