Running LLMs on Kubernetes: A Production Architecture Guide

A model can answer a prompt while the service around it remains unable to survive a node failure. The missing piece is usually not another Kubernetes controller. It is a capacity and admission policy that still works while GPUs restart, weights load and clients hold streaming connections open.

Production LLM inference: Three separate flows

Production LLM inference needs a coordinated design for requests, GPU memory and recovery. Kubernetes provides workload orchestration. The inference runtime schedules token computation. A gateway enforces who may consume capacity. None can compensate for the others being designed in isolation.

This architecture guide uses vLLM as the runtime example. It focuses on decisions and failure consequences rather than a deployment recipe. Start with the service contract: acceptable time to first token, generation speed, maximum context and output length, tenant boundaries and recovery objectives. Those constraints determine the platform.

Running a model is not operating an inference platform

A demonstration proves that a particular model fits and produces output. A service must also bound the work it accepts, handle cancellations and expose overload without letting every request become a timeout.

LLM requests have uneven costs. A short question with a long output stresses decoding differently from a long document with a short answer. Request counts alone hide that distinction. Larger contexts also consume more key-value (KV) cache, which stores attention state during inference. Increasing concurrency can exhaust memory before compute utilization reaches the level an operator expected.

Define separate latency objectives for interactive chat and asynchronous work. If both share one pool, long jobs can consume the capacity needed by interactive sessions. Separate pools buy predictable service at the cost of lower aggregate utilization; shared pools require explicit admission budgets and fair scheduling. Kubernetes does not make that product decision.

Availability must include the time needed to restore useful capacity. A replacement pod that exists but is still downloading weights is not a recovered inference replica.

Reference architecture for production LLM inference

Keep three flows distinct: user requests, model distribution and telemetry. They have different access controls, bandwidth patterns and failure modes.

The gateway is a role, not necessarily one product. Authentication, policy enforcement and load balancing may span several components. A Kubernetes Service provides endpoint connectivity; it does not understand model authorization, token budgets or the runtime’s queue. Add inference-aware routing only when its benefit justifies the additional state and failure handling.

Authenticate at the edge and preserve a trusted tenant identity downstream. Encrypt client traffic and, where required by the threat model, traffic between the gateway and serving tier. Validate the server identity on those connections. Do not expose a second public route directly to the runtime that bypasses policy.

Artifact access should remain separate from request authorization. A serving pod needs a specific model revision, not broad credentials to a model registry. Observability collectors need restricted access to metrics, not unrestricted access to prompts. Keep the model-serving namespace private and explicitly allow the required flows.

The diagram uses a single serving pool for readability. Replicas may occupy different nodes or zones; a multi-GPU replica can span a larger failure domain. For the broader architectural context, see the AI knowledge hub and AI gateway knowledge .

Routing strategy and prefix caching

Round-robin and least-connections routing do not account for prefix-cache locality, but they do not disable cache reuse within a replica. vLLM’s Automatic Prefix Caching uses hash-based KV-cache blocks; RadixAttention is associated with SGLang. When compatible cached blocks for an identical token prefix remain resident, reuse can reduce prefill work. It does not eliminate decoding work. [9][10]

With replica-local caches, distributing repeated prefixes across replicas can duplicate prefill work and reduce cache locality. The effect on TTFT depends on prefix repetition, cache capacity and eviction, replica count, and queueing. Common prefixes may already be cached on every replica. Affinity can also overload a popular replica, so compare cache benefits against load balance and tail latency before adding routing state:

• Stateful hash routing: Hash a stable prefix or conversation session ID to bias requests toward replicas. Affinity is a heuristic, not proof that matching token blocks remain cached; account for replica changes and permit overload fallback.
• Cache-aware gateway state: A router may use runtime-supported cache events or an approximate index of prefix assignments. Aggregate KV-cache occupancy alone does not identify which prefixes are resident. Account for stale state and eviction, and balance expected reuse against queue depth.
• Tenant boundary risks: Shared prefix caches can expose timing side-channels. Where supported, enforce trusted tenant-specific cache isolation, such as vLLM cache salts, or disable cross-tenant reuse or dedicate replica pools. Cache isolation alone does not provide complete tenant isolation. [9]

GPU capacity, model placement and scheduling

Begin with a memory budget: weights, KV cache, activations and runtime overhead, plus operational headroom. Parameter count multiplied by bytes per parameter estimates weight storage only. For illustration, eight billion parameters at two bytes each occupy roughly 16 GB in decimal units before other allocations. This is arithmetic, not a hardware sizing result.

Quantization can reduce weight memory, but support depends on the model, quantization format, runtime kernels and GPU architecture. It can also change output quality and performance. Validate the actual combination. Context length, concurrent sequences and KV-cache precision affect the remaining memory budget; a model fitting at idle says little about safe peak admission.

With the conventional NVIDIA device-plugin path, a device plugin advertises extended resources such as nvidia.com/gpu. Kubernetes schedules advertised resource quantities, not an estimate of model memory or token throughput. GPU limits can be specified without requests; Kubernetes uses the limit as the request. If both are specified, they must match. Device exposure also requires compatible host drivers and container runtime integration. [1]

Use node labels and required affinity for supported GPU types; use taints and tolerations to reserve a GPU pool. A toleration permits placement on a tainted node—it does not select that node. Labels must come from a controlled inventory process. Do not assume every GPU advertising the same resource name has interchangeable memory or kernel support.

Choose parallelism separately from replica count. Adding independent replicas increases aggregate serving capacity when each replica has the complete required model state. Tensor parallelism splits computation across devices; pipeline parallelism assigns model layers to stages. Either can make a larger model feasible, while adding communication and coordination costs. Topology, interconnect bandwidth and runtime support matter, particularly across nodes. A Kubernetes device count alone is not proof of a suitable topology. [3]

Exclusive devices provide a simpler interference model. NVIDIA MIG partitions supported hardware into GPU instances with hardware resource isolation; time-slicing instead shares access and does not provide the same memory or fault isolation. Neither replaces tenant authorization or host security. Select the sharing mode according to the threat model and measured interference, not only apparent GPU utilization. [2]

Inference runtimes and model distribution

vLLM manages execution concerns that Kubernetes cannot see: batching, KV-cache allocation and token scheduling. Its supported models, dtypes, quantization methods and parallel execution paths depend on the release and hardware backend. Continuous batching lets requests enter and leave execution as others progress; it does not remove the need for admission limits. [3]

Treat the release unit as more than a container tag. Record the runtime image digest, model revision and artifact checksums, tokenizer and chat template, quantization settings, context limits and tested driver/CUDA compatibility. A tokenizer or template change can alter behavior even when the weight files are unchanged. Use version-specific runtime documentation when qualifying a release; this guide does not prescribe a tested version matrix.

Object storage is an artifact source, not automatically a filesystem the runtime can load from. A staging process may need to fetch and validate files before startup. Shared filesystems avoid repeated distribution but do not remove weight loading into host and GPU memory. Measure the full path to readiness, including runtime initialization and warm-up.

Avoid mutable model references in a release. Keep the previous artifact set accessible for rollback, and verify integrity before loading. Stagger startup so a rollout does not turn every worker into a simultaneous large downloader. Restrict network access after staging where practical, and avoid enabling remote model code unless it has undergone explicit review.

Packaging weights inside images offers another distribution path, but makes images large and couples model promotion to image distribution. Select it deliberately; it does not make cold-node transfer costs disappear.

Traffic management, performance and autoscaling

Admission control should bound prompt length, requested output length, concurrent work and queue residence time. Per-tenant request limits protect fairness only partially: token budgets better reflect expensive requests, though final output usage is known only as generation proceeds. Reserve a maximum budget on admission and reconcile actual usage afterward.

Reject work that cannot meet its deadline instead of building an unbounded queue. A tenant quota failure and global overload are different conditions; document the response and retry policy for each. Choose one authoritative place to bound waiting and account for any additional runtime queue, so gateway and runtime do not independently accumulate invisible backlogs.

Time to first token (TTFT) measures when useful output begins. Client-observed TTFT includes network transit, gateway processing, queues and runtime work. A runtime’s own TTFT histogram can have a narrower boundary. Inter-token latency describes spacing between generated tokens; transport buffering may make client-visible chunks differ from individual tokens. End-to-end latency ends only when the response completes. [4]

Prefill processes the prompt and establishes attention state; decoding generates subsequent tokens. Long prompts and long outputs therefore stress different parts of the service. Runtime scheduling can interleave work, so the lifecycle diagram is conceptual rather than a strict GPU execution timeline.

Use the exact metric names and histogram semantics of the selected vLLM release. Connect a chosen scaling signal through a suitable metrics adapter if using Kubernetes HPA custom or external metrics. CPU-only scaling misses GPU memory pressure and token queues; CPU remains useful for tokenizer and gateway bottlenecks. [4][5]

Replica scaling and GPU-node provisioning are separate control loops. Include both delays, plus model loading, in capacity planning. Keep enough warm capacity for the promised burst envelope, use bounded queues for brief fluctuations and reject excess work. Scale-to-zero is reasonable for delay-tolerant workloads, but incompatible with a first-request latency promise shorter than measured cold start.

Streaming resilience and cancellation propagation

Streaming responses, including Server-Sent Events (SSE) carried over HTTP/1.1 or HTTP/2, introduce long-lived requests that fail differently from unary request-response loops. A primary failure mode is phantom GPU consumption: an end-user closes their browser tab, terminates an API client, or triggers an application cancellation while the model is halfway through a 2,048-token generation.

If an ingress controller, reverse proxy, or API gateway fails to propagate cancellation, the upstream inference request can remain active. The runtime may continue decoding and retain KV-cache blocks for output that nobody will consume. An open transport connection alone does not tell the runtime whether an individual request has been cancelled.

To prevent orphan generation load:

1. Request-level cancellation detection: Monitor downstream disconnects and stream cancellation. HTTP/2 can cancel one request with RST_STREAM while the underlying connection continues carrying other requests; socket monitoring alone is insufficient. [11]
2. Explicit upstream abortion: Propagate cancellation to the corresponding upstream request. On HTTP/2, reset that stream without closing the shared connection and disrupting unrelated requests. An HTTP/1.1 hop may require closing its upstream connection. Ensure the runtime’s supported disconnect or abort handling actually stops generation; this depends on the proxy, runtime version and API path. Test cancellation through every hop and verify resource release.
3. No automatic retries mid-stream: Never configure automated proxy retries for a request once the first token byte has been streamed to the client. Retrying after partial streaming causes duplicate execution, double-counts tenant token quotas, and delivers divergent or corrupt outputs.
4. Buffer flushing and chunk pacing: Configure each proxy to forward streaming output promptly, using its documented buffering controls. Buffering can inflate client-observed TTFT and delay delivery; disabling it does not itself guarantee cancellation propagation. Test both streaming latency and cancellation separately.

Security and multi-tenant isolation

A gateway should authorize the tenant for a specific model and budget, not merely accept any valid identity. Keep administrative, health and metrics endpoints private. Within Kubernetes, use least-privilege service accounts and RBAC; a serving process that does not call the Kubernetes API should not receive an unnecessary API token.

Start from default-deny ingress and egress policies, then allow gateway traffic, required DNS, approved artifact sources and telemetry. NetworkPolicy enforcement requires a supporting network implementation. Policies are additive, and ordinary NetworkPolicy is primarily a layer-3/4 control; it does not authorize an inference request or provide TLS. Egress through dynamic external addresses may require additional controls. Test reachability from both allowed and disallowed sources. [6]

Namespace separation and tenant headers are logical boundaries. They do not eliminate interference or vulnerabilities in a shared runtime, GPU or host. Stronger requirements may call for dedicated serving processes, exclusive GPUs, separate nodes or separate clusters. MIG can address particular device-sharing risks, but cannot supply the entire tenant isolation boundary.

Run serving containers without unnecessary privilege, drop unused capabilities, scan and pin images, and apply compatible filesystem restrictions. Host-level GPU management components may need privileges that ordinary serving pods do not. Review these as separate trust domains rather than copying operator permissions into application manifests.

Prompts, responses, token traces and cached prefixes can contain confidential data. Disable raw-content logging by default; restrict any diagnostic capture by access, retention and purpose. Keep tenant identifiers out of unbounded metric labels. Review shared-cache behavior against tenant isolation requirements. Model provenance, license approval and remote-code review belong in promotion, before artifacts reach serving credentials.

High availability, failure recovery and day-2 operations

A controller can replace a failed pod; it cannot conjure a compatible GPU or recover an inaccessible artifact store. Design for useful capacity remaining while a failure is repaired.

Spread independent replicas across failure domains when the GPU supply allows it. Maintain headroom so losing a node does not immediately overload survivors. Pod disruption budgets constrain supported voluntary evictions; they do not prevent node failures. Multi-GPU replicas can fail as a unit, making independent spare capacity more important. [7]

Decoupling readiness, startup, and liveness probes

A common root cause of cluster-wide inference outages is a coupled probe configuration. When an inference pod encounters heavy load, high KV-cache allocation, or momentary scheduling delays, poorly tuned probes misinterpret operational backpressure as process deadlocks. If a liveness check times out during transient queue saturation, the kubelet kills the container. The remaining replicas must immediately absorb the terminated pod’s traffic, pushing their own response latencies over probe thresholds and triggering a self-reinforcing cascading restart storm.

Probe type determines the response to failure; the endpoint determines what is measured. Startup success enables readiness and liveness checks. Readiness failure marks the pod unready for normal Service traffic without restarting it. Liveness failure at its threshold triggers container termination and restart according to policy, respecting the applicable termination grace period; it does not mean an immediate SIGKILL. [8][12]

Using /health for all three probes changes their timing and consequences, not their health semantics. vLLM’s documented /health handler checks engine health; it is not a queue-depth or KV-cache admission check, nor a guarantee of detecting every driver deadlock. Verify its behavior for the selected runtime release. Keep overload control in admission and routing; any custom readiness policy needs an explicit signal and safeguards against removing all busy replicas. [13]

This illustrative container fragment is not a tested deployment configuration. It shows timing only; tune thresholds against measured startup and peak-load behavior. The startup budget covers initialization in this container, not downloads performed by a separate init container.

# Startup: gates readiness and liveness until /health succeeds
startupProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 60  # Nominal 600-second failure budget after the initial delay

# Readiness: engine health, not queue saturation or available KV-cache capacity
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3   # Marks unready after repeated failures; no restart

# Liveness: same health signal with a more tolerant failure threshold
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 15
  timeoutSeconds: 5
  failureThreshold: 5   # Triggers termination/restart, honoring termination grace

These shared checks remain vulnerable to a health endpoint that stalls under load. Use liveness only when its failure reliably indicates that restarting will help, and qualify that assumption with overload and failure tests.

Define separate recovery objectives for a pod restart, node loss and loss of the artifact distribution path. Existing loaded replicas may continue through an artifact outage, but replacement capacity cannot be assumed. Retain local artifacts only as a documented mitigation, not as evidence that the storage dependency has disappeared.

Roll out a small candidate pool first. Validate representative prompt lengths, concurrency, output quality, cancellations and overload, then shift traffic gradually. Budget GPU capacity for the candidate and the old pool simultaneously; otherwise rollout surge settings may create pods that cannot schedule. Drain connections with a bounded termination grace period. A long stream can still be interrupted, so clients need an explicit partial-response policy. [8]

Rollback restores the previous runtime, model artifacts, templates and serving settings together. Retain those artifacts and enough capacity to load them. Exercise rollback and node-loss recovery under representative load before promising recovery times. Alert on unavailable ready capacity, sustained queue delay, artifact failures and device errors, with ownership for each failure path.

A production-readiness decision framework

Make the launch decision against explicit evidence:

• Capacity: Does the selected model and precision meet context, concurrency and latency targets with one expected failure present?
• Admission: Can the gateway bound expensive work and reject overload before queues violate deadlines?
• Recovery: Can a cold replacement load an approved revision within the recovery objective, and is there warm capacity during that interval?
• Isolation: Do identity, network controls, artifact access and GPU placement satisfy the actual tenant threat model?
• Change: Can the team compare a candidate release and restore the previous release without depending on mutable artifacts?

If a requirement fails, narrow the service promise, reserve more capacity or change the architecture before accepting traffic. Kubernetes remains a useful foundation, but the service contract is what determines whether the platform is ready.

For an assessment of these trade-offs in a specific environment, ape factory’s AI & Data services cover architecture and platform integration.

Primary references

These are the primary documentation entry points for review. Check release-specific behavior against the runtime, driver and cluster versions selected for deployment.

1. Kubernetes: scheduling GPUs .
2. NVIDIA GPU Operator: GPU sharing and MIG User Guide .
3. vLLM documentation : supported models, quantization and distributed serving.
4. vLLM metrics .
5. Kubernetes: horizontal pod autoscaling .
6. Kubernetes: NetworkPolicy .
7. Kubernetes: disruptions .
8. Kubernetes: pod lifecycle .
9. vLLM: Automatic Prefix Caching .
10. SGLang: Efficient Execution of Structured Language Model Programs , describing RadixAttention.
11. RFC 9113: HTTP/2, section 6.4 , stream cancellation with RST_STREAM.
12. Kubernetes: liveness, readiness and startup probes .
13. vLLM: health endpoint implementation .