featured-img

Let’s face it. Getting basic HTTP traffic routing to work in a local kind cluster is easy. It takes five minutes, a couple of dummy endpoints, and everything looks fantastic on your screen. But nobody runs unencrypted HTTP in production—unless they enjoy explaining massive data breaches to their board members.

The moment you try to take a local prototype and prepare it for the real world, you hit the security wall: TLS certificates, automated domain verification, and key management.

In the old days of ingress-nginx, automating your SSL certificates via cert-manager was standard practice. You threw a quick cert-manager.io/cluster-issuer annotation onto your Ingress manifest, crossed your fingers, and hoped the NGINX controller would correctly map the ACME HTTP-01 challenge behind the scenes.

But as we discussed in our last post, that monolithic model had a dangerous flaw: your TLS secrets had to sit exactly where your routing rules sat.

Today, we are going to look at how to build a production-ready, secure-by-default ingress layer using Kubernetes Gateway API and Cert-Manager. More importantly, we are going to dive into a crucial security mechanism that prevents your application developers from accidentally exposing or hijacking your corporate SSL certificates: the ReferenceGrant.

The Cross-Namespace Boundary Nightmare

Before we look at the code, we need to address a fundamental multi-tenant security vulnerability that plagued the legacy Kubernetes Ingress architecture.

In a traditional enterprise setup, you have a platform engineering team running the infrastructure edge, and dozens of isolated development teams running microservices in separate namespaces (payments, inventory, crm).

If your company owns a wildcard certificate for *.apefactory.com, where does that TLS Secret live? Under the old Ingress model, because the Ingress resource could only look inside its own namespace to find a TLS certificate, you had two equally terrible options:

  1. The Insecure Copy: You copied that master wildcard TLS Secret into every single developer namespace in the cluster. Now, any rogue application deployment or compromised namespace could read your master private key.
  2. The Shared Namespace: You forced all teams to deploy their Ingress rules in one single, massive, shared namespace. Goodbye isolation, hello massive blast radius.

The Gateway API completely solves this structural nightmare by turning cross-namespace references into an explicit, secure handshake.

Enter ReferenceGrant: Securing the Cross-Namespace Handshake

The Gateway API introduces a dedicated cryptographic security resource called a ReferenceGrant.

With a ReferenceGrant, a resource inside Namespace A can explicitly say: "I trust the following types of resources from Namespace B to reference me." Without this explicit, bidirectional permission, any cross-namespace reference attempted in a Gateway or an HTTPRoute will be flat-out rejected by the cluster controller.

Let’s look at a concrete real-world setup. We want our core infrastructure Gateway to live safely in a protected namespace (infra-gateways), but we want cert-manager to manage our Let’s Encrypt certificates centrally, or allow an application namespace (app-prod) to securely use a wildcard certificate without ever being able to read its raw private key.

Here is the exact YAML configuration you need to make this work.

The ReferenceGrant (Deployed in the Secret's Namespace)

If your corporate TLS secret lives in a central, locked-down namespace like infra-secrets, you deploy this manifest directly alongside it. This tells the cluster that the Gateway engine sitting in infra-gateways is officially allowed to bind to your certificate.

gatreway-certmanager-1

The Secure Gateway Listener (Deployed in infra-gateways)

Now, when your platform team sets up the public-facing edge, they configure the HTTPS listener to target port 443, set the TLS mode to Terminate, and point directly across the namespace boundary to your central secret.

gateway-certmanager-2

If you apply this Gateway without applying the ReferenceGrant first, a quick kubectl describe gateway will output a clear error: ResolvedRefs: False with the reason RefNotPermitted. The gateway will refuse to spin up the proxy listener, completely protecting your network edge from unauthorized configuration hijacking.

Automating Let's Encrypt with Cert-Manager

Manually managing static secrets is fine for an example, but in production, we want certificates to rotate automatically. Cert-Manager natively supports the Gateway API, meaning it can watch your Gateway resources, read the hostnames, talk to Let's Encrypt via ACME (HTTP-01 or DNS-01 challenges), and automatically generate the necessary secrets.

To link them together, you simply deploy a standard ClusterIssuer and configure cert-manager's gateway integration.

gateway-certmanager-3

Once your issuer is running, you don't need messy, non-portable annotations inside your routing layer. Cert-manager will spin up a transient HTTPRoute to solve the Let's Encrypt challenge challenge, drop the issued TLS certificate straight into your protected secret namespace, and the ReferenceGrant will ensure your infrastructure edge consumes it smoothly.

The Big Picture: True Least-Privilege Security

By moving away from legacy Ingress and utilizing the Gateway API security model, you have successfully decoupled your platform mechanics:

  • Your Security Team controls who can see and modify the raw private keys inside infra-secrets.
  • Your Platform Team controls what public ports are exposed and manages load balancer profiles inside infra-gateways.
  • Your Software Developers get to live entirely within their application namespaces, writing clean, vanilla HTTPRoute objects that link back to the edge without ever knowing or caring how the SSL/TLS sausage is made.

This is what a real cloud-native platform looks like: secure by default, decoupled by architecture, and completely free from proprietary vendor scripts.

In our next post, we are going to look at the market. Because the Gateway API is a specification rather than a single tool, you have to choose a concrete software engine to implement it. We are going to cut through the vendor fluff and run a definitive comparison of Envoy Gateway vs. Cilium vs. Istio vs. Kong.

 

------

Beyond Annotation Hell: The Series Roadmap

This article is part of our comprehensive guide to mastering the modern Kubernetes traffic plane. Check out the rest of the series to fully stabilize your infrastructure:

  1. Part 1: The Post-Ingress Era: Why the Kubernetes Gateway API is Taking Over

  2. Part 2: Secure by Default: Setting Up Gateway API + Free SSL (Cert-Manager)

  3. Part 3: The Blueprint: Migrating from Ingress NGINX Using ingress2gateway

  4. Part 4: The Ultimate Gateway API Provider Comparison: Cutting Through the Marketing Fluff

  5. Part 5: The Next Frontier: Elevating Gateway API to Handle LLMs, MCP, and AI Agents