Your
Trusted
Get Fully AWS Funded
Cloud Migration
Cloud Migration
A Linkerd service mesh is easy to evaluate through the signals we can see: a successful Terraform apply, a healthy Helm release, and a certificate with a distant expiry date. Those signals matter. But they do not prove that the Linkerd control plane has loaded the certificate and private key that it was given.
That distinction matters at the admission boundary. An expired or unusable certificate on a Linkerd admission webhook can mean that a workload is admitted without the expected mutation. In the most visible case, a new pod that is eligible for Linkerd proxy injection starts without the linkerd-proxy sidecar. The deployment succeeds; the expected service-mesh behavior simply never appears.
This is a failure mode to design for during Kubernetes upgrades, certificate changes, and Terraform module changes: validate the material, then prove that the runtime consumer can use it.
Kubernetes admission webhooks are HTTP callbacks in the API-server request path. Mutating webhooks run first and may modify the incoming object; validating webhooks run after mutation and may reject it. A Linkerd admission webhook is therefore control-plane infrastructure, not a background helper.
Linkerd uses Kubernetes admission webhooks for core features including automatic proxy injection and service-profile validation. For an eligible pod, the proxy injector mutates the pod specification before it is persisted. The result should be observable in the final PodSpec: the Linkerd data-plane proxy is present alongside the application containers.
The API server reaches an in-cluster webhook through the Service reference in the webhook configuration. TLS is part of that path. Kubernetes requires the server certificate to be valid for the Service DNS name and validates it against the configured PEM CA bundle. A certificate may look well-formed in isolation while still failing this request path because of expiry, an incorrect SAN, an incorrect CA bundle, or a key the serving process cannot load.
Kubernetes decides what to do when a webhook call fails through failurePolicy. With failurePolicy: Fail, a connection failure or timeout rejects the API request. With failurePolicy: Ignore, Kubernetes allows the request to continue without the webhook result.
The latter can be an intentional availability choice. It also creates an observability obligation. If a proxy-injector webhook is configured to fail open and its TLS call fails, the workload may still be created. There may be no Terraform error, no failed Helm release, and no application deployment error. The missing mutation is the signal.
Do not treat “silent” as a universal property of Linkerd or of Kubernetes. It is configuration-dependent: inspect the actual MutatingWebhookConfiguration and its failurePolicy in the version and chart values running in the cluster. Kubernetes documents Ignore and Fail behavior explicitly; failurePolicy is the control that determines whether a webhook-call error is bypassed or causes admission to fail.
Linkerd separates webhook TLS credentials from the mTLS identity credentials used by data-plane proxies. The current Linkerd documentation describes webhook TLS as a separate trust chain and lists the webhook secrets used by the control plane. When Linkerd CLI or Helm manages these credentials, an upgrade can regenerate them; environments that need scheduled rotation can use cert-manager.
The source change behind this article found two distinct risks. First, chart-generated webhook certificates had a one-year validity period and no automatic rotation in that deployment model. Second, static certificate inspection did not reveal that an ECDSA private key encoding was unacceptable to one control-plane consumer. The Rust/rustls-based policy controller could not load the SEC1-form key emitted by the Terraform TLS provider, even while the certificate, subject alternative names, and certificate/key relationship looked correct outside the cluster. A PKCS#8 key was required for a runtime-compatible result.
The broader rule is portable: certificate compatibility is a tuple, not a date field. Confirm the issuing chain, CA bundle, Service DNS names, validity window, and the precise encoding and algorithm accepted by the component that loads the key.
Certificate path passing through a runtime verification point
Terraform fmt, validate, tests, plan, and apply all answer valuable but limited questions. They can establish that values exist, that a Helm release accepts them, and that a certificate parses. They cannot automatically establish that every control-plane binary consumes the key successfully.
This layered proof catches two different classes of defects: “the infrastructure rendered the material” and “the control plane used the material.” Both are necessary.
Use this sequence when preparing a Kubernetes platform change:
"A certificate is not “good” because Terraform generated it. It is good when the Kubernetes API server can validate it, the webhook process can load its key, and an admission request produces the expected observable result."
For a Linkerd service mesh, the most practical runtime assertion is simple: an eligible test workload receives the expected proxy injection after the change. Treat that assertion as a release gate, particularly when rotating certificates, upgrading Kubernetes, changing Helm values, or altering Terraform modules.
Planning a Kubernetes platform change? Let’s compare your upgrade-verification approach before the next production rollout.