Troubleshooting "tls: failed to verify certificate: x509: certificate signed by unknown authority" for Consul Connect Injector with external Kubernetes API (RKE)

Avatar
Himanshu Sharma
  • Updated

Summary

In some multi‑cluster Consul deployments, Consul components such as the connect injector may fail ACL login with errors similar to:

rpc error: code = Unauthenticated desc = Post "https://<kubernetes-api-endpoint>:6443/apis/authentication.k8s.io/v1/tokenreviews": tls: failed to verify certificate: x509: certificate signed by unknown authority

This article explains:

  • When this can happen in Consul environments that use an external TLS endpoint in front of the Kubernetes API.

  • Why simply changing the Consul auth method name for the injector does not resolve the issue.

  • How to resolve the problem by updating Consul’s Kubernetes auth methods to trust the correct CA chain.

This behavior is related to TLS and certificate trust, not to a specific Kubernetes distribution.

Affected scenario (high‑level)

You may be affected if:

  • Consul servers (control plane) run in one Kubernetes cluster.

  • Consul dataplane components (e.g., connect injector, sidecars) run in another Kubernetes cluster.

  • The dataplane Kubernetes API is exposed through an external TLS proxy/Ingress (for example, a load balancer or ingress controller) that terminates TLS with its own CA chain (different from the cluster’s original Kubernetes CA).

  • Consul is configured to use Kubernetes ACL auth methods (auth-method type kubernetes) for ACL login.

Symptoms

On the dataplane cluster, the Consul connect injector pod:

  • Logs repeated ACL login failures like:

rpc error: code = Unauthenticated desc = Post "https://<kubernetes-api-endpoint>:6443/apis/authentication.k8s.io/v1/tokenreviews": tls: failed to verify certificate: x509: certificate signed by unknown authority

  • Often goes into CrashLoopBackOff, with startup probes failing, for example:

Warning Unhealthy Startup probe failed: Get "http://<pod_ip>:9445/readyz/ready": dial tcp <pod_ip>:9445: connect: connection refused Warning BackOff Back-off restarting failed container sidecar-injector

You may also observe that:

  • The injector container command includes a custom auth method flag, e.g.:

-acl-auth-method="custom-auth-method"

  • But the environment still shows:

CONSUL_LOGIN_AUTH_METHOD: consul-k8s-component-auth-method

and the injector continues to fail login with the same TLS error.

Why this happens

1. TLS trust for the Kubernetes API token review

For Kubernetes auth methods, the Consul server validates service account tokens by sending TokenReview requests to the Kubernetes API:

POST https://<kubernetes-api-endpoint>:6443/apis/authentication.k8s.io/v1/tokenreviews

To establish this TLS connection, the Consul server must trust the certificate chain used by <kubernetes-api-endpoint>. That chain is configured on the auth method via kubernetes-ca-cert (or equivalent).

If the auth method only trusts the original Kubernetes cluster CA, but your API is exposed through a different external CA chain (for example, from an ingress or proxy), the Consul server cannot validate the TLS certificate and you see:

tls: failed to verify certificate: x509: certificate signed by unknown authority

2. Default Kubernetes auth methods vs. custom auth method

When Consul is deployed via Helm on Kubernetes, it typically creates default Kubernetes ACL auth methods such as:

  • consul-k8s-auth-method

  • consul-k8s-component-auth-method

These default methods:

  • Usually trust the cluster’s built‑in Kubernetes CA.

  • Do not automatically include any external TLS proxy/Ingress CA chain.

Although you can create a custom auth method and set connectInject.overrideAuthMethodName to use it, the injector still has:

CONSUL_LOGIN_AUTH_METHOD: consul-k8s-component-auth-method

Internally, the injector continues to use this default method for login, so it still relies on an auth method that doesn’t trust the external API’s CA chain. The result is the same x509: certificate signed by unknown authority error.

Resolution

The recommended resolution is to keep using the default Kubernetes auth methods but update them to trust the full external CA chain used by the Kubernetes API endpoint.

Step 1 – Prepare the full CA chain file

From a system that can reach the external Kubernetes API endpoint (for example https://<kubernetes-api-endpoint>:6443):

  1. Export the full certificate chain (root + intermediate(s) + leaf) used by that endpoint.

  2. Save it as a file, for example:

ca-full-chain.pem

  1. Verify that this chain validates the endpoint:

openssl s_client -connect <kubernetes-api-endpoint>:6443 -CAfile ca-full-chain.pem 



# Expect: Verification: OK

If verification succeeds, the chain in ca-full-chain.pem is what Consul’s Kubernetes auth methods need to trust.

Step 2 – Update the default Consul Kubernetes auth methods

On a host where you can run consul CLI with permissions to manage ACL auth methods, update the default methods to use the full chain:

consul acl auth-method update \ -name consul-k8s-auth-method \ -kubernetes-ca-cert=@/path/to/ca-full-chain.pem 



consul acl auth-method update \ -name consul-k8s-component-auth-method \ -kubernetes-ca-cert=@/path/to/ca-full-chain.pem

Reference:

After this, when Consul servers call the Kubernetes API at the external endpoint, they can validate the certificate with the correct CA chain.

Step 3 – Simplify the connect injector configuration

In your dataplane Helm values (or equivalent configuration for the connect injector):

  • Remove the connectInject.overrideAuthMethodName entry, for example:

connectInject: 
  enabled: true 
  default: true 
  replicas: 3 



# Do NOT set overrideAuthMethodName here

  • Ensure that the external Kubernetes API endpoint is correctly configured in your Consul/Helm values (for example, a value like externalServers.k8sAuthMethodHost: "https://<kubernetes-api-endpoint>:6443" if applicable).

With the default auth methods now trusting the external CA chain, the injector can successfully log in using CONSUL_LOGIN_AUTH_METHOD=consul-k8s-component-auth-method.

Step 4 – Restart the injector if needed

If the injector is stuck in CrashLoopBackOff:

  • Restart the deployment:

kubectl rollout restart deployment consul-connect-injector -n <consul-namespace>

Or re‑run your Helm upgrade after removing overrideAuthMethodName to ensure the updated configuration is applied.

Verification

  1. Check injector pod status and environment

kubectl describe pod -l component=connect-injector -n <consul-namespace>

Confirm:

  • The pod is in Running status.

  • Readiness and startup probes are successful.

  • CONSUL_LOGIN_AUTH_METHOD (if present) points to a default Kubernetes auth method (for example, consul-k8s-component-auth-method).

  1. Review injector logs

kubectl logs -l component=connect-injector -n <consul-namespace>

Verify that there are no new occurrences of:

tls: failed to verify certificate: x509: certificate signed by unknown authority

  1. Validate mesh behavior

  • Deploy or restart a test workload with Consul sidecar injection enabled.

  • Confirm that sidecars start normally and are able to obtain ACL tokens from Consul without TLS/ACL errors.

Notes and caveats

  • Helm upgrades and auth methods
    Some Helm upgrades may recreate or modify the default Kubernetes auth methods (for example, via an ACL init job). If that happens, the kubernetes-ca-cert configuration may revert to the original Kubernetes CA. In that case, you might need to reapply the consul acl auth-method update commands above.

  • Custom auth methods vs. patching defaults
    While it is possible to create a custom Kubernetes auth method with the correct CA chain and point some components at it, this usually requires additional binding rules and manual management. In many cases, updating the existing default auth methods to trust the external CA chain is simpler and better aligned with future improvements in Consul’s Kubernetes integration.

Articles in this section

See more

Related articles