Issue
Trying to login/authenticate using the TLS cert authentication method when Vault is deployed behind an AWS Application Load Balancer "ALB" or an Network Load Balancer "NLB" fails with error message:
$ vault login -method=cert -client-cert=client.crt -client-key=client.key name=role
Error authenticating: Error making API request.
URL: PUT https://vault-lb.example.com/v1/auth/cert/login
Code: 400. Errors:
* 1 error occurred:
* error running lookahead function for mfa: no client certificate found
Cause
- Clients trying to authenticate or login to vault through an AWS Application Load Balancer "ALB" or Network Load Balancer "NLB" and the Load Balancer is not configured properly to send or forward client certificates to Vault.
- A TLS client certificate is required for the TLS cert authentication method to perform mutual TLS authentication (mTLS) with the client and will fail if the client certificate is not provided by the load balancer which is the case with the AWS NLB with a TLS listener that is terminating the TLS connection on the load balancer side.
- Or, an AWS ALB is used without adding the required parameters to Vault listener stanza in Vault 1.17 or higher.
- Or, an AWS ALB is used with Vault versions below 1.17.0.
Solutions:
-
AWS Network Load Balancer "NLB" with a "TLS" listener configured on the NLB:
- An AWS NLB with a TLS listener can not be used with the TLS cert auth method as it is terminating the TLS connection on the load balancer side without forwarding the client certificates to Vault, at the time of writing the knowledge base article, the recommendation from AWS is to use a TCP listener instead:
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html#secure-listeners
"Network Load Balancers do not support TLS renegotiation or mutual TLS authentication (mTLS). For mTLS support, create a TCP listener instead of a TLS listener. The load balancer passes the request through as is, so you can implement mTLS on the target." - Change AWS Network Load Balancer listener type to use TCP instead of TLS and make sure TLS certificates that were used on the NLB are now configured on Vault nodes' TCP listener stanza.
- An AWS NLB with a TLS listener can not be used with the TLS cert auth method as it is terminating the TLS connection on the load balancer side without forwarding the client certificates to Vault, at the time of writing the knowledge base article, the recommendation from AWS is to use a TCP listener instead:
-
AWS Application Load Balancer "ALB" or similar + Vault 1.17.0 or higher:
- Starting with Vault 1.17.0, Vault can now read the forwarded client TLS certificate from an application level "layer 7" load balancer or a reverse proxy by adding the expected HTTP header that is being used by the load balancer or reverse proxy to forward the client TLS certificate & the decoders to Vault TCP listener configuration then restart Vault nodes to pick up the changes.
- The required parameters are x_forwarded_for_client_cert_header and x_forwarded_for_client_cert_header_decoders.
- Refer to the AWS ALB or the reverse proxy documentation for the correct values for both parameters.
- On a AWS ALB, the "Mutual TLS passthrough" option should be selected to instruct the ALB to forward the client certificate to Vault service.
-
AWS Application Load Balancer "ALB" or similar + Vault 1.16.0 or lower:
- Support for forwarded client certificates via HTTP headers was introduced in Vault 1.17.0 or higher.
- Either upgrade Vault to version 1.17.0 or higher and configure the Application Load balancer and Vault to use the HTTP headers as seen in the point above.
- Or, use a Network Load Balancer "NLB" with a "TCP" listener so that the NLB will passthrough all TLS certificates to Vault.
Outcome
Once the load balancer is configured properly, and all other conditions are met and the TLS cert authentication method is configured properly then the authentication should succeed as follows:
$ vault login -method=cert -client-cert=client.crt -client-key=client-key.key name=role
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.CAESIGLiaop6pWWMv..TDduSHBxbkNyWVYwMXpGekNMWEc
token_accessor YNU58o0fKzLUnM4vJpE8FTmD
token_duration 768h
[..]
Additional Information
- KB Article: Best Practices - AWS NLB configuration for Vault
- Vault Documentation: TLS cert auth method
- GitHub Issue: Add support for forwarded Tls-Client-Cert