Introduction :
This knowledge base article aims to guide you by providing insights and solutions to resolve issues related to ACL Auth Method login errors due to TLS authentication handshake failure while verifying certificate in your Consul deployment on ECS.
Error :
[ERROR] ACL auth method login failed: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509:
Symptoms :
- The error message indicates a failure in the ACL auth method login during the deployment of Consul on Amazon ECS (Elastic Container Service).
- The issue is related to a connection error with a specific mention of TLS certificate verification failure.
Cause :
- This error may occur when using Consul ECS version 0.7 with Terraform to deploy Consul on ECS.
- The root cause is related to the verification of TLS certificates, specifically the
consul_ca_cert_arn
configuration.
Resolution :
To address the mentioned error and successfully deploy Consul on ECS, follow these steps:
-
Locate Terraform Configuration:
- Identify the Terraform configuration files used for deploying Consul on ECS.
-
Open Terraform Configuration File:
- Open the relevant Terraform configuration file main.tf (typically with an
.tf
extension and generally named as main.tf ).
- Open the relevant Terraform configuration file main.tf (typically with an
-
Find
consul_ca_cert_arn
Configuration:- Search for the line that contains the
consul_ca_cert_arn
configuration setting.
- Search for the line that contains the
-
Comment Out or Remove the Line:
- Comment out or remove the line containing
consul_ca_cert_arn
.
Example (Commented Out):
# consul_ca_cert_arn = aws_secretsmanager_secret.ca_cert_file.arn
- Comment out or remove the line containing
-
Save and Apply Changes:
- Save the Terraform configuration file after making the necessary changes.
-
Reapply Terraform:
-
Run the Terraform commands to reapply the configuration and deploy Consul on ECS:
terraform init
terraform apply
-
-
Verify Deployment:
- After the Terraform apply process completes, verify that Consul is deployed successfully on ECS without encountering the ACL auth method login error.
Additional Note :
- By commenting out or removing the line related to
consul_ca_cert_arn
, you are adjusting the TLS certificate verification settings to resolve the issue. - Ensure that the Terraform configuration aligns with your security and certificate requirements.
- Please note that for using Consul ECS version 0.7, the supported Consul cluster version should be 1.17.x. Refer to the documentation given here
- For HCP Consul, 8502 is the gRPC port and is TLS-protected. Refer documentation given here
- Also, make sure you are using the below code snippet in
main.tf
configuration file to enable thegrpc
TLS
tls = true
http_config = {
port = 443
}
grpc_config = {
port = 8502
}
-
With the Consul ECS version
0.7
, we moved away from a Consul client-based architecture and adopted the Consul data-plane architecture. The individual ECS tasks now directly talk to the servers rather than using the client as an intermediary. Since HCP uses let's encrypt to sign server certs, you won't have to pass them explicitly. Consul data-plane will connect to the Consul servers over gRPC (which will use the let's encrypt cert) and the servers will return the mTLS cert along with Envoy configs.
References:
https://registry.terraform.io/modules/hashicorp/consul-ecs/aws/latest
https://developer.hashicorp.com/consul/docs/ecs
Conclusion :
This resolution provides a workaround for the ACL auth method login error encountered during the deployment of Consul on ECS version 0.7 using Terraform. It involves adjusting the TLS certificate verification configuration to enable a successful deployment.