Troubleshooting ACL Auth Method Login Error in Consul ECS Deployment using Terraform

Avatar
Abhineet Khanwalkar
  • Updated

Introduction : 

This knowledge base article aims to guide you by providing insights and solutions to resolve issues related to ACL Auth Method login errors due to TLS authentication handshake failure while verifying certificate in your Consul deployment on ECS.

 

Error :

[ERROR] ACL auth method login failed: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509:

 

Symptoms :

  • The error message indicates a failure in the ACL auth method login during the deployment of Consul on Amazon ECS (Elastic Container Service).
  • The issue is related to a connection error with a specific mention of TLS certificate verification failure.

 

Cause :

  • This error may occur when using Consul ECS version 0.7 with Terraform to deploy Consul on ECS.
  • The root cause is related to the verification of TLS certificates, specifically the consul_ca_cert_arn configuration.

 

Resolution :

To address the mentioned error and successfully deploy Consul on ECS, follow these steps:

  1. Locate Terraform Configuration:

    • Identify the Terraform configuration files used for deploying Consul on ECS.

  2. Open Terraform Configuration File:

    • Open the relevant Terraform configuration file main.tf (typically with an .tf extension and generally named as main.tf ).

  3. Find consul_ca_cert_arn Configuration:

    • Search for the line that contains the consul_ca_cert_arn configuration setting.

  4. Comment Out or Remove the Line:

    • Comment out or remove the line containing consul_ca_cert_arn.

    Example (Commented Out):

    # consul_ca_cert_arn = aws_secretsmanager_secret.ca_cert_file.arn

  5. Save and Apply Changes:

    • Save the Terraform configuration file after making the necessary changes.

  6. Reapply Terraform:

    • Run the Terraform commands to reapply the configuration and deploy Consul on ECS:

      terraform init
      terraform apply

  7. Verify Deployment:

    • After the Terraform apply process completes, verify that Consul is deployed successfully on ECS without encountering the ACL auth method login error.

 

Additional Note :

  • By commenting out or removing the line related to consul_ca_cert_arn, you are adjusting the TLS certificate verification settings to resolve the issue.
  • Ensure that the Terraform configuration aligns with your security and certificate requirements.
  • Please note that for using Consul ECS version 0.7,  the supported Consul cluster version should be 1.17.x. Refer to the documentation given here 
  • For HCP Consul, 8502 is the gRPC port and is TLS-protected. Refer documentation given here 
  • Also, make sure you are using the below code snippet in main.tf configuration file to enable the grpc TLS 
tls = true

http_config = {
  port = 443
}

grpc_config = {
  port = 8502
}
  • With the Consul ECS version0.7 , we moved away from a Consul client-based architecture and adopted the Consul data-plane architecture. The individual ECS tasks now directly talk to the servers rather than using the client as an intermediary. Since HCP uses let's encrypt to sign server certs, you won't have to pass them explicitly. Consul data-plane will connect to the Consul servers over gRPC (which will use the let's encrypt cert) and the servers will return the mTLS cert along with Envoy configs.      

 

References:

https://registry.terraform.io/modules/hashicorp/consul-ecs/aws/latest 
https://developer.hashicorp.com/consul/docs/ecs 

 

Conclusion :

This resolution provides a workaround for the ACL auth method login error encountered during the deployment of Consul on ECS version 0.7 using Terraform. It involves adjusting the TLS certificate verification configuration to enable a successful deployment.

Articles in this section

Related articles