Introduction
Amazon Elastic Kubernetes Service (EKS) is a powerful managed Kubernetes service that simplifies the deployment and management of containerised applications. Consul is a widely used service mesh and service discovery tool that can be integrated with EKS to enhance networking and security for your applications. However, like any complex system, issues may arise during the installation and configuration process. This knowledge-based article provides a step-by-step guide to troubleshooting common problems when installing Consul on Amazon EKS.
Prerequisites
Before proceeding with the troubleshooting steps, ensure that you have the following prerequisites in place:
- An AWS account with Amazon EKS configured.
- A Kubernetes cluster set up on Amazon EKS.
-
helmandkubectlinstalled on your local machine. - A basic understanding of Amazon EKS and Helm Charts.
Scenario 1 -
While installing Consul on an Amazon EKS cluster, you may encounter the "UnauthorizedOperation" error (See below block for full error description), indicating a lack of permissions to create Amazon Elastic Block Store (EBS) volumes.
status code: 403, request id: 185cf3dd-69ef-4fe0-a4a4-27c7e560ff5a
Warning ProvisioningFailed 11m ebs.csi.aws.com_ebs-csi-controller-5f75f65dc6-l78vq_7197af03-2a62-4cb7-91a6-81c7a6410ac1 failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-365d13b6-78bd-4736-8a85-f54bbc5b4775": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation
Solution -
Step 1 - Identify the IAM Role:
Determine the IAM role associated with your Amazon EKS cluster. This role is typically attached to the worker nodes and controls the permissions of your EKS cluster.
Step 2 - Grant EBS Permission:
To resolve the "UnauthorizedOperation" error, you need to grant the necessary permissions to the IAM role identified in Step 1. You will need to attach an IAM policy arn, that is "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy", to this role that allows the creation of EBS volumes.
Step 3 - Validate "Amazon EBS CSI Driver" Addon enablement:
Validate in your Amazon EKS cluster whether the "Amazon EBS CSI Driver" add-on is enabled and attached to it or not.
Ensure that this EBS CSI Driver add-on's Service account role should be set to "Inherited from node".
Also, Amazon EKS doesn't automatically update Amazon EBS CSI for your cluster when new versions are released or after you update your cluster to a new Kubernetes minor version. To update Amazon EBS CSI on an existing cluster, you must initiate the update and then Amazon EKS updates the add-on for you.
Step 4 - Retry Consul Installation:
With the updated IAM role permissions, retry the installation of Consul on your EKS cluster. Ensure that the necessary storage classes and resource requirements are correctly configured in your Consul Helm chart.
Step 5 - Monitor Installation Progress
Check the installation progress by running the command:
kubectl get pods -n consul
Ensure that all Consul pods are in the Running or Completed state without encountering EBS-related errors.
Scenario 2 -
While Installing Consul on an Amazon EKS cluster, you may encounter the "connection error" error in connect-injector pods.
[INFO] consul-server-connection-manager: trying to connect to a Consul server
[ERROR] consul-server-connection-manager: connection error: error="failed to discover Consul server addresses: failed to resolve DNS name: consul-server.consul.svc: lookup consul-server.consul.svc on X.X.X.X:53: no such host"
Solution -
Step 1 - Identify the IAM Role:
Determine the IAM role associated with your Amazon EKS cluster. This role is typically attached to the worker nodes and controls the permissions of your EKS cluster.
Step 2 - Grant VPC CNI Permission:
To resolve the error, you need to grant the necessary permissions to the IAM role identified in Step 1. You will need to attach an IAM policy arn, that is "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy", to this role that allows the creation of EBS volumes.
Step 3 - Validate "Amazon VPC CNI" Addon enablement:
Validate in your Amazon EKS cluster whether the "Amazon VPC CNI" add-on is enabled and attached to it or not.
Also, ensure that this VPC CNI add-on's Service account role should be set to "Inherited from node".
Step 4 - Redeploy the connect-injector pods:
Delete the connect-injector pods using the command:
kubectl delete pod <connect-inject-pod-full-name> -n consul
Step 5 - Monitor the new connect-injector pod's log:
Check the logs of connect-injector pods by running the command:
kubectl logs -f <new-connect-inject-pod-full-name> -n consul
Ensure that all Consul pods are in the Running or Completed state without encountering a "connection error".
Scenario 3 -
After creating an EKS cluster, when you configure your local machine (using aws eks update-kubeconfig --region <region where you deployed your cluster> --name <your cluster name> command), where you want your local kubectl agent to talk to your provisioned EKS cluster, then sometimes you face the error "Unknown output type: JSON"
Unknown output type: JSON
Unable to connect to the server: getting credentials: exec: executable aws failed with exit code 255
Solution -
- You can check the currently configured output format for the AWS CLI by using the
aws configure listcommand. This command will display a list of AWS CLI configuration settings, including the currently configured output format. Look for the "output" setting in the output of theaws configure listcommand. - Validate whether your "output" setting is configured to use the "json" format. If your "output" setting is configured differently, you can use the aws configure set output command to change it to your desired format. For this error, you need to set it to "json", you can run:
aws configure set output json
This will update the AWS CLI configuration to use JSON as the output format.
- Validate the output by running
kubectl get pods. This will discplay the correct output without any "Unknown output type" error.
Conclusion
By following the above troubleshooting steps for multiple scenarios, you should be able to resolve these errors. This will allow you to successfully deploy Consul on your EKS cluster and enhance the networking and security of your containerized applications. If you encounter persistent issues or require further assistance, consult the AWS documentation or seek support from the AWS community forums.
Reference