Troubleshoot Frequent Timeout to Kubernetes Control Plane API or Cluster Unreachable Issues

Introduction

This article walks you through the general troubleshooting steps for frequent timeouts to kubernetes control plane API failing with error - Error: failed to create kubernetes rest client for read of resource: Get "https://xxxxxxxx.eks.amazonaws.com/api?timeout=32s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

                                                                      OR

Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials

Problem

These frequent timeouts to Kubernetes cluster causes speculative and apply plans to fail until the timeout issue goes away.

Prerequisites (if applicable)

• Running Kubernetes (K8s) Cluster 

Cause

• One of the reasons that cluster becomes unreachable is due to aws_eks_cluster_auth used for cluster authentication.  Since in this mechanism the credentials gets cached, any apply or plan approved later than the expiration of the temporary token fails, as the token is no longer valid.
  
  
• Timeout issues can also occur when we try to create CRD with kubernetes and a CR in the same workspace which is not recommended. We have the best practice to work with k8s resources and cluster to have them in separate apply operations as mentioned here.

Possible Solutions:

• Since the token generated using data.aws_eks_cluster_auth are short-lived with a non-configurable 15 mins timeout, the apply results in the failure with the error - 'the server has asked for the client to provide credentials'. To avoid these connection failures, you need to ensure that the time after your Pull Request (PR) is merged and the apply is triggered is not exceeding the default 15 mins timeframe.

• You can check out the TFC Workload Identities aka Dynamic Credentials. This is the most flexible option, but requires a bit of upfront configuration.
  Configuring the integration requires the following steps:

  1. Configure Kubernetes
  2. Configure HCP Terraform
  3. Configure the Kubernetes or Helm provider

  Once the setup is completed, HCP Terraform automatically authenticates to Kubernetes during each run where the authentication is valid for the length of a plan or apply operation.

Outcome

Using dynamic credentials, HCP Terraform authenticates to K8s cluster and the credentials are valid for the entire length of execution. This way there is no cluster connection or unreachable issues encountered.

Additional Information

For additional assistance please contact HashiCorp Support