How to map an Agent pod to a specific Run

Use Case

When troubleshooting the stuck runs on the Kubernetes or OpenShift deployment of Terraform Enterprise, you may need to locate the specific Agent pod in order to identify the reason why the run is not processing. 

Prerequisites 

• Terraform Enterprise deployed on Kubernetes or OpenShift

Procedure

• From TFE UI, get a Run ID.

• Authenticate to your cluster and execute kubectl get pods -n <agent_namespace> --show-labels | grep <run_id> command.
  
  Example and output:

  kubectl get pods -n terraform-enterprise-agents --show-labels | grep run-V1QUQ4o12HG1evbK
  
  tfe-task-65aee913-5194-4b78-8715-8bdae5becc52-k5fkh 1/1 Running 0 4m4s app=terraform-enterprise,batch.kubernetes.io/controller-uid=3e2e8c93-11b0-42f8-835b-5603034e5e99,batch.kubernetes.io/job-name=tfe-task-65aee913-5194-4b78-8715-8bdae5becc52,controller-uid=3e2e8c93-11b0-42f8-835b-5603034e5e99,job-name=tfe-task-65aee913-5194-4b78-8715-8bdae5becc52,organization_name=test_organization,run_id=run-V1QUQ4o12HG1evbK,run_type=Apply,workspace_name=test_workspace

• Use the Pod name to run kubectl describe pod <pod_name> -n <agent_namespace> to obtain the details.
  

  Example and possible output:

  kubectl describe pod tfe-task-65aee913-5194-4b78-8715-8bdae5becc52-k5fkh -n terraform-enterprise-agents
  ...
  Labels: app=terraform-enterprise
  ...
  organization_name=test_organization
  run_id=run-V1QUQ4o12HG1evbK
  run_type=Apply
  workspace_name=test_workspace
  ...
  Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 1s (x2 over 5m8s) default-scheduler 0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  

• Alternatively, you can run kubectl logs <pod_name> -n <agent_namespace> command to display the logs.