Introduction
Problem
When you have deployed Terraform Enterprise (TFE) Flexible Deployment Options (FDO) on Kubernetes, workspaces with a name larger than 63 characters will fail to run.
A run will get stuck in the `Plan Queued` stage.
The error that can be observed in the `task-worker.log`:
{"@level":"error","@message":"error running task instance","@module":"task-worker.executor","@timestamp":"2024-02-28T09:45:15.607799Z","err":"error creating kubernetes job: Job.batch \"tfe-task-32428532-468d-4b0e-b065-ccfa77df0271\" is invalid: [metadata.labels: Invalid value: \"this-workspace-has-a-name-with-70-characters-which-is-causing-an-issue": must be no more than 63 characters, spec.template.labels: Invalid value: \"this-workspace-has-a-name-with-70-characters-which-is-causing-an-issue": must be no more than 63 characters]"}
Prerequisites
- Terraform Enterprise Flexible Deployment Options on Kubernetes.
- A workspace with a name larger than 63 characters.
Currently all TFE FDO Kubernetes versions are affected.
Cause
-
In Kubernetes there is a limitation on the amount of characters a label can have on a job.
Overview of possible solutions (if applicable)
Solutions:
- Off load the workspace to an agent pool
-
Rename the workspace to under 63 characters
1) In the UI
- Go to the General Settings off your workspace
- Change the name in the `Name` box
- Click `Save settings` at the bottom.
2) With the API
Have the following payload.json: (replace the value for name with your new workspace name)
{
"data": {
"attributes": {
"name": "new-workspace-name-under-63-characters"
}
}
}
By having only the name attribute in the payload, this will only change the name of the workspace and leave all other settings intact.
Export a token to use with the API: (replace <your_token_here> with your actual token)
export TOKEN=<your_token_here>
From the directory where you have your `payload.json`, run the following curl command to rename your workspace: (replace <your_workspace_id_here> with your actual workspace id)
curl \
--header "Authorization: Bearer $TOKEN" \
--header "Content-Type: application/vnd.api+json" \
--request PATCH \
--data @payload.json \
https://fdo-k8s-latest.xxx.com/api/v2/workspaces/<your_workspace_id_here>
Outcome
Able to run the workspace again.