Terraform Enterprise Error creating agent: "error creating data dir "/.tfc-agent": mkdir /.tfc-agent: permission denied"

Introduction

Problem

Trying to deploy TF agent on  kubernetes fails with this following error message

kubectl logs deployment/tfc-agent
Error creating agent: error creating data dir "/.tfc-agent": mkdir /.tfc-agent: permission denied"

kubectl get events
LAST SEEN   TYPE      REASON              OBJECT                                       MESSAGE
15m         Normal    Scheduled           pod/terraform-cloud--agent-b54578f5-p4xs4    Successfully assigned xxx-dev/terraform-cloud--agent-b54578f5-p4xs4 to 9c01e657-cde0-49bb-a087-0ea3344b9337
45s         Normal    Pulled              pod/terraform-cloud--agent-b54578f5-p4xs4    Container image "harbor-pks-dev01.server.rbsgrp.net/hstfe/hashicorp/tfc-agent:latest" already present on machine
13m         Warning   Failed              pod/terraform-cloud--agent-b54578f5-p4xs4    Error: container has runAsNonRoot and image has non-numeric user (tfc-agent), cannot verify user is non-root (pod: "terraform-cloud--agent-b5xxx-dev(c8caa3ff-xxxx)", container: terraform-cloud-agent)

Cause

• From YAML file securityContext (acting as UID) is set to 1000

 securityContext:
        runAsUser: 1000

• When checking from docker the UID of "tfc-agent" in this docker container is 999

docker run --rm --entrypoint /bin/id 'hashicorp/tfc-agent'
uid=999(tfc-agent) gid=999(tfc-agent) groups=999(tfc-agent)

Solutions:

• In YAML security context needs to be with correct UID of tfc user in that case 999

     securityContext:
runAsUser: 999