You may incur a general Forbidden type response when attempting to login into a newly setup Kubernetes authentication mount or when dealing with existing mounts that were previously working. The error may resemble:
[DEBUG] auth.kubernetes.auth_kubernetes_e8b38ff8: login unauthorized: err="Post \"https://10.0.191.36:6443/apis/authentication.k8s.io/v1/tokenreviews\" Forbidden"
The port mentioned in error message 6443is a clue to where the problems resides namely the Kubernetes API server (on K8S API server / nodes) cannot be reached and thus the issue is likely network related.
Using curl confirm that the route as well as responses are directly from Kubernetes and not from an intermediate appliances or device such as a Load-Balancer or proxies for example.
curl -v -k https://10.0.191.36:6443
The lines with a less than symbol (<) are HTTP responses from the server which we're interested in reviewing:
< НТТР/1.1 403 Forbidden
< Server: squid/4.15
< Mime-Version: 1.0
< Date: Thu, 24 Apr 2025 12:24:21 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3473
< X-Squid-Error: ERR_ACCESS_DENIED 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from proxy2-test
< X-Cache-Lookup: NONE from proxy2-test:3128
< Via: 1.1 proxy2-test (squid/4.15)
< Connection: keep-alive
Here it's clear that the 403 response and error are actually from a Squid proxy and not directly from the Kubernetes API.
In these scenarios check your network configurations including any parameters such as: NO_PROXY & HTTPS_PROXYthat may be needed in the Vault systemd unit-file definition (systemctl show -P FragmentPath vault.service).
After making the required change confirm that the upstream Kubernetes API server is now in reach without any intermediate responses from other devices; thereafter the mount and login request that were previously working should be restored or otherwise you should be able to proceed further with your setup.