Background
On some of the vault versions i.e., 1.13, 1.14 & 1.15; an issue was observed in OIDC auth method. Due to this issue; Some users part of specific groups are unable to log in with OIDC, the error being "internal error".
Issue
If a user is a member of a group that gets a policy from a namespace other than the one they are trying to log into, and that policy doesn’t exist; Vault returns an internal error.
Fix
The issue was fixed in versions 1.13.10, 1.14.6, and 1.15.2 release and is subsequently fixed in all vault versions above 1.15.2 as well. Therefore, an upgrade to the mentioned version will fix the issue.
This impacts other auth methods as well; where a non-existent policy is mapped to groups within the Vault. The article explains the use-case of OIDC auth, but the same applies to other auth methods in the Vault.
Workaround
- The creation of the non-existent policy in the concerned namespace will fix the issue.
- Deletion of the non-existent policy from the list of policies in the concerned namespace will fix the issue.
- As there can be many namespaces in your vault cluster, it is tricky to find all such namespaces without issues being reported by group members. The script below can help nest through all namespaces and find all non-existent policies mapped to certain groups:
export VAULT_ADDR=https://localhost:8200
export VAULT_TOKEN=<token_with_necessary_permissions>
for i in $(vault namespace list -format=json | jq -r '.[]'); do
echo "Looking for missing policies in: $i "
# Fetch namespace's policies
namespace_policies=()
for p in $(vault policy list -namespace=$i -format=json | jq '.[]'); do
namespace_policies+=($p)
done
# Loop through each group and check if their policies exist in the namespace
# if not, print group and missing policy
missing_count=0
for id in $(vault list -namespace=$i -format=json identity/group/id | jq -r '.[]'); do
echo "----"
echo "Group(s): " $id
for p in $(vault read -format=json -namespace=$i identity/group/id/$id | jq '.data.policies[]'); do
if [[ ${namespace_policies[@]} =~ $p ]]; then
continue
else
missing_count=$((missing_policies + 1))
echo "Policy Not Defined in namespace: " $p
fi
done
done
echo "----"
if [[ ${missing_count} -gt 0 ]]; then
echo "Remove policy mappings, or create missing policies to address issue"
else
echo "$i, No missing policies found"
fi
done
Sample output :
In the below run; the output suggests that a policy named "etc" mapped to Group Id d2a9adb0-8986-5a60-9923-7e64f17a69e9 within namespace ns1 does not exist. In ns2, there is no such non-existent policy mapped to any group.
Looking for missing policies in: ns1/
----
Group(s): 16ebb68c-9c89-bd42-02da-43b3b338bf43
----
Group(s): d2a9adb0-8986-5a60-9923-7e64f17a69e9
Policy Not Defined in namespace: "etc"
----
Remove policy mappings, or create missing policies to address issue
Looking for missing policies in: ns2/
----
Group(s): 21bd69ac-8f1e-21a7-e92c-35f29dd3e529
----
Group(s): cbe43504-4c4b-446b-14a2-28df6c7fd4de
----
ns2/, No missing policies found
Note: This policy will run through all namespace which comes as output of "vault namespace list" (when run in root namespace). Please review the script and modify it accordingly as per your use case and requirements. This is a sample shell script written only for demonstration.