How to integrate CircleCI with Vault using JWT/OIDC and how Entities/Clients are counted

To integrate the CircleCI with Vault using JWT/OIDC, we must have the following setup at a minimum:

1. A running HashiCorp Enterprise Vault or an HCP Vault Cluster. This article was tested on Vault v1.15.2+ent.
2. A CircleCI managed service account.
3. A GitHub, GitLab, or Bitbucket project to control the source code.

4. You must also gather the following information from the CircleCI UI:

   • Your organization Name and ID: Open the CircleCI web app and navigate to Organization Settings > Overview on the CircleCI web app to find both.
   • Your project ID: Open the CircleCI web app and navigate to your project’s page and click on Project Settings > Overview.

Configure Vault:

:~$ vault auth enable jwt

# Policy is to only read a KV2 secret stored at the path "secret/data/circleci-demo/".
:~$ vault policy write circleci-demo-policy - <<EOF
path "secret/data/circleci-demo/*" {
capabilities = ["read", "list"]
}
EOF

:~$ vault write auth/jwt/config \
oidc_discovery_url="https://oidc.circleci.com/org/<your org id>" \
bound_issuer="https://oidc.circleci.com/org/<your org id>"

:~$ vault write auth/jwt/role/circleci-demo -<<EOF
{
"role_type": "jwt",
"user_claim": "sub",
"user_claim_json_pointer": "true",
"bound_claims": { "oidc.circleci.com/project-id": "<your project id>" },
"policies": ["default", "circleci-demo-policy"],
"ttl": "10m"
}
EOF

:~$ vault secrets enable -version=2 -path=secret kv

:~$ vault kv put secret/circleci-demo/demo-secrets \
username="paul.atreides" \
password="lis@n-al-g4ib"

Configure CircleCI:

• After creating a CircleCI managed service account, the next step is to add GitHub, GitLab, or Bitbucket project to connect to the source code. In my case, I used Github, and
  • I created a repo name "circleci" on Github.
  • Connected my Github repo with the CircleCI organization, and gave the repo READ & WRITE permissions. (The permissions are subject to change based on the use cases.)

• • Added Github as a project trigger to execute the pipeline post the changes in source code is merged to the branch.
    
    • At Github:

• • • At CircleCI:

• Instead of putting the Vault Server endpoint and role name into version control where they could be inadvertently leaked, we can store them securely in a CircleCI context.

• Now we’ll have to write a CircleCI config file that uses a CircleCI OIDC token to authenticate with Vault, uses Vault’s auto-auth functionality to retrieve the secrets we created, and then finally exports them as environment variables for use within a CircleCI job.

  In the repository you created for this article, create a file at .circleci/config.yml.
  
  

• Next, create a templated Vault agent config file at .circleci/vault/agent.hcl.tpl. This file will tell the Vault agent which Vault server endpoint to connect to and how it should authenticate.
  
  
• Finally, we will create a Consul template that will tell the Vault agent what to do with the secrets it retrieves. In this article, we will simply export them as environment variables for use in our pipeline. Create a file at .circleci/vault/secrets.ctmpl.
  
  
• All the codes, that needs to go in .circleci/config.yml , .circleci/vault/agent.hcl.tpl , and .circleci/vault/secrets.ctmpl can be mocked from here.
  
  

 CircleCI authentication against Vault:

If everything has been configured correctly, you should see the final step print the secrets that were retrieved from Vault as shown below:

Note: To trigger the pipeline again, make any minor change (maybe update any comment in the code) to the source code in giuthub.

Possible Issues during runtime:

• If you end up facing the following error during the pipeline execution:

[ERROR] agent: runtime error encountered:
  error=
  | Error making API request.
  | 
  | URL: PUT https://$VAULT_ADDR/v1/auth/jwt/login
  | Code: 400. Errors:
  | 
  | * error validating token: invalid not before (nbf) claim: token not yet valid
   exitCode=1

• • You must need to disable all the "leeway"s temporarily on JWT/OIDC auth role by running and later set them as per the need:

 :~$ vault write auth/jwt/role/circleci-demo \
role_type="jwt" not_before_leeway="-1" \
clock_skew_leeway="-1" expiration_leeway="-1"

• If you end up facing the following error during the pipeline execution:

[ERROR] agent: runtime error encountered: error="Put \
"http://$VAULT_ADDR/v1/auth/jwt/login\": 
dial tcp $VAULT_ADDR: i/o timeout" exitCode=1

• • This could probably be because of the CircleCI is not able to communicate with the VAULT_ADDR because it has IP in its address.
    • CircleCI has their own set of IP ranges which needs to be whitelisted in the .circleci/config.yml by setting circleci_ip_ranges: true inside "jobs->build" for enabling the IP based access.
    • However, this feature is not available on free tier of CircleCI. So,
    • Another alternative is to use a public utility called "ngrok" to advertise your IP based Vault address to a random name based address.

Vault Client/Entity Count:

Vault counts the clients based on the uniqueness of userID/projectID/organizationID. Following are the entities with their aliases/clients created on Vault from the two different users who triggered the same pipeline within the same project and organization. The format of the entity aliases will be "org/organizationID/project/projectID/user/userID" for distinguishing the new clients.

Thank you.