Introduction
Problem
A Sentinel Policy Set has been added from a Github repository (VCS connected) and Terraform Enterprise (TFE) is not updating the policy set to the latest commit available in Github.
Prerequisites
- TFE application
- A VCS connected Sentinel policy set
Cause
- The Github webhooks are broken
How to confirm the Cause
- Use the Policy Sets Terraform API to get the webhook URL of the Sentinel Policy
curl --request GET \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/vnd.api+json" \
https://<your-tfe-hostname>/api/v2/policy-sets/polset-<ID>?include=current_version- We can generate an API Token from the TFE installation and use it as the value for $TOKEN
- Fill in the TFE hostname
- Get the polset-<ID> from TFE at Policy Sets > Select the Sentinel Policy that is failing.
- Check in the browser what is the URL of the Sentinel Policy.The URL should be similar to:
https://<your-tfe-hostname>/app/<your-org-name>/settings/policy-sets/polset-cK6KnrpQwpMhbB52/edit
- Copy that polset-cK6KnrpQwpMhbB52 ID and use it in your API call for polset-<ID>
- We will be getting an output that will return, among other values, a webhook URL like this:
"webhook-url":"https://<your-tfe-hostname>/webhooks/vcs/2641bdca-09ad-4252-bd9b-fbba8a091dc2"
- Copy the ID at the end of the URL, in this case it would be 2641bdca-09ad-4252-bd9b-fbba8a091dc2
- Go to Github and verify that this is the same webhook that is used in your Github repository
- Verify that the webhook is received by the NGINX Docker container of TFE
TFE Replicated version
# NOTE to replace the ID used here with your own value
docker logs tfe-nginx 2>&1 | grep 2641bdca-09ad-4252-bd9b-fbba8a091dc2
TFE FDO
# NOTE to replace the ID used here with your own value
docker exec terraform-enterprise-tfe-1 grep 2641bdca-09ad-4252-bd9b-fbba8a091dc2 /var/log/terraform-enterprise/nginx.log
- Confirm the output is similar to the following, with the status code of 200
140.82.115.148 - - [06/Sep/2023:12:33:58 +0000] "POST /webhooks/vcs/641bdca-09ad-4252-bd9b-fbba8a091dc2 HTTP/1.1" 200 12 "-" "GitHub-Hookshot/435afa9"
- If the output is different or you don't get any result, try the following:
- Confirm on your server that nothing is blocking the network like a Firewall/WAF/Proxy
- Confirm the certificate used on TFE for TLS is trusted by the VCS. This can be done by executing the below command on your VCS environment and should return an OK response
curl https://<your_TFE_fqdn>/_health_check
Solutions:
-
Fix the broken webhooks from Github
Outcome
If you are still experiencing issues and the webhooks are correct, please open a support ticket with the following information attached:
- Share the commit numbers that are on your Github repository and not in TFE
- The webhook URL you got from the API call
- A support bundle
- The output of the CURL command
curl https://<your_TFE_fqdn>/_health_check