Introduction
This article provides troubleshooting guidance for resolving issues encountered when exposing metrics from Terraform Enterprise Flexible Deployment Options (FDO) to a Prometheus.
Users may observe that Prometheus discovers the Terraform Enterprise scrape target successfully, but expected tfe_* metrics do not appear in queries. In other cases, the metrics endpoint may return data when accessed directly, yet Prometheus does not ingest or store the metrics as expected.
This guide outlines common causes, validation steps, and configuration adjustments required to ensure Terraform Enterprise metrics are correctly exposed and scraped by Prometheus.
Validation and Troubleshooting Steps
Follow the steps below to validate that Terraform Enterprise metrics are properly exposed and accessible to Prometheus.
Step 1: Verify Metrics Are Enabled in Terraform Enterprise
Metrics are disabled by default and must be explicitly enabled.
Confirm the following configuration parameters are set:
TFE_METRICS_ENABLE=true
TFE_METRICS_HTTP_PORT=9090
TFE_METRICS_HTTPS_PORT=9091
After updating configuration values, restart Terraform Enterprise.
Validate Metrics Service Startup
Review Terraform Enterprise logs to confirm the metrics service has started successfully.
Look for log entries similar to:
"starting tfe-metrics"
"starting server"
metrics entered RUNNING state
The presence of these log entries confirms that the internal metrics service is active.
Validate the Metrics Endpoint
Test the metrics endpoint directly from a system that can reach the Terraform Enterprise instance.
HTTP endpoint:
curl http://<TFE_HOST>:9090/metrics?format=prometheus
HTTPS endpoint:
curl https://<TFE_HOST>:9091/metrics?format=prometheus
Expected result:
The response should return Prometheus-formatted metrics output, including entries similar to:
# HELP tfe_run_count
# TYPE tfe_run_count gauge
tfe_run_count{...}
If using HTTP, ensure port 9090 is open and reachable from the Prometheus server.
If using HTTPS, ensure port 9091 is open and valid TLS certificates are configured.
If the endpoint does not return metrics:
Confirm metrics are enabled
Confirm Terraform Enterprise has been restarted
Verify firewall or security group rules
Step 2: Verify Prometheus Scrape Configuration
If the target appears UP but tfe_* metrics are not visible in queries, Prometheus may not be requesting the correct format or may be scraping an incorrect endpoint.
Terraform Enterprise requires the format=prometheus query parameter.
Example prometheus.yml configuration:
scrape_configs:
- job_name: 'tfe'
metrics_path: /metrics
params:
format: ['prometheus']
static_configs:
- targets: ['<TFE-HOST-IP>:9090']
Step 3: Validate Prometheus Target Status
Open Prometheus UI:
http://<PROMETHEUS_HOST>:9090/targets
Confirm:
The
tfejob appearsTarget status is UP
The label
__param_format="prometheus"is present
If the status is:
DOWN → Review error message in target details
Timeout → Network connectivity issue between Prometheus and Terraform Enterprise
404 → Incorrect metrics path
403 → Network restriction, proxy, or firewall blocking access
Step 4: Verify Stored Metrics in Prometheus
To confirm whether Prometheus is storing Terraform Enterprise metrics, query the Prometheus API:
http://<PROMETHEUS_HOST>:9090/api/v1/label/__name__/values
The returned list should include Terraform Enterprise metric names such as:
tfe_run_counttfe_container_memory_used_bytestfe_container_cpu_usage_user_ns
If tfe_* metrics are not present:
Confirm the scrape configuration includes
format=prometheusConfirm the correct target port
Confirm the metrics endpoint returns valid output
Additional Information
https://developer.hashicorp.com/terraform/tutorials/enterprise/tfe-metrics
https://developer.hashicorp.com/terraform/enterprise/deploy/reference/metrics
https://developer.hashicorp.com/terraform/enterprise/deploy/manage/monitor/metrics