The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.
All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.
Introduction
In the context of Consul Helm deployment, it is possible to enable the built-in demo Prometheus functionality, which provides users with the ability to investigate metrics via Prometheus and evaluate the performance of the Consul cluster. This feature proves particularly useful in troubleshooting scenarios where an issue may be difficult to diagnose, as users are able to analyze historical data to understand the issue's root cause.
By following these steps, you will have successfully enabled and configured the built-in Prometheus on a Consul cluster with TLS and ACL-enabled environment.
Prerequisites
- Familiar with deploying Consul cluster with Helm Chart or Consul-K8S.
- Have a running Consul cluster with with TLS and ACL-enabled.
- Have a basic understanding of using Prometheus.
Instructions
Table of Contents
Enabling both HTTP and HTTPS
Enabling Metrics for Prometheus
Prometheus needs an ACL token
Configure Prometheus with the ACL Token
Additional Resources
Enabling both HTTP and HTTPS
In the example provided, the Consul cluster is configured with TLS encryption and ACL enabled, as shown in the following Helm snippet:
global:
tls:
enabled: true
enableAutoEncrypt: true
httpsOnly: false
acls:
manageSystemACLs: true
In order to use the built-in Prometheus with a Consul cluster that has TLS encryption enabled, it is necessary to set the global.tls.httpsOnly parameter to false, as shown in line 4 of the Helm snippet above. This will enable the Consul server to expose both a secured HTTPS/8501 endpoint and an insecure HTTP/8500 endpoint.
To verify that both endpoints are available, you can use the netstat command.
$ netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::8300 :::* LISTEN 12/consul
tcp 0 0 :::8301 :::* LISTEN 12/consul
tcp 0 0 :::8302 :::* LISTEN 12/consul
tcp 0 0 :::8500 :::* LISTEN 12/consul
tcp 0 0 :::8501 :::* LISTEN 12/consul
tcp 0 0 :::8600 :::* LISTEN 12/consul
Instead of hitting the HTTPS endpoint, you can now verify the availability of the HTTP/8500 endpoint.
$ curl -k http://127.0.0.1:8500/v1/status/leader
The Prometheus can now be configured to use the HTTP/8500 endpoint for scraping metrics, since it does not currently support TLS configurations.
Enabling Metrics for Prometheus
To enable the built-in Prometheus in your Helm deployment, add the following stanza to your Helm config file. Once enabled, the Prometheus pod will be deployed in the same namespace as your Consul cluster.
prometheus:
enabled: true
To expose Prometheus metrics, add the following stanza to the Helm configuration file. If you want to enable additional metrics, such as gateway metrics, you can visit the Configure metrics for Consul on Kubernetes | Consul | HashiCorp Developer
global:
metrics:
enabled: true
enableAgentMetrics: true
After completing the additional Prometheus configurations, execute helm upgrade consul hashicorp/consul -n consul -f YOUR_VALUE_YAML command to apply the changes.
After enabling metrics for Prometheus, you can verify the output by checking the Prometheus endpoints. It is important to note that the format of the output will differ significantly from the original JSON format. This is because Prometheus uses its own query language, PromQL, to retrieve and process the metrics data.
$ curl -s \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
http://localhost:8500/v1/agent/metrics?format=prometheus
Prometheus needs an ACL token
The Prometheus pod should now be running in the same namespace as your Consul cluster. To access the Prometheus WebUI from your web browser, run the kubectl port-forward command.
$ kubectl -n consul port-forward svc/prometheus-server 8080:80
Forwarding from 127.0.0.1:8080 -> 9090
Forwarding from [::1]:8080 -> 9090
Handling connection for 8080
Now you can access Prometheus WebUI at http://127.0.0.1:8080 from your local web browser.
However, you may notice that none of the Consul-related metrics are being scraped. You can confirm this by checking the Status > Targets section in the Prometheus web UI. All targets under kubernetes-pods resources should be connected except for the pods with the Consul HTTP/8500 endpoint.
This is due to Prometheus not having ACL permission to access any resources in the Consul cluster. To resolve this issue, you can simply create an ACL token with proper permission and pass it to Prometheus server.
The next step is to create an ACL token for Prometheus to access Consul agents. This token will provide the minimal permissions for Prometheus to scrape Consul-related metrics.
Create a policy that all agents can be read
$ consul acl policy create -name read-agent-metrics -rules - << EOH
agent_prefix "" {
policy = "read"
}
EOH
Generate a token for Prometheus
$ consul acl token create \
-policy-name read-agent-metrics \
-description "allow prometheus to read agent metrics"
Save the generated ACL token as it will be used in the next step.
Configure Prometheus with the ACL Token
This section explains how to insert the ACL token into the Prometheus instance. To do this, you can edit the ConfigMap of the prometheus-server.
$ kubectl -n consul edit cm prometheus-server
First, search for the keyword kubernetes-pods which is one of the job_names in the ConfigMap. Then, add the previously generated ACL token as the value of the key bearer_token as shown in line 2 of the code snippet below.
...
- job_name: kubernetes-pods
bearer_token: '< the_generated_ACL_token >'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
...
Once you've made these changes, save and exit. It will take about a minute for the changes to take effect once you reload the Prometheus web UI. You should now see that the targets of Consul HTTP/8000 under kubernetes-pods are successfully connected as it will shows Green.
In the search bar of the Prometheus web UI, you can now start typing any keyword related to Consul, and it will auto-complete with all available metrics related to that keyword. This feature makes it easier to explore and analyze Consul metrics using Prometheus.