Telemetry for Vault Agent running on VM

Introduction:-

This article demonstrates the use of telemetry for vault agents and reading metrics from the agent endpoint. 

Prerequisites :-

This article requires the following prerequisites.

•    Vault version: 1.12.x +ent, 1.13.x +ent
•    Vault agent version: 1.12.x +ent, 1.13.x +ent

Setup: - 

  1) First set up the vault agent file with a telemetry stanza. In the template below, the vault agent is listening on port 8125 (which can be any port).

# Vault agent config file with telemetry stanza.
vault {
  address = "https://VAULT_ADDR:8200"
      }
auto_auth {
  method {
    type      = "approle"
    config = {
      role_id_file_path = "/root/roleid"
      secret_id_file_path = "/root/secretid"
      remove_secret_id_file_after_reading = false
             }
          }
  sink {
    type = "file"
    config = {
      path = "/root/sink_file_wrapped_1.txt"
             }
       }
     }
 listener "tcp" {
       address = "HOSTNAME_OF_AGENT_VM:8125"
       tls_disable = true
}
 telemetry {
  disable_hostname = true
  prometheus_retention_time = "10m"
}

2) Restart the vault service (vault agent) on the system.

3) There is a separate endpoint to read metrics related to vault-agent. “agent/v1/metrics”.

     Endpoint to read telemetry on vault agent.

curl --header "X-Vault-Token: <token with right privilege>" 'https://192.168.64.36:8125/agent/v1/metrics?format=prometheus'

Token Permission:-

# Token should have below mentioned policy to read metric. 
path "sys/metrics" {
capabilities = ["read"]
}

Expected Output:-

 Two metrics below show the count of success and failure.

root@:~# curl --header "X-Vault-Token: <token with right privilege>" 'https://HOSTNAME_OF_AGENT_VM:8125/agent/v1/metrics?format=prometheus' | grep -i agent
vault_agent_auth_success 2

root@:~# curl --header "X-Vault-Token: <token with right privilege>" 'https://HOSTNAME_OF_AGENT_VM:8125/agent/v1/metrics?format=prometheus' | grep -i agent
vault_agent_auth_failure 9