Introduction
Response code 307 is returned when accessing the Vault standby node while hitting "/sys/metrics" endpoint.
Error example
Trying ip...
* TCP_NODELAY set
* Connected to vault-url (ip) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* ALPN, server accepted to use h2
* Server certificate:
* subjectAltName: host "vault_url" matched cert's "vault_url"
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle--)
> Host: vault_url> user-agent: curl/7.68.0
> accept: */*
* TLSv1.3 (IN), TLS handshake:
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 307
< cache-control: no-store
< content-length: 0
Understanding Behaviour
When a user is using authenticated metrics in their Vault cluster and performance standby is either disabled or not included as part of the Vault license then the only node that can respond to sys/metrics request would be the active node of the cluster. Hence, the standby nodes cannot serve authenticated read requests which is an expected behavior. For all the requests hitting the load balancer IP/DNS from standby nodes will end up in a redirection loop.
Recommended Action
- Users will need to enable unauthenticated metrics as true in their vault config.
- Alternatively, the Users can request for performance standby feature in their Vault license which will allow them to use authenticated metrics and all standby nodes acting as performance standbys will also be able to respond to "sys/metrics" API calls.
- To check if your license include the performance standby feature or not you can either use the below curl command or can run vault license get command.
curl -v -H "X-Vault-Request: true" -H "X-Vault-Token: $(vault print token)" https://node_with_307:8200/v1/sys/health
{"initialized":true,"sealed":false,"standby":true,"performance_standby":false,"replication_performance_mode":"disabled","replication_dr_mode":"primary","server_time_utc":1708922305,"version":"1.15.4+ent.hsm","cluster_name":"vault-cluster-df39855a","cluster_id":"aa7c2370-8b56-118f-16ee-dcd153fcf13b","license":{"state":"autoloaded","expiry_time":"2024-10-31T00:00:00Z","terminated":false}}
Reference
https://developer.hashicorp.com/vault/docs/configuration/listener/tcp#unauthenticated_metrics_acc