How-to identify which Vault node in the cluster is processing requests

Avatar
Rowan Smith
  • Updated

Introduction

When requests are routed to a Vault cluster via a load balancer it is not always immediately obvious which node in the cluster processed the request, however for multiple reasons it can be beneficial to have this information included in the HTTP response headers to help with any required investigation. We can make use of two additional configuration parameters to help surface this information, namely enable_response_header_hostname and enable_response_header_raft_node_id; these are outlined at https://www.vaultproject.io/docs/configuration#enable_response_header_hostname

Expected Outcome

After following this guide you will be able to inspect the headers included in responses from Vault in order to determine which node processed a request.

Prerequisites

  • Vault 1.7.0 or newer.
  • Write access to the Vault configuration file (vault.hcl).
  • Sufficient privileges on the host node(s) to restart the Vault service.
  • If not using an auto unseal mechanism you will need to manually unseal the Vault nodes after the service has been restarted, for which you will require the unseal keys.
  • To make use of the enable_response_header_raft_node_id configuration parameter the Vault cluster must be using Raft/Integrated Storage as the storage backend.

Use Case

Identifying the Vault node that processed a request can be useful for many reasons, not limited to:

  •  If you are performing load testing and see slower responses intermittently that you wish to investigate further.
  • If you are reading data from Vault that has only very recently been written and may not have replicated to all nodes in the cluster due to Vault's Eventual Consistency model.
  • Validating that your load balancer is routing requests to all nodes within the Vault cluster, or based on your configured pattern.

Procedure

  • Define the order to make changes to each nodes configuration and perform a restart of the Vault service; usually you would identify the follower / standby nodes and process those first, leaving the leader till last. Each of the steps below should be completed one node at a time, ensuring the Vault service restarts and the node rejoins the cluster before moving on to the next. vault operator raft list-peers and vault operator members are useful commands in helping to determine the leader and follower nodes.

  • Open the Vault configuration file (vault.hcl/vault.json) in your text editor of choice ensuring you utilise an account with write privileges to the file.

  • Place the following entries in the configuration file at the root level, i.e. on the same level as  ui / disable_mlock. The entries should not be included within any existing configuration stanza such as the ones for storage or listener :
    enable_response_header_hostname = "true" enable_response_header_raft_node_id = "true"

  • Note that you should only include the second configuration parameter if the cluster is using Raft/Integrated Storage as the storage backend.

  • Save the file and exit the text editor.
  • Perform a restart of the Vault service on the node.
  • Check the Vault service has entered an active / running state and rejoined the cluster. If the service fails to start the Vault operational logs can be queried for further diagnostic information with a command such as journalctl -u vault --no-pager
  • Repeat the process until all standby / follower nodes have been reconfigured and the Vault service restarted, then run the vault operator step-down command on the remaining (active / leader) node to have the cluster elect a new leader. Validate that a new leader has been elected before making the same changes outlined above and restarting the Vault service on the final node.
  • Generate a sample request to Vault in order to inspect the response headers and check for the presence of the x-vault-hostname and x-vault-raft-node-id values, for example:
➜  ~ curl -v https://vault.internal.nicecorp/ui/
Trying 192.168.50.22:8200...
Connected to vault.internal.nicecorp (192.168.50.22) port 8200 (#0)
.....
< x-vault-hostname: node3.internal.nicecorp
< x-vault-raft-node-id: node3
  • If needed, update any software such as load testing toolkits to save the HTTP response headers for future viewing and analysis.

Additional Information

Articles in this section

See more

Related articles