How-to identify which Vault node in the cluster is processing requests

Avatar
Rowan Smith
  • Updated

Introduction

When requests are routed from a load balancer to a Vault cluster, it is not always obvious which node in the cluster processes the request. It is possible to have this information included in the HTTP response headers to help with any required investigation. We can make use of two additional configuration parameters to help surface this information:

  • enable_response_header_hostname
  • enable_response_header_raft_node_id 

This article explains how to add these parameters to then have the ability to inspect headers, included in responses from Vault, in order to determine which node processed the request.

Prerequisites

  • Vault 1.7.0+
  • Access to edit the Vault configuration file
  • Sufficient privileges to restart the Vault service
  • If not using an auto unseal mechanism, ability to unseal Vault after a restart
  • Note the enable_response_header_raft_node_id parameter are for clusters using Integrated Storage as a storage backend

Use Case

Identifying the Vault node that processed a request can be useful for many reasons, not limited to:

  • Performing load testing and see slower responses intermittently that you wish to investigate further.
  • Reading data from Vault that has only very recently been written and may not have replicated to all nodes in the cluster due to Vault's Eventual Consistency model.
  • Validating the load balancer is routing requests to all nodes within the Vault cluster, or based on your configured pattern.

Procedure

Determine the order of operations to make changes to each node's configuration and perform a restart of the Vault service; identify the standby / follower nodes and process those first, leaving the leader till last. Each of the steps below should be completed one node at a time, ensuring the Vault service restarts and the node rejoins the cluster before moving on to the next. vault operator raft list-peers and vault operator members are useful commands in helping to determine the leader and follower nodes.

  1. Access and edit the Vault configuration file with a write privileges on the file.

  2. Place the following entries in the configuration file at the root level:
    enable_response_header_hostname = "true" enable_response_header_raft_node_id = "true"
    These should be added to the same level as  ui / disable_mlock. These should not be included within any existing configuration stanza, such as storage or listener.

  3. Save changes to the Vault configuration.
  4. Perform a restart of the Vault service.
  5. Check the Vault service has entered a running state and rejoined the cluster. If the service fails to start, the Vault operational logs can be queried for further diagnostic information with a command such as journalctl -u vault --no-pager
  6. If the previous node starts back up as expected, repeat the process until all standby nodes have been reconfigured, the Vault service restarted, and the node has rejoined the cluster.
  7. Run the vault operator step-down command on the remaining (active / leader) node to have the cluster elect a new leader. Validate that a new leader has been elected before making the same changes outlined above and restarting the Vault service on the final node. 
  8. Generate a sample request to Vault. In order to inspect the response headers and check for the presence of the x-vault-hostname and x-vault-raft-node-id values, for example:
➜  ~ curl -v https://vault.internal.nicecorp/ui/
Trying 192.168.50.22:8200...
Connected to vault.internal.nicecorp (192.168.50.22) port 8200 (#0)
.....
< x-vault-hostname: node3.internal.nicecorp
< x-vault-raft-node-id: node3

If needed, update any software such as load testing toolkits to save the HTTP response headers for future viewing and analysis.

Additional Information

Articles in this section

See more

Related articles