All Vault traffic, reads and writes, forwarded to active node only

 

Introduction

Problem

In performance testing on a Vault cluster disproportionate CPU load is noted in Active vs Performance Standby nodes.

Upon further investigation, all traffic is being forwarded to active node only.

Nodes do not appear to be functioning as performance standby nodes as described in: https://developer.hashicorp.com/vault/docs/enterprise/performance-standby

 

Cause

• Your load balancer is not configured to forward traffic to performance standby nodes
  • e.g. health check is configured as:

     /v1/sys/health?standbyok=true

• Your license is not Enterprise Premium and therefore does not include performance standby functionality as evidenced by the following output from reading the license.  Note the bolded indicative text ""performance_standby_count": 0,"

Symptoms

Upon running a read of the licence you see the following output:

$ vault read sys/license/status -format=json

{

"request_id": "eb50590e-775b-a698-8c7a-591ef280196e",

"lease_id": "",

"lease_duration": 0,

"renewable": false,

"data": {

"autoloaded": {

"expiration_time": "2026-01-29T00:00:00Z",

"features": [

"HSM",

"DR Replication",

"MFA",

"Sentinel",

"Seal Wrapping",

"Control Groups",

"Namespaces",

"Entropy Augmentation",

"Lease Count Quotas",

"Automated Snapshots"

],

"license_id": "6872465d-23fc-c644-99a1-f9cd5c4b26eb",

"performance_standby_count": 0,

"start_time": "2022-12-08T00:00:00Z",

"termination_time": "2036-01-28T00:00:00Z"

},

"autoloading_used": true,

"persisted_autoload": {

"expiration_time": "2026-01-29T00:00:00Z",

"features": [

"HSM",

"DR Replication",

"MFA",

"Sentinel",

"Seal Wrapping",

"Control Groups",

"Namespaces",

"Entropy Augmentation",

"Lease Count Quotas",

"Automated Snapshots"

],

"license_id": "6872465d-23fc-c644-99a1-f9cd5c4b26eb",

"performance_standby_count": 0,

"start_time": "2022-12-08T00:00:00Z",

"termination_time": "2036-01-28T00:00:00Z"

}

},

"warnings": [

"Endpoint ignored these unrecognized parameters: [-format]"

]

}

 

Overview of possible solutions (if applicable)

Solutions:

• Your load balancer is not configured to forward traffic to performance standby nodes:

  • health check setting only has /v1/sys/health?standbyok=true
  • performance standbys return 473.
  • Add 473 to success codes or change the health check to include something like perfstandby=true the traffic should be distributed as expected:
    e.g. 
    

    /v1/sys/health?standbyok=true&perfstandbyok=true&drsecondarycode=200&performancestandbycode=200

• You have an Enterprise License but not an Enterprise Premium license for Vault.

Outcome

Upon reconfiguration of the load balancer or application of the Enterprise premium license, read requests are also serviced by the performance standby nodes.

 

Additional Information

• Note: this article is meant to address two specific configuration issues.  Additional considerations and causes are discussed here: https://developer.hashicorp.com/vault/docs/concepts/ha