How to Upgrade Plugins on Performance Secondary Clusters

Avatar
Clayton Platt
  • Updated

Introduction

     There are some nuances with the plugin system with regards to upgrade procedures when it comes to performance secondary clusters. This article aims to provide some clarity by expanding upon our existing  HashiCorp plugin upgrade docs referenced in the prerequisites section below.

     Below are two procedures on upgrading plugins in a HA cluster with perf replication enabled.  The first outlines the method available to Vault versions 1.12to 1.15, and second highlights the advantage of using the sys/plugins/pins endpoint. This endpoint was made available in version 1.16+.  Both procedures will demonstrate upgrading the built-in JWT plugin to version 0.20.2as an example.  This example was performed on an Amazon Linux 3 EC2 Instance.

 

Expected Outcome

     Understand that the new sys/plugins/pins endpoint introduced in Vault v1.16 allows for replication of the performance primary's plugin version across to the performance secondary clusters auth and secret mounts without any user action needed on the performance secondary.  Also, to understand how this differs from the older method for Vault versions 12-1.15 where manually tuning and reloading the mount for each cluster was necessary. 

 

Use case

     This article does not apply to any new mounts created after the plugin upgrade procedure is executed on the performance primary. New mounts will be reflect the upgraded version on performance secondary clusters provided the updated plugin binary exists on the performance secondary cluster plugin directory for each node (see Plugin Installation below).

          

Prerequisites 

Upgrade Procedures

  • All procedures use the upgrade of the built-in JWT auth plugin v0.20.1 -> v0.20.2 as an example
  • Plugin Installation -- must be done first, before either procedure, for all nodes on each cluster
  • Vault >= v1.16.0 -- sys/plugins/pins endpoint automates this reload/mount tune process
  • Vault v1.12-v1.15 -- must manually tune the mount and reload plugin on secondary and active

Plugin Installation

Performance Primary, all nodes; Performance Secondaries, all nodes:

Ensure the plugin directory exists as a top-level parameter in Vault's config.hcl: 

# confirming plugin dir in my own config file, ensure dir has adequate perms
sudo grep plugin_directory /etc/vault.d/vault.hcl

plugin_directory = "/mnt/vault/plugins"

 

Install the most recent JWT plugin from official HashiCorp release page into your plugin directory:

cd /mnt/vault/plugins
sudo wget https://releases.hashicorp.com/vault-plugin-auth-jwt/0.20.2/vault-plugin-auth-jwt_0.20.2_linux_amd64.zip
sudo unzip vault-plugin-auth-jwt_0.20.2_linux_amd64.zip
sudo rm vault-plugin-auth-jwt_0.20.2_linux_amd64.zip
# rename binary to match old version plugin name; confirm via 'vault plugin list'
# may need to give necessary ownership/perms via chown/chmod
sudo mv vault-plugin-auth-jwt jwt
  • Note: Renaming the plugin is not required.

Vault Versions >= 1.16.0

Performance Primary:   

Ensure plugin installation procedure above is done for each node

List current plugin version for JWT (the plugin I will be using for this example), and for its mount:

# use "secret" for secrets plugins
vault plugins list auth

Name   Version
----   -------
...
jwt    v0.20.1+builtin

# skip if already enabled
vault auth enable jwt

# confirm running version in output matches above
vault auth list -detailed

# truncated response
Plugin   Running Version
------   ---------------
jwt      0.20.1+builtin

 

Register plugin under same name as existing, but with new version, and confirm both in registry:

vault plugin register \
-sha256=$(sudo sha256sum /mnt/vault/plugins/jwt | awk '{print $1}') \
-version=v0.20.2 \
auth \
jwt

# can confirm two versions now exist in registry 
vault plugin list auth

Name   Version
----   -------
...
jwt   v0.20.1+builtin
jwt   v0.20.2

Note: If the plugin was not renamed you can still register the plugin by specifying the command flag. 

vault plugin register \
-sha256=$(sudo sha256sum /mnt/vault/plugins/jwt | awk '{print $1}') \
-command=vault-plugin-auth-jwt \
-version=v0.20.2 \
auth \
jwt

Pin the version to signal all performance secondary clusters to auto-update their mounts, then reload plugins globally

# all performance secondary clusters will now update their mounts to reflect new version
vault write sys/plugins/pins/auth/jwt version=v0.20.2

# global reload applies for all nodes; no further user-action required on secondaries
vault plugin reload -type=auth -plugin=jwt -scope=global

 

Vault Versions 1.12-1.16

Performance Primary:   

Ensure plugin installation procedure above is done for each node

List current plugin version for JWT (the plugin I will be using for this example), and for its mount:

# use "secret" for secrets plugins
vault plugins list auth

Name   Version
----   -------
...
jwt    v0.20.1+builtin

# skip if already enabled
vault auth enable jwt

# confirm running version in output matches above
vault auth list -detailed

# truncated
Plugin   Running Version
------   ---------------
jwt      0.20.1+builtin

 

Register plugin under same name as existing, but with new version, and confirm both in registry:

vault plugin register \
-sha256=$(sudo sha256sum /mnt/vault/plugins/jwt | awk '{print $1}') \
-version=v0.20.2 \
auth \
jwt

# can confirm two versions now exist in registry 
vault plugin list auth

Name   Version
----   -------
...
jwt   v0.20.1+builtin
jwt   v0.20.2

 

Tune plugin mount, reload globally, and confirm new mount version (note this only applies to primary):

vault auth tune -plugin-version=v0.20.2 jwt

# setting scope=global means each server in cluster will apply change
vault plugin reload -type=auth -plugin=jwt -scope=global

vault auth list -detailed

# truncated
Plugin   Running Version
------   ---------------
jwt      0.20.2

 

For Each Performance Secondary:

Ensure plugin installation procedure above is done for each node

Note: mount version does not match primary's as it is still on pre-upgrade version

# can also confirm 'vault plugin list' shows both versions in registry
vault auth list -detailed

# truncated
Plugin   Running Version
------   ---------------
jwt      0.20.1+builtin

 

Tune plugin mount, reload globally, and confirm new mount version:

vault auth tune -plugin-version=v0.20.2 jwt

# setting scope=global means each server in cluster will apply change
vault plugin reload -type=auth -plugin=jwt -scope=global

vault auth list -detailed

# truncated
Plugin   Running Version
------   ---------------
jwt      0.20.2

Articles in this section

See more

Related articles