Vault Server Running Old Version After Upgrade Due to Incorrect Binary Path (Systemd Environments)

Introduction

Problem

After upgrading HashiCorp Vault, administrators may observe that:

• vault version correctly shows the new CLI version
• vault status still reports the old server version

This occurs on Linux systems where Vault is managed as a systemd service. Although the Vault CLI binary may be upgraded, systemd may still be launching the server using an older binary located in a different path.

This mismatch does not occur in Kubernetes deployments, as Kubernetes does not use systemd to start Vault.

Prerequisites

This issue applies to environments with:

• Vault Enterprise or OSS
• Linux hosts running Vault via systemd
• Manual upgrade procedures (binary replacement)
• Any storage backend (Integrated Storage/Raft, Consul, etc.)
• Any Vault version

Cause

This issue occurs when the Vault CLI binary and the Vault server binary originate from different file paths, typically due to:

• The CLI being upgraded in /usr/bin
• systemd still running the older binary from /usr/local/bin (or vice versa)

Key points:

• vault version shows the version of the CLI binary in your $PATH
• vault status shows the version of the Vault server binary running under systemd
• systemd always launches Vault using the path defined under ExecStart

Example mismatch:

CLI binary upgraded to:        /usr/bin/vault
systemd server binary path:    /usr/local/bin/vault

As long as systemd references the old path, Vault continues running the old server version regardless of the CLI upgrade.

Additional context

Reboots may worsen this issue if the upgraded binary was placed in a temporary directory, such as /tmp. Many Linux systems automatically clear temporary directories on startup (via systemd’s temporary files mechanism). If the upgraded Vault binary is deleted during reboot, systemd will revert to launching the older binary still present in the original ExecStart location, causing the version mismatch to reappear.

Overview of Solution

To resolve the version mismatch, install the upgraded Vault binary directly into the exact directory specified in the systemd ExecStart directive, then restart Vault and verify that both CLI and server versions match.

Solution

Install the upgraded Vault binary into the systemd‑configured directory

1. Identify the binary path used by systemd

   systemctl cat vault
    

   Look for the ExecStart line.
   
   Example:

   ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl

   This is the directory where the upgraded Vault server binary must be installed.

2. Install the upgraded Vault binary into that exact directory

   Use the following commands (safer version):

   sudo rm -f /usr/local/bin/vault
   
   curl -O https://releases.hashicorp.com/vault/1.21.2+ent/vault_1.21.2+ent_linux_amd64.zip
   
   sudo unzip vault_1.21.2+ent_linux_amd64.zip -d /usr/local/bin
   
   sudo chmod 755 /usr/local/bin/vault

   These commands ensure that:

   • The old server binary is removed
   • The new binary is downloaded from HashiCorp
   • The new binary is extracted into the correct systemd directory
   • Permissions are applied correctly so systemd can execute it

3. Restart the Vault service
   sudo systemctl start vault

4. Verify the upgrade

   Check CLI version:vault version

   Check server version:vault status

   Both should now report the same upgraded version (e.g., 1.21.2+ent).

Outcome

The issue is resolved when:

• vault version and vault status show the same version
• Vault successfully starts using the new binary
• Reboots no longer revert back to the old server binary

If the mismatch persists:

• Verify no old binaries exist in /usr/bin, /usr/local/bin, or /usr/sbin
• Check for systemd override files under /etc/systemd/system/
• Confirm systemd is loading the correct service file (systemctl cat vault)
• Review Vault server logs at startup to confirm which binary is executed

Additional Information

Vault Releases:
https://releases.hashicorp.com/vault

Official Vault Upgrade Guide:
https://developer.hashicorp.com/vault/docs/upgrade