Vault nodes unable to join with error: invalid cross-device link

Introduction

Problem

While trying to join a Vault node to a cluster using Integrated Storage (aka RAFT) sync break with the following error message:

failed to restore snapshot 6664-94705722-1709295690252: 1 error occurred:...
| \t* failed to install snapshot database: rename <vault_mount>/raft/snapshots/6664-94705722-1709295690252/vault.db <vault_mount>/vault.db:
invalid cross-device link

Prerequisites

• All Vault Enterprise versions.
• The file system used to store data was changed into a nested file system.
• A new Vault node is unable to sync with issues occurring on renaming (moving) the snapshot to the vault.db resulting in an error message already show above.

This error message may be caused by the following:

• Clean the Vault RAFT storage for the node

• Mount the RAFT using a different filesystem (nested one)

• Wait for sync

• The sync will break upon trying to move the file.

Overview of possible solutions

Solution:

• Linux CLI tools like: findmnt or mount can be pursued on the Vault host as well as host machines were mounts may be originating from in order to obtain leads of the filesystem or devices requiring further administration.

• Additional OS administrative tasks are required to increase the storage filesystem used for Vault RAFT are required. While performing all the administrative activities on this Vault node, ensure that the Vault service is stopped and that there are no other Vault processes running.

• After removing the nested file system (*) and cleaning up the content of storage path the Vault node can get in sync and join the Vault cluster. 

(*) a nested file system is a file system mount point that is contained within another file system.
For example: /test /test/migfs1. The /test file system is a parent file system and /test/migfs1 is a nested file system within /test. They are both mount points.

Outcome

• Successful join a new node to the Vault Enterprise cluster.

Additional Information

• Vault Docs - Raft Snapshots

• Vault API - Raft Snapshot Auto
• Linux Filesystem Hierarchy - Common Software Directory