Vault Service Fails to Start on Linux Due to Read-Only Filesystem – HashiCorp Help Center

Introduction

Problem

Attempting to initialize the Vault service using systemd on a linux operation system. After starting the service is enters a failed state. The associated error log for the failed process is something along the lines of:

Error initializing storage of type raft: failed to create fsm: 
failed to open bolt file: open /etc/vault/data1/vault.db: read-only file system

The actual file path will be whatever you have defined as the data directory in your Vault configuration file. https://developer.hashicorp.com/vault/docs/configuration/storage/raft#path

Prerequisites (if applicable)

Vault
Systemd to manage the Vault Service
Any Linux OS
Integrated Storage for the Vault backend

Cause

Data directory in /etc/ with Systemd ProtectSystem setting set to full:
In the example above, we see that the data directory is set to save data to a file located within the /etc directory. If your Vault is configured in the same way, check your systemd configuration file for the parameter ProtectSystem. If the parameter is set to 'strict' or 'full' consider changing this setting to a more lenient setting such as 'true'. If that's not feasible, consider moving your data directory to a different file location that is mounted on a device with read write access for the process.

From the documentation for ProtectSystem
"If set to "full", the /etc/ directory is mounted read-only, too. If set to "strict" the entire file system hierarchy is mounted read-only,"
https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#ProtectSystem=
Location of the data directory in the filesystem is improperly mounted:
If the data is not located in /etc and/or the ProtectSystem parameter is not set to strict or full, run the following command passing in the data directory's path as the argument. This will display the filesystem on which the path is mounted:

~ df /opt/vault/data
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 8310764 2215348 6095416 27% /

We can now use the findmnt utility to verify whether the filesystem is in a read-only mode. This will be indicated by the text "ro" displayed in the options column of the output

~ findmnt --list | grep -i /dev/xvda1
/ /dev/xvda1 xfs 
ro,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota

The specific steps for remounting a drive as read-write are outside the scope of this ticket and will be highly specific to your environment. Discuss with your team the best approach. Both the mount command and editing the /etc/fstab and rebooting Linux are both viable options.

Outcome

Vault starts successfully!

Additional Information