How-to update journald configuration to persist Vault operational logs

Introduction

When Vault is used on a distribution of Linux utilising systemd the Vault operational logs are written to journald, however the default journald setting for log storage is `auto`, as a result Vault operational logs are lost after a node is restarted. Using this guide you can update the journald configuration and define log retention settings based on criteria such as time to retain logs or maximum allotment of space on disk to consume.

Expected Outcome

Once completed and a Vault node is restarted the Vault operational logs should still be present.

Prerequisites

• As with any system level change we recommend testing the process in a development or pre-production environment before making changes to production.
• Any permissions required to update the /etc/systemd/journald.conf file.
• Any permissions required to create the /var/log/journal directory.
• Both of these are usually owned by the root user, meaning su or sudo access will typically be the requirement to satisfy.

Use Case

As the Vault operational logs are usually required in order to troubleshoot potential issues if the steps in this guide are implemented then the Vault operational logs will be available for viewing and analysis even if a node has been restarted.

Procedure

• Consider the appropriate strategy for log storage retention - should retention settings be based on a pre-defined disk space allocation, a time based value such as one month, or both? Consider starting with small values and adjusting based on the rate of log generation observed after a one week period and making informed adjustments if necessary or unsure.

• Validate on a node running Vault that you are able to view the operational logs from yesterday. Value of date included for demonstration purposes only.

  [user@node1]~% date
  Mon May 30 15:46:45 AEST 2022
  [user@node1]~% sudo journalctl -u vault --since=yesterday | head
  -- Logs begin at Thu 2022-05-19 02:22:17 AEST, end at Mon 2022-05-30 15:46:47 AEST. --
  May 29 00:00:01 node1 vault[1302]: 2022-05-29T00:00:01.745+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  May 29 00:00:06 node1 vault[1302]: 2022-05-29T00:00:06.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  May 29 00:00:11 node1 vault[1302]: 2022-05-29T00:00:11.747+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  May 29 00:00:16 node1 vault[1302]: 2022-05-29T00:00:16.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  

• [Optional] Restart the Vault node and repeat the same step as above to confirm logs are no longer available for viewing.

• If it does not already exist, create the folder /var/log/journaland ensure it is owned and writeable by the root user.

• Open the file /etc/systemd/journald.conf in your text editor of choice.
• Change the value for Storagefrom auto to Persistent
• Change the value for SystemMaxUse to a value of your choosing, i.e 512M or 1G

• Change the value for MaxRetentionSec to a value of your choosing, i.e 1d or 1month or 1year

• Save the file and exit the text editor.
• Confirm that log files are being written to the folder /var/log/journal.
• Restart the node and repeat the second step to validate logs are still visible from the previous day.

Additional Information

• Journald configuration reference: https://www.freedesktop.org/software/systemd/man/journald.conf.html