How-to update journald configuration to persist Vault operational logs

Introduction

When Vault is used on a distribution of Linux utilizing systemd, the Vault operational logs by default are written to journald. The default journald setting for log storage is auto and as a result, Vault operational logs are lost after a node is restarted. Using this guide, you can update the journald configuration and define log retention settings based on criteria such as time to retain logs or maximum allotment of space on disk to consume.

Expected Outcome

Once completed and a Vault node is restarted, the Vault operational logs should still be present.

Prerequisites

• As with any system level change, we recommend testing the process in a development or pre-production environment before making changes to production.
• Permissions required to update the /etc/systemd/journald.conf file.
• Permissions required to create the /var/log/journal directory.

Permissions are normally owned by the root user, meaning su or sudo access will typically be the requirement to satisfy.

Use Case

As the Vault operational logs are usually required in order to troubleshoot any potential issues with Vault, it is helpful to ensure logs are accessible after a node restart.

Procedure

• Consider the appropriate strategy for log storage retention. Should retention settings be based on a pre-defined disk space allocation, time based value, or both?
• Consider starting with small values and adjusting based on the rate of log generation observed. After having an adequate sample size, make informed adjustments as necessary.

• Validate on a node running Vault that you are able to view the operational logs from the recent past. Value of date included for demonstration purposes only:

  [user@node1]~% date
  Mon May 30 15:46:45 AEST 2022
  [user@node1]~% sudo journalctl -u vault --since=yesterday | head
  -- Logs begin at Thu 2022-05-19 02:22:17 AEST, end at Mon 2022-05-30 15:46:47 AEST. --
  May 29 00:00:01 node1 vault[1302]: 2022-05-29T00:00:01.745+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  May 29 00:00:06 node1 vault[1302]: 2022-05-29T00:00:06.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  May 29 00:00:11 node1 vault[1302]: 2022-05-29T00:00:11.747+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  May 29 00:00:16 node1 vault[1302]: 2022-05-29T00:00:16.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
  

• [Optional] Restart the Vault node and repeat the same step as above to confirm logs are no longer available for viewing.

• Verify the /var/log/journal directory exists and if not, create it. Ensure it is owned and writable by the root user.

• Open the file /etc/systemd/journald.conf in your text editor of choice.
• Change the value for Storagefrom auto to Persistent.
• Change the value for SystemMaxUse to a value of your choosing, i.e 512M or 1G.

• Change the value for MaxRetentionSec to a value of your choosing, i.e 1d or 1month or 1year.

• Save the file and exit the text editor.
• Confirm that log files are being written to the folder /var/log/journal.
• Restart the node and repeat the second step to validate logs are still visible from the previous day.

Additional Information

• Journald configuration reference: https://www.freedesktop.org/software/systemd/man/journald.conf.html