Introduction
This article is to highlight the top causes of permissions issues encountered by customers when enabling audit logging in Vault. This error will typically look like this:
* sanity check failed; unable to open "/var/log/vault_audit.log" for writing: open /var/log/vault_audit.log: permission denied
Or this:
* 1 error occurred:
* permission denied
Simple "Permission Denied" Errors
The second error above, which simply reads "Permission Denied", is usually the result of insufficient user privileges attached to your Vault policy.
You can look this up with the following command:
vault token lookup
And then verify the policies listed, with:
vault policy read <policy-name>
Your user will require the following configuration in order to be able to read, write and update the audit log:
# Allow a token to read audit devices
path "/sys/audit" {
capabilities = ["read", "sudo"]
}
# Allow a token to create/delete/update audit devices
path "/sys/audit/*" {
capabilities = ["update", "create", "delete", "sudo"]
}If you are unable to look up your policies due to a similar permission denied error, you will need to log in as a user who has sufficient privileges to read Vault policy information.
File Permissions Issues
Check which user and group is running Vault:
~$ ps aux | grep vault
vault 941 0.6 24.0 210858064 240256 ? Ssl Oct21 436:06 /usr/local/bin/vault server -config /etc/vault.d -log-level=trace
Ensure that the file in question has both the correct owner and group, and that they have adequate permissions to read and write to that file:
~$ sudo touch /var/log/vault_audit.log
~$ sudo chown vault:vault /var/log/vault_audit.log
~$ sudo chmod 600 /var/log/vault_audit.log
~$ ls -la /var/log/vault_audit.log
-rw------- 1 vault vault 148500 Dec 6 02:54 /var/log/vault_audit.log
SELinux is Stopping Vault Writing to File
You can verify whether SELinux is running on your machine with the sestatus command:
~$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
If it is running and in an enforcing mode, modify /etc/selinux/config (or equivalent directory) and change enforcing to permissive:
~$ vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of these two values:
# default - equivalent to the old strict and targeted policies
# mls - Multi-Level Security (for military and educational use)
# src - Custom policy built from source
SELINUXTYPE=default
A reboot is required for this change to take effect.
Note that disabling SELinux could open your Vault instance up to potential security flaws - it is recommended you configure SELinux to allow Vault read/write permissions to your desired directory, rather than simply disabling it, in production.
AppArmor Restrictions
Aside from SELinux, AppArmor is a popular way to harden Linux and provide mandatory access controls to processes running on the machine. You can check the allowed paths with the apparmor_status command.
An example of an AppArmor configuration that may stop Vault writing to /var/log would be:
/usr/bin/vault {
#include <abstractions/base>
network inet stream,
network netlink raw,
..
/etc/vault/* mr,
/etc/vault/config.hcl mr,
/etc/vault/raft/raft/raft.db mrwk,
/etc/vault/raft/raft/snapshots/ r,
/etc/vault/raft/vault.db mrwk,
/etc/vault/ssl/DigiCertCA.crt r,
owner /etc/vault/ssl/* rw,
owner /var/log/vault/* rw,
owner /etc/vault/vault_audit.log mrw,
owner /home/*/.vault-token rw,
owner /home/*/.vault-token.tmp rw,
}
The above is edited to display a smaller amount of allowed directories. It allows read and write permissions to /var/log/vault and its subsequent directories, but any attempt from Vault to write to /var/log will fail. You would need to add the following line:
/var/log/* rw,
In order for this to succeed. However this permissions set may be considered excessive in some environments.
Note that disabling AppArmor could open your Vault instance up to potential security flaws - it is recommended you configure AppArmor to allow Vault read/write permissions to your desired directory, rather than simply disabling it, in production.
Third Party Access Control Applications
Always verify whether you have any third party application control services running which may interfere with Vault writing to specific directories.
In the following, for example example:
~$ sadmin status
McAfee Solidifier: Enabled
McAfee Solidifier on reboot: Enabled
ePO Managed: Yes
Local CLI access: Lockdown
[fstype] [status] [driver status] [volume]
* xfs Solidified Attached /
ext4 Solidified Attached /boot
xfs Solidified Attached /home
xfs Solidified Attached /var
xfs Solidified Attached /var/log
xfs Unsolidified Attached /tmp
xfs Solidified Attached /var/log/audit
We see that McAfee Application Control is using its McAfee Solidifier service to block access to all filesystems marked as Solidified. In this case, Vault would be able to write to any directory not listed, or marked as Unsolidified - such as /etc/vault.d, or /tmp.
Permissions Issues due to LogRotate Misconfiguration
When configuring LogRotate or equivalent services, it's important to make sure that when rotating the audit log, the newly created replacement file is owned by the correct group and user.
In most Linux distributions, the location of the logrotate file is /etc/logrotate.conf. A correct configuration for Vault should look similar to this:
/var/log/vault_audit.log {
...
create 600 <user> <group>
...
}