Note: This article is an example reference on how you can enable file audit devices. There are other type of audit devices too which are not discussed here.
Introduction:
Audit Devices are the components in Vault that keep a detailed log of all requests and response to Vault. The audit logs contain the full request and response objects for every interaction with Vault. Vault will not respond to requests if audit devices are blocked because audit logs are critically important and ignoring blocked requests opens an avenue for attack.
Expected Outcome:
This article provide steps to enable file audit devices on vault enterprise setup, which might include Performance Standby Nodes and/or Replication Secondary clusters.
Prerequisites:
Though this article discusses about enabling Audit device for Vault Enterprise setup, we will discuss how to setup File audit devices only. Similar steps can be followed to enable other type (Syslog/Socket) of Audit devices too. We will cover two scenarios here.
Scenario 1: we assume vault enterprise is installed on a 3 node vault cluster in HA mode with one node is Active node and other 2 are performance Standby Nodes.
Scenario 2: vault enterprise is setup using replication:
- 3 Node vault cluster running Enterprise version setup as Primary Cluster.
- 3 node vault cluster running Enterprise version setup as Performance Secondary Cluster.
- 3 node vault cluster running Enterprise version setup as DR Secondary Cluster.
Theory and Concepts:
Audit Device - An audit device is responsible for storing/managing audit logs. Every request to Vault and response from Vault goes through the configured audit devices. This provides a simple way to integrate Vault with multiple audit logging destinations of different types.
Vault High Availability - In HA mode, Vault servers have two additional states they can be in: standby and active. For multiple Vault servers in HA mode, only a single instance will be active at any time. The active server operates in a standard fashion and processes all requests. The standby servers do not process requests, and instead redirect to the active Vault. Meanwhile, if the active server is sealed, fails, or loses network connectivity then one of the (unsealed) standbys will take over and become the active instance.
- In this setup (Using Vault OSS), only Active node is writing to Audit devices. As Standby nodes forwards are queries to Active node, they dont write to Audit device. It is still recommended to verify vault process has write permission to vault audit device location on all vault nodes, in case standby node becomes active in future.
- Audit logs are local to each node i.e. When a Standby Node becomes Active it will write audit logs locally to the location as configured in vault audit device.
Performance Standby Nodes (Vault Enterprise) - Performance Standby Nodes are just like traditional High Availability standby nodes but they can service read-only requests. If a request comes into a Performance Standby Node that causes a storage write the request will be forwarded onto the active server. If the request is read-only the request will be serviced locally on the Performance Standby.
- In this setup (Using Vault Enterprise), all nodes can write to their own audit devices because all nodes respond to client queries. It is recommended to verify vault process has write permission to vault audit device location on all vault nodes.
- Audit logs are local to each node, thus Performance Standby nodes have their own audit logs (in same locations as Active node in cluster) as these nodes respond to client readonly queries without going through Active node.
- Performance standby node can become Active node (in scenario where Active node goes down etc) and will start writing audit logs to its own audit device locations locally as configured for HA cluster.
- Audit logs from all nodes in vault HA cluster need to be analysed to audit any event. Thus it is best practice to use centralised logging tool.
Replication (Vault Enterprise) - Replication is based on a primary/secondary (1:N) model with asynchronous replication, focusing on high availability for global deployments. When replication is enabled, a cluster is set as either a primary or secondary. The primary cluster is authoritative, and is the only cluster allowed to perform actions that write to the underlying data storage, such as modifying policies or secrets. Secondary clusters can service all other operations, such as reading secrets or sending data through transit, and forward any writes to the primary cluster.
- In this setup, in addition to verifying audit device permission on Primary cluster, it should be checked on all Secondary cluster nodes too.
- There is an option to create local audit devices. If enabled these audit devices will not be replicated across clusters. Local audit devices are not replicated nor (if a secondary) removed by replication.
$ vault audit enable -local=true -path=audit1 file file_path=/var/log/vault_audit_1.log
Policy to admin and configure vault audit devices
- To list/modify vault audit devices, permission is required on sys/audit path, a policy like below should be associated with token:
$ vault policy read auditpol
# Allow a token to read audit devices
path "/sys/audit" {
capabilities = ["read", "sudo"]
}
# Allow a token to create/delete/update audit devices
path "/sys/audit/*" {
capabilities = ["update", "create", "delete", "sudo"]
}
- Token with above policy can be created as below. This policy could be assigned to user authenticating with other auth methods too. This token gives permissions to create audit device for vault cluster.
$ vault token create -policy=auditpol
Key Value
--- -----
token s.dep....
.
.
.
- A way to use this token is to export it as VAULT_TOKEN variable.
$ export VAULT_TOKEN=s.dep....
Procedure: Steps to enable/disable file audit devices on vault:
- Vault audit device can be enabled using command:
$ vault audit enable file file_path=/var/log/vault_audit.log
Success! Enabled the file audit device at: file/
$ vault audit list
Path Type Description
---- ---- -----------
file/ file n/a
- To enable another file audit device on same vault cluster:
$ vault audit enable -path="vault_audit_1" file file_path=/tmp/vault_audit.log
Success! Enabled the file audit device at: vault_audit_1/
$ vault audit list
Path Type Description
---- ---- -----------
file/ file n/a
vault_audit_1/ file n/a
- To check detailed information of audit devices:
$ vault audit list -detailed
Path Type Description Replication Options
---- ---- ----------- ----------- -------
file/ file n/a replicated file_path=/var/log/vault_audit.log
vault_audit_1/ file n/a replicated file_path=/tmp/vault_audit.log
- to delete audit device
#delete audit device with path vault_audit_1
$ vault audit disable vault_audit_1
Success! Disabled audit device (if it was enabled) at: vault_audit_1/
$ vault audit list -detailed
Path Type Description Replication Options
---- ---- ----------- ----------- -------
file/ file n/a replicated file_path=/var/log/vault_audit.log
Procedure: Steps to enable file audit devices on vault HA cluster:
- In case of file audit device, vault process (or user running vault process) should have permissions to write to the location of the file.
- In example below, vault process is running as vault user. User running vault command is vagrant.
$ id
uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)
$ ps -ef | grep vault
vault 5889 1 0 00:58 ? 00:00:24 /usr/local/bin/vault server -config=/etc/vault/vault.hcl
- If vault user does not have permission to write to file provided as file_path as per https://www.vaultproject.io/docs/audit/file#examples, vault will give * sanity check failed; unable to open "<file_path>" for writing: open <file_path>: permission denied error.
$ vault audit enable file file_path=/var/log/vault_audit.log
Error enabling audit device: Error making API request.
URL: PUT https://localhost:50520/v1/sys/audit/file
Code: 400. Errors:
* sanity check failed; unable to open "/var/log/vault_audit.log" for writing: open /var/log/vault_audit.log: permission denied
- if file_path is under a directory which does not exist, vault will give error * sanity check failed; unable to open "<file_path>" for writing: mkdir <file_directory_path>: permission denied:
$ vault audit enable -path=audit1 file file_path=/var/log/audit1/audit.log
Error enabling audit device: Error making API request.
URL: PUT https://localhost:50520/v1/sys/audit/audit1
Code: 400. Errors:
* sanity check failed; unable to open "/var/log/audit1/audit.log" for writing: mkdir /var/log/audit1: permission denied
- We must make sure vault has permission to write to the file if its already present or create file in the directory if file is not present already. These files and directories should be present with proper permission on all Vault nodes. As vault HA nodes in performance standby nodes can respond to client queries these nodes also create their own audit logs.
-
- in case audit file already present
# create audit file
$ sudo touch /var/log/vault_audit.log
# change ownership to vault user
$ sudo chown vault:vault /var/log/vault_audit.log
$ ls -ld /var/log/vault_audit.log
-rw-r--r-- 1 vault vault 0 Jun 7 03:27 /var/log/vault_audit.log
$ vault audit enable file file_path=/var/log/vault_audit.log
Success! Enabled the file audit device at: file/
2. In case audit file path is inside directory, but file is not created
# make sure directory is present and owned by vault user
$ ls -l /var/log/audit1
ls: cannot access '/var/log/audit1': No such file or directory
$ sudo mkdir /var/log/audit1
$ sudo chown -R vault:vault /var/log/audit1
# now vault audit device can be created with file path inside above directory
$ vault audit enable -path=audit1 file file_path=/var/log/audit1/audit.log
Success! Enabled the file audit device at: audit1/
$ vault audit list -detailed
Path Type Description Replication Options
---- ---- ----------- ----------- -------
audit1/ file n/a replicated file_path=/var/log/audit1/audit.log
file/ file n/a replicated file_path=/var/log/vault_audit.log
- to delete audit device
#delete audit device with path audit1
$ vault audit disable audit1
Success! Disabled audit device (if it was enabled) at: audit1/
$ vault audit list -detailed
Path Type Description Replication Options
---- ---- ----------- ----------- -------
file/ file n/a replicated file_path=/var/log/vault_audit.log
Procedure: Steps to enable file audit devices on vault HA cluster with replication enabled:
- To setup vault audit file device, follow the steps as above to ensure correct permissions on audit file/directory.
- In case of replication by default audit devices are configured as replicated. i.e. configuration (location of file etc) is replicated to secondary clusters. Thus we should setup correct permissions to location where audit device is going to write on all nodes in all clusters.
- This should be checked for both DR ad Perf Replication Clusters.
- Performance Replication cluster nodes will not have any active node untill one of these can write to audit device. vault logs will show error as below if perf replication cluster nodes cannot write to audit device.
2021-06-17T07:29:36.542Z [ERROR] core: post-unseal setup failed: error="failed to setup audit table"
2021-06-17T07:29:36.542Z [DEBUG] storage.raft: starting leadership transfer: id=<nil> address=<nil>`
- In case of DR cluster, it cannot be promoted to primary if nodes in this cluster cannot write to Audit device.
- Audit devices can also be configured as local, meaning these will be local to the cluster in a replication setup. Thus there is no requirement to setup permissions to audit devices(files/directories) on replicated cluster before setting up replication. Below command creates an audit device local to the cluster where it is run.
$ vault audit enable -local -path=audit1 file file_path=/tmp/audit.log
Success! Enabled the file audit device at: audit1/
$ vault audit list --detailed
Path Type Description Replication Options
---- ---- ----------- ----------- -------
audit1/ file n/a local file_path=/tmp/audit.log
file/ file n/a replicated file_path=/var/audit/audit.log
In above case, audit device audit1/ is local to the cluster and it is not replicated to secondary clusters.