This article is a quick start guide with tips on using the Command Line Interface (CLI) tool jq to perform brief analysis and selections on Vault audit log files.
We recommend using the most recent versions of jq (1.7 or higher) to extend these examples in your own aliases or scripts.
You can find the download and installation instructions for jq for all operating systems (Linux, macOS, Windows, etc.) on the official website.
The examples in this guide refer to the environment variable VAFILE, denoting the path of a Vault audit log file. This environment variable is also appended VAFILE1, VAFILE2, etc. to handle multiple files.
$ VAFILE=/path/to/audit.log;
$ jq . $VAFILE
$ VAFILE1=/path/to/audit.log; VAFILE2=/path/to/audit.log;
$ jq . $VAFILE1 $VAFILE2
The CLI options passed to jq vary. Commonly used arguments in this guide are noted below.
- -R
- --arg
- -s
- -c
- -r
- -n
Extracting Audits & Ranges
If the audit logs are combined together with Vault operational logs or other system level events, they can be separated into different files using jq.
$ VLOG=/path/to/combined_file.log
$ VAFILE=/path/to/audit.log
$ cat $VLOG | sed -n '/{"time":/,$p' | jq . > $VAFILE
Once the audit entries have been separated into a new file, the new VAFILE can be parsed using jq.
To narrow the time span of audits entries to a particular range, you can define the variables for the start date SD and end date ED in ISO 8601 format with the time-separator T and ending Z characters.
$ SD=2024-09-25T18:28:45Z
$ ED=2024-09-25T18:30:25Z
$ jq --arg SD $SD --arg ED $ED -nc '[inputs|select(.time|sub("\\..*Z";"Z")|([.,$SD,$ED|fromdate]|sort)[1]==fromdate)]' $VAFILE > audit_filtered.log
The provided event times in the Vault operational logs include fractional milliseconds which are not compatible with the date functions provided by jq such as fromdate. Therefore a concatenation is performed in the sub portion of the query above when dealing with dates throughout this guide.
Rates, Frequency & Totals
The total number of request entries can be used to measure activity for the entire time span of the audit logs.
The total number of requests can be determined using request as a counter.
$ jq -s 'map(select(.type=="request"))|length' $VAFILE
A brief summary of the audit log span can be extrapolated using the start and end dates.
$ jq -s '(del(.[]|select(.type!="request"))|length) as $L|sort_by(.time)|(.[-1].time|sub("\\..*Z";"Z")|fromdate) as $d1|(.[0].time|sub("\\..*Z";"Z")|fromdate) as $d2|($d1-$d2) as $d3|{time_start: .[0].time, time_ended: .[-1].time, span_seconds: $d3, span_minutes: ($d3/60), span_hours: ($d3/3600), span_days: ($d3/93600), requests: $L, qps_average: ($L/$d3)}' $VAFILE
You can use the example below to calculate the rate of queries per second (QPS) using the logged times of both response & request fields that match their request_id.
$ jq -sr 'map(select(.type=="request").time|sub("\\..*Z";"Z"))|group_by(.)[]|"\(first): \(length)"' $VAFILE
# QPS low to high sorted by seconds (grouped)
$ jq -sr 'map(select(.type=="request").time|sub("\\..*Z";"Z"))|group_by(.)|sort_by(length)[]|"\(first): \(length)"' $VAFILE
The following total and frequency examples use the optional CLI sed & column tools to format tabular summaries of interest.
# Summary of types
$ jq -sr 'map(select(.type=="request")|{type: .request.operation})|group_by(.type)|sort_by(length)[]|"\(first): \(length)"' $VAFILE|sed 's/\}:/\}+/'|column -ts'+'
# Total number of token types used
$ jq -sr 'map(select(.type=="request")|{token_type: .auth.token_type, type: .request.operation})|group_by(.token_type)| sort_by(length)[]| "\(first): \(length)"' $VAFILE | sed 's/,/,+/g'|sed 's/\}:/\}+/'|column -ts'+'
# Hottest paths and type
$ jq -sr 'map(select(.type=="request")|{mtype: .request.mount_type, path: .request.path})|group_by(.path)|sort_by(length)[]|"\(first): \(length)"' $VAFILE|sed 's/,/,+/g'|sed 's/\}:/\}+/'|column -ts'+'
# Hottest operation types by path
$ jq -sr 'map(select(.type=="request")|{mtype: .request.mount_type, path: .request.path, type: .request.operation})|group_by(.path)|sort_by(length)[]|"\(first): \(length)"' $VAFILE|sed 's/,/,+/g'|sed 's/\}:/\}+/'|column -ts'+'
# Busiest clients, request paths by client display name (CDN)
$ jq -sr 'map(select(.type=="request")|{path: .request.path, cdn: .auth.display_name})|group_by(.cdn)|sort_by(length)[]|"\(first): \(length)"' $VAFILE|sed 's/,/,+/g'|sed 's/\}:/\}+/'|column -ts'+'
# Specific paths, for example '/ui/' & 'XYZ' and where the request is coming from
$ jq -sr 'map(select(.type=="request")|select(.request.path|contains("/ui/","XYZ"))|{cdn: .auth.display_name, path: .request.path})|group_by(.path)|sort_by(length)[]|"\(first): \(length)"' $VAFILE|sed 's/,/,+/g'|sed 's/\}:/\}+/'|column -ts'+'
Time Calculations, Error Responses & Other Selectors
Calculate the estimated duration of a request that was responded to.
# Duration of requests (response + request pairs with matching ID), low to high
$ jq -sc '[group_by(.request.id)|map(select(length==2))|.[]|((.[1].time|sub("\\..*Z";"Z")|fromdate)-(.[0].time|sub("\\..*Z";"Z")|fromdate)) as $delta|{duration: $delta, rid: .[0].request.id, time: .[0].time}]|sort_by(.duration)[]' $VAFILE
# Get average response times as well as minimum and maximum duration for all request and response pairs
$ jq -s '[group_by(.request.id)|map(select(length==2))| .[] | ((.[1].time|sub("\\..*Z"; "Z")|fromdate) - (.[0].time|sub("\\..*Z"; "Z")|fromdate)) as $delta|{duration: $delta}]|sort_by(.duration)|length as $L|.[0].duration as $d1|.[-1].duration as $d2|([.[].duration]|add) as $d3|[select(.[].duration==0)]|length as $L0|{rtime_fastest: $d1, rtime_slowest: $d2, rtime_average: ((($L0*20)+$d3)/$L)}' $VAFILE
# List only responses containing an error and the authorized users that invoked the request
$ jq -s 'map(select(.type=="response")|select(.error!=null)|{error: .error, path: .request.path, cdn: .auth.display_name})' $VAFILE
# Scope request & response pair ranges for a specific path, i.e.pki/roles/example9
$ jq -sr '[group_by(.request.id)|map(select(length==2))|.[]|select(.[1].request.path|contains("pki/roles/example9"))]|.[]' $VAFILE
Resources
- Vault tutorials: Query Audit Device Logs
- jq official docs: jq Manual
- Article: Reshaping JSON with jq
- Vault Support KB: Audit Device Notes