Audit and Operational Log Details

Avatar
Brian Shumate
  • Updated

This guide describes the formats of Vault audit and operational log output and details their contents. It is an introductory resource and deep dive through the log outputs with examples, but it is not a comprehensive reference to all possible emitted log messages.

Vault Audit Log Details

Vault CLI with Token Example

The following are some example audit log entries which demonstrates the request and response logging generated when a user interacts with Vault CLI.

Earlier a token was generated with some specific policies and metadata:

$ vault token create \
-policy=sudo \
-policy=surf-admin \
-metadata=loglevel=raw \
-metadata=remote=false \
-metadata=surf=moderate
Key                    Value
---                    -----
token                  0150e834-20cf-4eab-4196-b6f1f4d70ce8
token_accessor         3786dfc3-29e2-3874-8053-60204896d0a0
token_duration         50000h
token_renewable        true
token_policies         [default sudo surf-admin]
token_meta_surf        moderate
token_meta_loglevel    raw
token_meta_remote      false

Now a user is going to login to Vault with the token:

$ vault login 0150e834-20cf-4eab-4196-b6f1f4d70ce8

Request Log

The above login produces this output in the Vault audit log as the audit log request object:

{
  "time": "2018-04-09T21:04:29.6406536Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:eb3da855a3fb8b1c3574064f7edd080a97c4ebcf4a8e6674126710915fe464ae",
    "accessor": "hmac-sha256:f5a8798113bb65c0676abb3eef5ca5482c0c7daac38da36d402282a5414fcf3d",
    "display_name": "token",
    "policies": [
      "default",
      "sudo",
      "surf-admin"
    ],
    "metadata": {
      "loglevel": "raw",
      "remote": "false",
      "surf": "moderate"
    },
    "entity_id": ""
  },
  "request": {
    "id": "b2f72168-6cba-1bab-808a-72d9304b82f8",
    "operation": "read",
    "client_token": "hmac-sha256:eb3da855a3fb8b1c3574064f7edd080a97c4ebcf4a8e6674126710915fe464ae",
    "client_token_accessor": "hmac-sha256:f5a8798113bb65c0676abb3eef5ca5482c0c7daac38da36d402282a5414fcf3d",
    "path": "auth/token/lookup-self",
    "data": null,
    "policy_override": false,
    "remote_address": "172.17.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "error": ""
}

The audit log request object fields are described as follows:

  • time: RFC3339 timestamp for the request
  • type: Log entry type; there are currently just two types, request and response and in this case it is request
  • auth: Authentication details, including:
    • client_token: This is an HMAC of the client’s token ID that can be compared as described in the /sys/audit-hash API documentation
    • accessor: This is an HMAC of the client token accessor that can be compared as described in the /sys/audit-hash API documentation
    • display_name: This is the display name set by the auth method role or explicitly at secret creation time; this is often useful for determining from which auth method mountpoint or user this request could be related to
    • policies: This will contain a list of policies associated with the client_token
    • metadata: This will contain a list of metadata key/value pairs associated with the client_token
    • entity_id: This is a token entity identifier; some tokens (such as AppRole) can have this attached.
  • request: This is the request object, containing the following:
    • id: This is the unique request identifier
    • operation: This is the type of operation which corresponds to path capabilities and is expected to be one of: create, read, update, delete, or list
    • client_token: This is an HMAC of the client’s token ID that can be compared as described in the /sys/audit-hash API documentation
    • client_token_accessor: This is an HMAC of the client token accessor that can be compared as described in the /sys/audit-hash API documentation
    • path: The requested Vault path for operation
    • data: The data object will contain secret data in key/value pairs, for example:
    • policy_override: This is a boolean, and true when a soft-mandatory policy override was requested
    • remote_address: The IP address of the client making the request
    • wrap_ttl: If the token is wrapped, this displays configured wrapped TTL value as numeric string (e.g., ‘5s’ or ‘10m’)
    • headers: Additional HTTP headers specified by the client as part of the request
  • error: If an error occurred with the request, the error message is included in this field’s value

Audit Response Log

The response to this login is:

{
  "time": "2018-04-09T21:04:29.6420203Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:eb3da855a3fb8b1c3574064f7edd080a97c4ebcf4a8e6674126710915fe464ae",
    "accessor": "hmac-sha256:f5a8798113bb65c0676abb3eef5ca5482c0c7daac38da36d402282a5414fcf3d",
    "display_name": "token",
    "policies": [
      "default",
      "sudo",
      "surf-admin"
    ],
    "metadata": {
      "loglevel": "raw",
      "remote": "false",
      "surf": "moderate"
    },
    "entity_id": ""
  },
  "request": {
    "id": "b2f72168-6cba-1bab-808a-72d9304b82f8",
    "operation": "read",
    "client_token": "hmac-sha256:eb3da855a3fb8b1c3574064f7edd080a97c4ebcf4a8e6674126710915fe464ae",
    "client_token_accessor": "hmac-sha256:f5a8798113bb65c0676abb3eef5ca5482c0c7daac38da36d402282a5414fcf3d",
    "path": "auth/token/lookup-self",
    "data": null,
    "policy_override": false,
    "remote_address": "172.17.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "response": {
    "data": {
      "accessor": "hmac-sha256:f5a8798113bb65c0676abb3eef5ca5482c0c7daac38da36d402282a5414fcf3d",
      "creation_time": 1523307682,
      "creation_ttl": 180000000,
      "display_name": "hmac-sha256:e38035c165f0076d9288ba0363eb36733379cc5d370bec5e82f11632519c26a8",
      "entity_id": "hmac-sha256:2fced7e2c77266f5079d733bea71dc8c8413d3838584ca9d0f4867271df7a220",
      "expire_time": "2023-12-23T05:01:22.8929692Z",
      "explicit_max_ttl": 0,
      "id": "hmac-sha256:eb3da855a3fb8b1c3574064f7edd080a97c4ebcf4a8e6674126710915fe464ae",
      "issue_time": "2018-04-09T21:01:22.8929624Z",
      "meta": {
        "loglevel": "hmac-sha256:eac4a7deb2df94609ab14ae48b9edea81d91de51be1dd59df6ca6852537227c5",
        "remote": "hmac-sha256:aa2d1dd64d4468bbd9c6b0ca275cdffb7473a2d91b5f42a047161620245fcc79",
        "surf": "hmac-sha256:8b29af9294da23c72de8d8d847ccebd450d978af5565807d0c9922b6b2e92988"
      },
      "num_uses": 0,
      "orphan": false,
      "path": "hmac-sha256:36ea987a227a2c7aefe055a98f99751383f601955e9f1925bd3c2d6f9931a025",
      "policies": [
        "hmac-sha256:451623ebbe12fb9b1b3f444ceb5a5a46102452e46d640925c7b0dcb93a65a99a",
        "hmac-sha256:9a76c609b073848f2d9cb4a7fcddfc2103c0063480b87a9ee585e9e072e901d9",
        "hmac-sha256:8924f876eca967c68bbc8ac138e9f876f2144e300c08b1898224fc76902c1fe3"
      ],
      "renewable": true,
      "ttl": 179999812
    }
  },
  "error": ""
}

The response objects contain many of the same fields found in the request object and so those are not covered here (please see details the request section above for any unclear fields). There are some additional token specific fields which can be expected in response output depending on the operations and auth methods in question; those fields are detailed here.

  • creation_time RFC3339 format timestamp of the token’s creation
  • creation_ttl: Token creation TTL in seconds
  • expire_time: RFC3339 format timestamp representing the moment this token will expire
  • explicit_max_ttl: Explicit token maximum TTL value as seconds (‘0’ when not set)
  • issue_time: RFC3339 format timestamp
  • num_uses: If the token is limited to a number of uses, that value will be represented here
  • orphan: Boolean value representing whether the token is an orphan
  • renewable: Boolean value representing whether the token is renewable
  • ttl: 179999812

BONUS

At the beinginning of this guide, there was mention of creating the token used in the audit log request and response log examples. If you are curious about the audit logs of the actual token creation request, well here they are!

Request:

{
  "time": "2018-04-09T21:01:59.2157659Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:cea805e955310099f47fcd89af2043458afef82d4598c91baef18d87b7f98223",
    "accessor": "hmac-sha256:c79f127c2745ddc6490812f17476350e52bdd9e5e4f36d40e61a998a093a70cb",
    "display_name": "token",
    "policies": [
      "default",
      "sudo"
    ],
    "metadata": {
      "loglevel": "raw",
      "remote": "false",
      "surf": "moderate"
    },
    "entity_id": ""
  },
  "request": {
    "id": "e2a3bb3e-9e0c-0e8d-b276-096d30724bd2",
    "operation": "update",
    "client_token": "hmac-sha256:cea805e955310099f47fcd89af2043458afef82d4598c91baef18d87b7f98223",
    "client_token_accessor": "hmac-sha256:c79f127c2745ddc6490812f17476350e52bdd9e5e4f36d40e61a998a093a70cb",
    "path": "auth/token/create",
    "data": {
      "display_name": "hmac-sha256:2fced7e2c77266f5079d733bea71dc8c8413d3838584ca9d0f4867271df7a220",
      "explicit_max_ttl": "hmac-sha256:f332ef7005e24f2fb2fd11b492775b2589305606fa4406369e23137dec6208ad",
      "meta": {
        "loglevel": "hmac-sha256:eac4a7deb2df94609ab14ae48b9edea81d91de51be1dd59df6ca6852537227c5",
        "remote": "hmac-sha256:aa2d1dd64d4468bbd9c6b0ca275cdffb7473a2d91b5f42a047161620245fcc79",
        "surf": "hmac-sha256:8b29af9294da23c72de8d8d847ccebd450d978af5565807d0c9922b6b2e92988"
      },
      "num_uses": "hmac-sha256:7736474b452ed98a91775ebfe8d8008ea7a9d39e2d34ae13f0a00264a61e2465",
      "period": "hmac-sha256:f332ef7005e24f2fb2fd11b492775b2589305606fa4406369e23137dec6208ad",
      "policies": [
        "hmac-sha256:9a76c609b073848f2d9cb4a7fcddfc2103c0063480b87a9ee585e9e072e901d9",
        "hmac-sha256:8924f876eca967c68bbc8ac138e9f876f2144e300c08b1898224fc76902c1fe3"
      ],
      "renewable": true,
      "ttl": "hmac-sha256:f332ef7005e24f2fb2fd11b492775b2589305606fa4406369e23137dec6208ad"
    },
    "policy_override": false,
    "remote_address": "172.17.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "error": ""
}

and the response:

{
  "time": "2018-04-09T21:01:59.2754117Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:cea805e955310099f47fcd89af2043458afef82d4598c91baef18d87b7f98223",
    "accessor": "hmac-sha256:c79f127c2745ddc6490812f17476350e52bdd9e5e4f36d40e61a998a093a70cb",
    "display_name": "token",
    "policies": [
      "default",
      "sudo"
    ],
    "metadata": {
      "loglevel": "raw",
      "remote": "false",
      "surf": "moderate"
    },
    "entity_id": ""
  },
  "request": {
    "id": "e2a3bb3e-9e0c-0e8d-b276-096d30724bd2",
    "operation": "update",
    "client_token": "hmac-sha256:cea805e955310099f47fcd89af2043458afef82d4598c91baef18d87b7f98223",
    "client_token_accessor": "hmac-sha256:c79f127c2745ddc6490812f17476350e52bdd9e5e4f36d40e61a998a093a70cb",
    "path": "auth/token/create",
    "data": {
      "display_name": "hmac-sha256:2fced7e2c77266f5079d733bea71dc8c8413d3838584ca9d0f4867271df7a220",
      "explicit_max_ttl": "hmac-sha256:f332ef7005e24f2fb2fd11b492775b2589305606fa4406369e23137dec6208ad",
      "meta": {
        "loglevel": "hmac-sha256:eac4a7deb2df94609ab14ae48b9edea81d91de51be1dd59df6ca6852537227c5",
        "remote": "hmac-sha256:aa2d1dd64d4468bbd9c6b0ca275cdffb7473a2d91b5f42a047161620245fcc79",
        "surf": "hmac-sha256:8b29af9294da23c72de8d8d847ccebd450d978af5565807d0c9922b6b2e92988"
      },
      "num_uses": "hmac-sha256:7736474b452ed98a91775ebfe8d8008ea7a9d39e2d34ae13f0a00264a61e2465",
      "period": "hmac-sha256:f332ef7005e24f2fb2fd11b492775b2589305606fa4406369e23137dec6208ad",
      "policies": [
        "hmac-sha256:9a76c609b073848f2d9cb4a7fcddfc2103c0063480b87a9ee585e9e072e901d9",
        "hmac-sha256:8924f876eca967c68bbc8ac138e9f876f2144e300c08b1898224fc76902c1fe3"
      ],
      "renewable": true,
      "ttl": "hmac-sha256:f332ef7005e24f2fb2fd11b492775b2589305606fa4406369e23137dec6208ad"
    },
    "policy_override": false,
    "remote_address": "172.17.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "response": {
    "auth": {
      "client_token": "hmac-sha256:dd00f904baf7a8726346dda0472485cd73fe298238d49c8659b667ec0325a1e7",
      "accessor": "hmac-sha256:7289c034f4283e712e18ea2c1e084a9ebdfdccd40965df40ffc52e938b654f8c",
      "display_name": "token",
      "policies": [
        "default",
        "sudo",
        "surf-admin"
      ],
      "metadata": {
        "loglevel": "raw",
        "remote": "false",
        "surf": "moderate"
      },
      "entity_id": ""
    }
  },
  "error": ""
}

Vault Operational Log Details

The operational log is derived from an internal logging package such as the go-hclog package, and output is in a single line format which follows a format similar to many popular server tools.

Example Vault operational log entries:

2018-04-09T20:26:38.606Z [INFO ] core: security barrier not initialized
2018-04-09T20:26:38.626Z [INFO ] core: security barrier initialized: shares=5 threshold=3

The log entries are whitespace separated, and detailed as follows:

  • Timestamp: 2018-04-09T20:26:38.626Z
  • Level: [INFO ]
  • Component: core:
  • Message: security barrier not initialized

These log message fields are further described as follows:

  • Timestamp: RFC3339 timestamp for the log entry
  • Level: Configurable logging levels for Vault’s operational logs, in order from most to least verbose:
    • trace: provide extreme detail from all Vault components including storage backends, auth methods, secrets engines and Enterprise features, such as HSM interaction and replication
    • debug Lower level messages from Vault components which are helpful for QA/test/staging environments, and troubleshooting but generally too verbose for production
    • info typical production level logging of nominal system information messages
    • warn a warning message signifies a problem that does not necessarily impact production operations, but should be further examined and resolved - warnings should be alerted on in monitoring solutions as a heads up to operators.
    • err an error typically signifies conditions which impact operation of Vault and should be immediately investigated to resolution - errors should be alerted on in monitoring solutions as a heads up to operators.
  • Component: The Vault component that is the source of the log message — consists of a range of possible values:
    • audit: Messages related to audit device functionality
    • core: Messages related to Vault core functionality
    • expiration: Messages related to Expiration Manager functionality
    • identity: Messages related to Identity Manager functionality
    • rollback: Messages related to Rollback Manager functionality
    • secrets.TYPE.ID: Messages related to secrets engines of TYPE with the identity of ID
    • storage.cache: Messages related to the storage cache
    • storage.TYPE: Messages related to the storage backend of type TYPE (i.e. storage.consul)
  • Message: Log message body

Note that each Vault component can emit specific log detail and this guide does not attempt to provide an exhaustive reference of all log messages. Generally speaking and for recent Vault versions, logged events can be found throughout the Vault source code with these strings:

  • logger.Trace
  • logger.Debug
  • logger.Info
  • logger.Warn
  • logger.Error

Resources

  1. Audit Devices
  2. vault/audit/format.go
  3. /sys/audit-hash
  4. Capabilities
  5. go-hclog package
  6. -log-level

Articles in this section

See more

Related articles