How-to: Dynamically Adjusting Vault Log Levels in Server Memory via API

Avatar
Matthew Kentala
  • Updated

Introduction

HashiCorp Vault provides detailed operational logs that are vital for debugging authentication flows and internal cluster health. While these are typically defined in a configuration file, administrators can use the Vault API to modify log levels directly in the server's memory. This allows for immediate, high-verbosity troubleshooting without requiring a service restart or a permanent configuration change.

Expected Outcome

After following this guide, you will be able to:

  • Identify the current log level of the Vault core and its sub-systems as they exist in active memory.

  • Temporarily shift the server or specific sub-loggers into trace or debug mode.

  • Revert the server memory to standard levels once troubleshooting is complete.

Prerequisites

  • Vault Token: A token with read and update capabilities on the sys/loggers path.

  • Network Access: Connectivity to the Vault API port (8200).

  • Tools: curl and jq installed on your machine.

  • Environment Setup: Set your token as an environment variable to simplify the commands:

    export VAULT_TOKEN=<your_vault_token>

Use Case

  • On-the-fly Debugging: You need to see verbose logging immediately, but do not want to restart the cluster or leave trace mode on permanently.

  • Incident Response: Quickly quieting a "noisy" server that is filling up disk space with debug logs by pushing an info level change directly to memory.

Procedure

1. See the Server's Current Log-Level

To see all the individual log levels for every subsystem currently running in memory:

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    https://127.0.0.1:8200/v1/sys/loggers | jq -r '.data'

Understanding the data Output

The command above returns a map of sub-loggers. Key components include:

Sub-Logger Purpose
core The primary Vault engine; usually reflects the global log level.
mfa Detailed logs for Multi-Factor Authentication enforcement and validation.
storage.raft Integrated Storage consensus logs (heartbeats, snapshots, WAL).
expiration Logs related to lease management and token revocations.
audit Requests being sent to and processed by audit devices.
identity Entity and Group management within the Identity store.

If you only want to see the base level (the core engine), use this targeted command:

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    https://127.0.0.1:8200/v1/sys/loggers | jq -r '.data.core'

2. Update the Log-Level in Memory

To increase verbosity across the entire server to trace (the most detailed level) or debug:

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"level": "trace"}' \
    https://127.0.0.1:8200/v1/sys/loggers

3. Update a Specific Sub-Logger

If you only need to debug a specific component without flooding the logs with data from other subsystems, you can target a specific endpoint.

Instruction: To target a sub-logger, append the logger name (from the keys found in the data output in Step 1) to the end of the v1/sys/loggers/ path.

  • To target MFA: Use .../v1/sys/loggers/mfa

  • To target Raft Storage: Use .../v1/sys/loggers/storage.raft

Example Command (Targeting MFA):

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"level": "trace"}' \
    https://127.0.0.1:8200/v1/sys/loggers/mfa

 

4. Reverting to Standard Levels

Once troubleshooting is complete, you must return the server to a lower verbosity level to conserve resources. You have two options:

Option A: Force a specific level (e.g., info) This forces all in-memory loggers to info immediately.

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"level": "info"}' \
    https://127.0.0.1:8200/v1/sys/loggers

Option B: Reset to Configuration Defaults This removes all in-memory overrides and reverts every logger to the value defined in your vault.hcl file.

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request DELETE \
    https://127.0.0.1:8200/v1/sys/loggers

Verification

Monitor the effect of these changes in real-time by following the system logs. You should see the log tag change from [INFO] to [TRACE] immediately:

sudo journalctl -u vault -f

NOTE: Replication Log Behavior: In many Vault Enterprise environments, certain replication logs (specifically those regarding "dirty pages" or WAL flushing, e.g., replication.index.perf) may remain at a [DEBUG] level even when the global server memory is set to [INFO]. These specific sub-system logs are often hardcoded to maintain visibility into the cluster's data integrity and are not always impacted by global dynamic logger changes.

Additional Information

[!IMPORTANT]

Memory vs. Persistence: These changes affect the active process memory only. If the Vault service restarts or the server reboots, Vault will revert to the log_level defined in your physical vault.hcl configuration file.

  • Performance Impact: trace mode captures nearly every internal action, including hex dumps of network traffic. Only use this level for short periods of active troubleshooting.

     

Documentation & Resources

For more details on Vault logging and troubleshooting, refer to the official HashiCorp documentation:

Articles in this section

See more