How-to receive streaming logs from the HCP Vault server

Introduction

This article will provide instructions for using the /sys/monitor endpoint to stream logs back to the client from HCP Vault. This method can be helpful for troubleshooting OIDC/JWT, Kubernetes, and SAML auth method issues. 

Problem

Customers using managed HCP Vault clusters do not have access to the underlying server logs for troubleshooting purposes. 

Prerequisites

• Access to the admin namespace. 
• A token with permissions to access the /sys/monitor endpoint. 
• For OIDC/JWT auth method, ensure the Vault role is configured with verbose logging enabled (verbose_oidc_logging=true). 
• For SAML auth method, ensure the Vault config has verbose logging enabled (verbose_logging=true)

Procedure

• Authenticate to the admin namespace. Ensure the token has permissions to access the /sys/monitor endpoint (admin token can be used).
• Run vault monitor -log-level=trace
• In a separate session, send a new login request to Vault.
• Look for the auth request being streamed back to the client. Note - log events for permission denied errors usually start with login unauthorized:

Note - if Vault is emitting log messages faster than a receiver can process them, then some log lines will be dropped. If the auth attempt cannot be observed using this endpoint, reach out to Support for further assistance. 

Additional Information

For additional questions or support, please open a Support ticket.