How to Capture Logs from Vault Agent running as a Service on Windows – HashiCorp Help Center

Introduction

Expected Outcome

Logging will be enabled for Vault Agent running as a service in Windows.

Prerequisites (if applicable)

This KB assumes you have at least a working Dev environment on Windows Server from which to conduct testing.

Use Case

Normally once a program is run as a service in Windows, output to stdout and stderr is no longer visible and cannot be captured simply by passing a logging location to the Windows Service. This appears to be due to the inability of directing stderr/stdout to a file path as Windows services are not designed to return output interactively as in a command shell. Windows services direct output to named pipes.

Procedure

The following script is an optional way to setup the Dev environment.

If you are setting up the environment manually this script is not needed. Also the script assumes you have the Dev instance of Vault server started:

e.g. 'vault server -dev -dev-root-token-id=root'

#Start Vault Dev Server and Configure for Vault Agent

# Assumes this is started: vault server -dev -dev-root-token-id=root

$env:VAULT_ADDR="http://127.0.0.1:8200"
$env:VAULT_TOKEN="root"

vault auth enable approle
vault write auth/approle/role/vault-agent-role secret_id_ttl=90m token_num_uses=10 token_ttl=60m token_max_ttl=120m secret_id_num_uses=20
vault read auth/approle/role/vault-agent-role/role-id -format=json | jq -r '.data.role_id' | Out-File -encoding ascii C:\vault-agent\agent-role-id -NoNewline
vault write -f auth/approle/role/vault-agent-role/secret-id -format=json | jq -r '.data.secret_id' | Out-File -encoding ascii C:\vault-agent\agent-secret-id -NoNewline
cat c:\vault-agent\agent-role-id ; cat c:\vault-agent\agent-secret-id
$AgentConfiguration = @"
pid_file = "/vault-agent/agent.pid"

vault {
  address = "http://127.0.0.1:8200"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/vault-agent/agent-role-id"
      secret_id_file_path = "/vault-agent/agent-secret-id"
    }
  }

  sink "file" {
    config = {
      path = "/vault-agent/agent-token"
    }
  }

}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address     = "127.0.0.1:8100"
  tls_disable = true
}
"@
Set-Content C:\vault-agent\vault-agent.hcl $AgentConfiguration

Setting up the Vault Agent as a Service and Capturing Logging using nssm

Prerequisite: nssm installed https://nssm.cc/usage

Setting up Vault Agent using nssm Service helper through a Powershell script:

# nssm test
# this script tests installing the Vault Agent Service on Windows and redirecting stderr to stdout.log

# Normally you would set path to exe here but vault.exe is already in $PATH in this environment
nssm install VaultAgent vault.exe

#similarly I did not find the need to set this here:
#nssm set VaultAgent AppDirectory C:\path\to\vault.exe

# Here passing arguments and configuration to vault.exe to run as agent with debug level logging and redirecting stderr/stdout
nssm.exe set VaultAgent AppExit Default Restart
nssm set VaultAgent AppParameters "agent -config=C:\vault-agent\vault-agent.hcl -log-level=debug"

#As Vault Agent appears to only log to stderr, stdout here is superfluous
# nssm set VaultAgent AppStdout c:\temp\stderr.log

nssm set VaultAgent AppStderr c:\temp\stderr.log
nssm set VaultAgent Description "VaultAgent"
nssm set VaultAgent Start SERVICE_AUTO_START
nssm set VaultAgent Type SERVICE_INTERACTIVE_PROCESS
Start-Service VaultAgent
Get-Service VaultAgent

Which yields the following:

Get-Service VaultAgent

Status   Name               DisplayName                           
------   ----               -----------                           
Running  VaultAgent         VaultAgent

All stderr is redirected to stderr.log:

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

           Api Address 1: http://127.0.0.1:8100
                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.8.4
             Version Sha: 925bc650ad1d997e84fbb832f302a6bfe0105bbb

2021-10-25T23:28:24.850Z [INFO]  sink.file: creating file sink
2021-10-25T23:28:24.896Z [INFO]  sink.file: file sink configured: path=/vault-agent/agent-token mode=-rw-r-----
2021-10-25T23:28:24.898Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink
2021-10-25T23:28:24.898Z [INFO]  template.server: starting template server
2021-10-25T23:28:24.898Z [INFO]  template.server: no templates found
2021-10-25T23:28:24.898Z [INFO]  auth.handler: starting auth handler
2021-10-25T23:28:24.898Z [INFO]  auth.handler: authenticating
2021-10-25T23:28:24.899Z [INFO]  sink.server: starting sink server
2021-10-25T23:28:24.907Z [INFO]  auth.handler: authentication successful, sending token to sinks
2021-10-25T23:28:24.908Z [INFO]  auth.handler: starting renewal process
2021-10-25T23:28:24.914Z [INFO]  sink.file: token written: path=/vault-agent/agent-token
2021-10-25T23:28:24.914Z [DEBUG] cache.leasecache: storing auto-auth token into the cache
2021-10-25T23:28:24.927Z [INFO]  auth.handler: renewed auth token

Some Conclusions:
- Vault Agent is writing output through stderr only.

- It is possible to redirect stderr in a Windows Service using a script, c# or other program or in this case a service helper such as nssm: https://nssm.cc/usage

- nssm proved to be the most elegant solution tested.

Additional Information

Removing the service methods:

Registry
Requires reboot or reload of explorer.exe
Get-Item "HKLM:\SYSTEM\CurrentControlSet\Services\VaultAgent" | Remove-Item -Force -Verbose
on Powershell version lower than 6.0, this is the way to remove the service.

Powershell 6.0+ has a remove service function.
Remove-Service -Name "TestService"

Easiest way to remove service if using nssm
stop the VaultAgent Service then run:
nssm remove VaultAgent

Additional Tests:

Manual Run from cmd line:
vault.exe agent -config=C:\vault-agent\vault-agent.hcl 2>&1 > c:\temp\VaultAgentLog2.log
Works as expected. Incidentally this could also be setup as a scheduled task if desired.

Register with sc.exe
sc.exe create VaultAgent binPath= "C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -config=C:\vault-agent\vault-agent.hcl 2>&1 > c:\temp\stdout.log" displayName= "Vault Agent" start= auto
Successfully registers as a service with redirection but stderr/stdout NOT captured to file

Ref: https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent/winsvc

Testing with Powershell calling exe with args
Also tested registering the Vault Agent running in a Powershell script but ran into the issue that the Windows Service then no longer knows if the process has crashed within the script so this did not appear to be a viable solution. It may be possible that a more elaborate script would have caught this.

References: