How to Capture Logs from Vault Agent running as a Service on Windows

Avatar
Robert Highley
  • Updated

Introduction

Expected Outcome

Logging will be enabled for Vault Agent running as a service in Windows.

 

Prerequisites (if applicable)

Use Case

Normally once a program is run as a service in Windows, output to stdout and stderr is no longer visible and cannot be captured simply by passing a logging location to the Windows Service.  This appears to be due to the inability of directing stderr/stdout to a file path as Windows services are not designed to return output interactively as in a command shell.  Windows services direct output to named pipes.

 

Procedure

The following script is an optional way to setup the Dev environment as described here:

https://learn.hashicorp.com/tutorials/vault/agent-windows-service?in=vault/app-integration

If you are setting up the environment manually this script is not needed.  Also the script assumes you have the Dev instance of Vault server started:

e.g. 'vault server -dev -dev-root-token-id=root'

 

#Start Vault Dev Server and Configure for Vault Agent

# Assumes this is started: vault server -dev -dev-root-token-id=root
#This script follows tutorial on https://learn.hashicorp.com/tutorials/vault/agent-windows-service?in=vault/app-integration
$env:VAULT_ADDR="http://127.0.0.1:8200"
$env:VAULT_TOKEN="root"

vault auth enable approle
vault write auth/approle/role/vault-agent-role secret_id_ttl=90m token_num_uses=10 token_ttl=60m token_max_ttl=120m secret_id_num_uses=20
vault read auth/approle/role/vault-agent-role/role-id -format=json | jq -r '.data.role_id' | Out-File -encoding ascii C:\vault-agent\agent-role-id -NoNewline
vault write -f auth/approle/role/vault-agent-role/secret-id -format=json | jq -r '.data.secret_id' | Out-File -encoding ascii C:\vault-agent\agent-secret-id -NoNewline
cat c:\vault-agent\agent-role-id ; cat c:\vault-agent\agent-secret-id
$AgentConfiguration = @"
pid_file = "/vault-agent/agent.pid"

vault {
  address = "http://127.0.0.1:8200"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/vault-agent/agent-role-id"
      secret_id_file_path = "/vault-agent/agent-secret-id"
    }
  }

  sink "file" {
    config = {
      path = "/vault-agent/agent-token"
    }
  }

}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address     = "127.0.0.1:8100"
  tls_disable = true
}
"@
Set-Content C:\vault-agent\vault-agent.hcl $AgentConfiguration

 

Setting up the Vault Agent as a Service and Capturing Logging using nssm

Setting up Vault Agent using nssm Service helper through a Powershell script:

# nssm test
# this script tests installing the Vault Agent Service on Windows and redirecting stderr to stdout.log

# Normally you would set path to exe here but vault.exe is already in $PATH in this environment
nssm install VaultAgent vault.exe

#similarly I did not find the need to set this here:
#nssm set VaultAgent AppDirectory C:\path\to\vault.exe

# Here passing arguments and configuration to vault.exe to run as agent with debug level logging and redirecting stderr/stdout
nssm.exe set VaultAgent AppExit Default Restart
nssm set VaultAgent AppParameters "agent -config=C:\vault-agent\vault-agent.hcl -log-level=debug"

#As Vault Agent appears to only log to stderr, stdout here is superfluous
# nssm set VaultAgent AppStdout c:\temp\stderr.log

nssm set VaultAgent AppStderr c:\temp\stderr.log
nssm set VaultAgent Description "VaultAgent"
nssm set VaultAgent Start SERVICE_AUTO_START
nssm set VaultAgent Type SERVICE_INTERACTIVE_PROCESS
Start-Service VaultAgent
Get-Service VaultAgent

Which yields the following:

Get-Service VaultAgent

Status   Name               DisplayName                           
------   ----               -----------                           
Running  VaultAgent         VaultAgent

All stderr is redirected to stderr.log:

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

           Api Address 1: http://127.0.0.1:8100
                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.8.4
             Version Sha: 925bc650ad1d997e84fbb832f302a6bfe0105bbb

2021-10-25T23:28:24.850Z [INFO]  sink.file: creating file sink
2021-10-25T23:28:24.896Z [INFO]  sink.file: file sink configured: path=/vault-agent/agent-token mode=-rw-r-----
2021-10-25T23:28:24.898Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink
2021-10-25T23:28:24.898Z [INFO]  template.server: starting template server
2021-10-25T23:28:24.898Z [INFO]  template.server: no templates found
2021-10-25T23:28:24.898Z [INFO]  auth.handler: starting auth handler
2021-10-25T23:28:24.898Z [INFO]  auth.handler: authenticating
2021-10-25T23:28:24.899Z [INFO]  sink.server: starting sink server
2021-10-25T23:28:24.907Z [INFO]  auth.handler: authentication successful, sending token to sinks
2021-10-25T23:28:24.908Z [INFO]  auth.handler: starting renewal process
2021-10-25T23:28:24.914Z [INFO]  sink.file: token written: path=/vault-agent/agent-token
2021-10-25T23:28:24.914Z [DEBUG] cache.leasecache: storing auto-auth token into the cache
2021-10-25T23:28:24.927Z [INFO]  auth.handler: renewed auth token

 

Some Conclusions:
- Vault Agent is writing output through stderr only.

- It is possible to redirect stderr in a Windows Service using a script, c# or other program or in this case a service helper such as nssm: https://nssm.cc/usage

- nssm proved to be the most elegant solution tested.

 

Additional Information

Removing the service methods:

Registry
Requires reboot or reload of explorer.exe
Get-Item "HKLM:\SYSTEM\CurrentControlSet\Services\VaultAgent" | Remove-Item -Force -Verbose
on Powershell version lower than 6.0, this is the way to remove the service.

 

Powershell 6.0+ has a remove service function.
Remove-Service -Name "TestService"

 

Easiest way to remove service if using nssm
stop the VaultAgent Service then run:
nssm remove VaultAgent

 

Additional Tests:

Manual Run from cmd line:
vault.exe agent -config=C:\vault-agent\vault-agent.hcl 2>&1 > c:\temp\VaultAgentLog2.log
Works as expected.  Incidentally this could also be setup as a scheduled task if desired.

 

Register with sc.exe
sc.exe create VaultAgent binPath= "C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -config=C:\vault-agent\vault-agent.hcl 2>&1 > c:\temp\stdout.log" displayName= "Vault Agent" start= auto
Successfully registers as a service with redirection but stderr/stdout NOT captured to file

 

Testing with Powershell calling exe with args
Also tested registering the Vault Agent running in a Powershell script but ran into the issue that the Windows Service then no longer knows if the process has crashed within the script so this did not appear to be a viable solution. It may be possible that a more elaborate script would have caught this.

 

References:

https://docs.microsoft.com/en-us/dotnet/framework/windows-services/how-to-log-information-about-services
https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_redirection?view=powershell-7.1#example-3-send-success-warning-and-error-streams-to-a-file
http://woshub.com/run-powershell-script-as-windows-service/
https://docs.microsoft.com/en-us/archive/msdn-magazine/2016/may/windows-powershell-writing-windows-services-in-powershell
https://nssm.cc/usage

Articles in this section

See more

Related articles