Problem Statement:
When using Vault version 1.13.7 or newer on a Linux host where DBus is applicable you may encounter a scenario where many leftover zombie processes are being spawned after restarting or stopping the Vault service.
For example when peforming ps aux | grep vault or viewing top many lingering processes can be observed like:
vault 6756 0.0 0.2 9976 2124 ? Ss 15:35 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault 6774 0.0 0.2 9976 2120 ? Ss 15:35 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault 6902 0.0 0.2 9976 2124 ? Ss 15:35 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
vault 7116 0.0 0.2 9976 2120 ? Ss 15:35 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
# ... many more like the above
Requirements:
- Vault version >= 1.13.7
- Linux OS as host using DBus and SystemD
Solutions:
There are four solutions that you may consider as below. The first won't stop the dbus process to start with Vault, but will ensure it's closed when Vault is restarted/stopped.
- Configure SystemD unit file to kill all derived processes upon stop (instead of just the main one) by changing the below values to:
KillMode=control-group
KillSignal=SIGTERM - If
$XDG_RUNTIME_DIRis declared for the Vault service, set:Environment=DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus
- Manually set the socket address for the SystemD unit file:
ExecStart=env DBUS_SESSION_BUS_ADDRESS=/run/user/$(id -u vault)/bus /usr/bin/vault server -config=/etc/vault.d/vault.hcl
- If the above doesn't work, simply divert to /dev/null:
ExecStart=env DBUS_SESSION_BUS_ADDRESS=/dev/null /usr/bin/vault server -config=/etc/vault.d/vault.hcl
Issue continues after applying workarounds:
There might be niche cases where the issue is still happening after applying the workarounds. To see which services are spawning these processes, you can iterate over each PID and list the respective cgroups:
for pid in $(ps aux | grep '/usr/bin/dbus-daemon' | grep -v grep | awk '{print $2}'); do cat /proc/$pid/cgroup; done
This will show which service has the most occurrences, which exposes what is actually spawning these extra dbus processes.
In some cases, there could be other services listed there but Vault actually spawning the dbus processes instead. For example, if using a local service such as pacemaker to periodically check vault status, it can cause Vault to spawn another dbus process that will be reported as belonging to pacemaker instead (since that is the cgroup involved in starting it in the first place. If that is the case, using curl instead of vault-cli to get information from Vault periodically should fix the issue.