HSM PKCS11 Setup Issues & Errors

Avatar
Mehdi Ahmadi
  • Updated

There are two symptoms that are common when HSM's integrations are not complete, not correctly configured or otherwise have changed on the HSM, Hardware or Operating System since the time of its previous working state.

First Common HSM Issue

One scenario is when Vault is emitting PKCS#11 library: pkcs11: errors in its operational logs with a hex-decimal value such as: 0x5: CKR_GENERAL_ERROR; these codes may be reviewed further in the specification of the standard library: PKCS#11 Cryptographic Token Interface Base Specification

In the case of CKR_GENERAL_ERROR the function is applicable to many areas of the library yet it's indicative of any issues where Vault is not able to interface with the HSM in the configured manner. Other examples can also include: 0x6: CKR_FUNCTION_FAILED and more.

Second Common HSM Issue

The second scenario is when Vault is not able to properly interface with the PKCS11 libraries at all resulting in a segmentation violation exception like these:

… [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
… [DEBUG] storage.raft.fsm: time to open database: elapsed=364.199µs path=/vault/data/vault.db
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16ba1e2]
goroutine 1 [running]:
github.com/miekg/pkcs11.(*Ctx).Initialize.func1(0x0, 0xc000aa6000)
… [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
… [WARN] storage.consul: appending trailing forward slash to path
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14b4db4]
goroutine 1 [running]:
github.com/miekg/pkcs11.(*Ctx).Initialize.func1(0xc000875d01)

Solutions

The solution in both cases is to ensure proper configuration, privileges and environment variables needed for the executing Vault user as well as its launch definitions in the context of Systemd as per what's outlined in the HSM Vendors Instruction manuals.

A simple approach to confirm if HSM setup is correctly in place is to use opensc and execute listing operations via the same Vault user:

# // `yum -y install opensc` or `apt-get install -yq opensc`
# // path to Vault configuration
VHCL=/etc/vault.d/vault.hcl ;

# // set .so module path same as in HCL
HSM_LIB=$(grep -F 'lib' ${VHCL} | cut -d'"' -f2) ;

# // attempt HSM slot listing as 'vault' executing user
sudo -u vault pkcs11-tool --module ${HSM_LIB} -L ;

# // Use HSM slot via Hex or Short value same as what's set in Vault (HCL Decimal value)
sudo -u vault pkcs11-tool --module ${HSM_LIB} -l -t --slot=0x327f87a4 ;

Another simple approach is to attempt the startup of Vault on CLI as it's configured in its Systemd unit file and compare it to environment variables (env) from where it may launch correctly versus any Environment=variable(s) missing in the systemd unit file that's failing. Eg:

sudo -u vault /bin/bash ;
grep 'ExecStart' /etc/systemd/system/vault.service ;
  # ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
# // compare setup environments needed for HSM
env ; 
# // attempt launch:
/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl ;
  # ...

In some cases where Security-Enhanced Linux (SELinux) has been enabled and set to Enforcing mode, access to the HSM libraries by the Vault process will be denied unless explicitly granted. Temporarily disabling SELinux can be used as a troubleshooting step. A permanent solution would include adding an SELinux rule to grant access to the specified HSM library.

Related Links

Articles in this section

See more

Related articles