Issue
After upgrading to Vault 1.10.6 a PKCS11 error is encountered whenever a service reload (eg: systemctl reload vault) or SIGHUP (eg: kill -SIGHUP $(pidof vault)) signal is made to the Vault process for reasons of Audit file logrotate or a configuration reload which then result in an output similar to:
Error finalizing seals: pkcs11: error finalizing PKCS11 library: finalizing an uninitialized pkcs#11 context panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x2e6528e]
In the logs you may see an error and warning sequence similar to that of:
==> Vault reload triggered
2022-11-25T02:42:00.700Z [DEBUG] would have sent systemd notification (systemd not present): notification=RELOADING=1
2022-11-25T02:42:00.701Z [WARN] unknown or unsupported field tls_prefer_server_cipher_suites found in configuration at /opt/vault/config.hcl:34:3
2022-11-25T02:42:00.701Z [DEBUG] core: set config: sanitized config="{\"api_addr\":\"https://yourvaultserver.com:8200\",\"cache_size\":0,\"cluster_addr\":\"https://yourvaultserver.com:8201\",\"cluster_cipher_suites\":\"\",\"cluster_name\":\"\",\"default_lease_ttl\":0,\"default_max_request_duration\":0,\"disable_cache\":true,\"disable_clustering\":false,\"disable_indexing\":false,\"disable_mlock\":true,\"disable_performance_standby\":false,\"disable_printable_check\":false,\"disable_sealwrap\":false,\"disable_sentinel_trace\":false,\"enabl..., TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA\",\"tls_client_ca_file\":\"/opt/vault/certs/ca.pem\",\"tls_disable\":\"false\",\"tls_disable_client_certs\":\"true\",\"tls_key_file\":\"/opt/vault/certs/vault.key\",\"tls_min_version\":\"tls12\",\"tls_prefer_server_cipher_suites\":\"true\"},\"type\":\"tcp\"}],\"log_format\":\"\",\"log_level\":\"Debug\",\"log_requests_level\":\"\",\"max_lease_ttl\":0,\"pid_file\":\"/opt/vault/vault.pid\",\"plugin_directory\":\"/opt/vault/plugins/\",\"raw_storage_endpoint\":false,\"seals\":[{\"disabled\":false,\"type\":\"pkcs11\"}],\"storage\":{\"cluster_addr\":\"https://yourvaultserver.com:8201\",\"disable_clustering\":false,\"redirect_addr\":\"https://yourvaultserver.com:8200\",\"type\":\"raft\"},\"telemetry\":{\"add_lease_metrics_namespace_labels\":false,\"circonus_api_app\":\"\",\"circonus_api_token\":\"\",\"circonus_api_url\":\"\",\"circonus_broker_id\":\"\",\"circonus_broker_select_tag\":\"\",\"circonus_check_display_name\":\"\",\"circonus_check_force_metric_activation\":\"\",\"circonus_check_id\":\"\",\"cir...}}"
2022-11-25T02:42:00.701Z [INFO] audit: reloading file audit backend: path=file/
Error finalizing seals: pkcs11: error finalizing PKCS11 library: finalizing an uninitialized pkcs#11 context
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x2e6528e]
goroutine 1 [running]:
github.com/hashicorp/vault/vault.(*ManagedKeyRegistry).invalidateAllKeys(0xc001b0cb40)
/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/managed_key_registry_ent.go:575 +0x6e
github.com/hashicorp/vault/vault.(*Core).ReloadManagedKeyRegistryConfig(...)
/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/managed_key_registry_ent.go:131
github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc0001c7380, {0xc000116080, 0x1, 0x1})
/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/command/server.go:1639 +0x3fca
github.com/mitchellh/cli.(*CLI).Run(0xc0002b12c0)
/home/runner/go/pkg/mod/github.com/mitchellh/cli@v1.1.2/cli.go:262 +0x5f8
github.com/hashicorp/vault/command.RunCustom({0xc000116070, 0x2, 0x2}, 0xc0000001a0)
/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/command/main.go:180 +0x9f7
github.com/hashicorp/vault/command.Run(...)
/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/command/main.go:88
main.main()
/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/main.go:10 +0x52
WARNING! The following cipher suites defined by 'tls_cipher_suites' are
blacklisted by the HTTP/2 specification:
[TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA]
Please see https://tools.ietf.org/html/rfc7540#appendix-A for further information.
Overview of possible solutions
Solutions:
-
Upgrade to version 1.10.8 or higher - thereafter the error will no longer be encountered with any subsequent SIGHUP.
- If an upgrade is not immediately possible you may set the configuration parameter:
disable_cacheto false in the Vault (HCL) config file as an interim work-around - for example:
# // existing contents of vault config file: /etc/vault.d/vault.hcl
# // ... add:
disable_cache=false