Nomad uses two types of encryption keys to secure sensitive data such as Variables and workload identities: the Key Encryption Key (KEK) and the Data Encryption Key (DEK). Their storage and replication mechanisms have evolved significantly starting from Nomad version 1.9.
Key Encryption Key (KEK)
Legacy Keystore (Pre-Version 1.9)
- Nomad encrypts DEK with a unique KEK for each server node, and these KEKs are not shared between servers. This means each Nomad server node has its own distinct KEK to protect key material locally. However, there is functionality to use external Key Management Systems (KMS) or Vault transit encryption where the KEK is stored outside Nomad, improving security.
-
When using the default AEAD provider, the key encryption key (KEK) is stored cleartext in Raft alongside the encrypted data encryption keys (DEK).Nomad automatically migrates the key storage for all key material on the first
root_key_gc_interval
after all servers are upgraded to 1.9.0.Note : When using external KMS or Vault transit encryption for the
keyring
provider, the key encryption key (KEK) is stored outside of Nomad and no cleartext key material exists on disk
Data Encryption Key (KEK)
Nomad servers maintain an encryption keyring used to encrypt Variables, sign task workload identities, and sign OIDC client assertion JWTs.
Legacy Keystore (Pre-Version 1.9)
- key metadata is stored in Raft, but the encryption key material was stored in a separate file in the
keystore
subdirectory of the Nomad data directory. These files have the extension.nks.json
- Each server asynchronously fetched key material from the leader or other servers through a key rEach server runs a key replication process that watches for changes to the state store and fetches the key material from the leader asynchronously
- All key material (DEKs) is stored inside the Raft - https://github.com/hashicorp/nomad/issues/23977
- Migration to Raft-based key storage occurs automatically after an upgrade and the first scheduled root key garbage collection interval.
- Key replication is handled automatically via Raft eliminating the need for separate key replication processes.