Migrating Nomad Keyring from Default AEAD to New Keyring Provider

Avatar
Siddarth Surana
  • Updated

This article guides you through the step-by-step process of migrating your Nomad keyring from the default AEAD provider to a new keyring, including full variable re-encryption to ensure security and consistency.

 

Nomad Configuration File

# -----------------------------+
# BASE CONFIG                  |
# -----------------------------+

datacenter = "dc1"
region = "global"

name = "node-01-server"
data_dir = "/opt/nomad"

log_level = "INFO"
enable_debug = false

# -----------------------------+
# SERVER CONFIG                |
# -----------------------------+

server {
  enabled          = true
  bootstrap_expect = 1
  encrypt = "4ab3809ad92adc0ce44af5c2fa0e1eba"
}

ui {
  enabled = true
}

# -----------------------------+
# NETWORKING CONFIG            |
# -----------------------------+

bind_addr = "10.5.0.125"

ports {
  http = 6443
  rpc  = 4647
  serf = 4648
}

# -----------------------------+
# MONITORING CONFIG            |
# -----------------------------+

telemetry {
  publish_allocation_metrics = true
  publish_node_metrics       = true
  prometheus_metrics         = true
}

# -----------------------------+
# INTEGRATIONS CONFIG          |
# -----------------------------+

consul {
  address               = "10.5.0.125:8500"
  token                 = "99e05bc0-3bf5-f055-82d4-7583e5264201"
  auto_advertise        = true
  server_service_name   = "nomad"
  server_auto_join      = true
}

vault {
  enabled = true
  address = "https://node-01-server:8200"
  task_token_ttl = "1h"
  token = "hvs.B01Vr3k0NWeEtjmhZ1LmDK0C"
  tls_skip_verify = true
}

acl {
  enabled = true
}

tls {
  http      = true
  rpc       = true

  ca_file   = "/etc/nomad.d/nomad-agent-ca.pem"
  cert_file = "/etc/nomad.d/nomad-agent.pem"
  key_file  = "/etc/nomad.d/nomad-agent-key.pem"

  verify_server_hostname = true
}

 

Migration Steps

Step 1: Create a Nomad Variable (Using Default AEAD Keyring) and verify 
root@node-01-server:~# nomad var put project/example user=me password=passw0rd1
Created variable "project/example" with modify index 18

root@node-01-server:~# nomad var get project/example
Namespace   = default
Path        = project/example
Create Time = 2025-11-21T15:32:32Z
Check Index = 18
Items
password = passw0rd1
user     = me

root@node-01-server:~# nomad var list 
Namespace  Path             Last Updated
default    project/example  2025-11-21T16:15:03Z

Step 2: Add Both Old and New Keyring Providers to Configuration. 

Example for AEAD keyring migration (modify to your cloud KMS accordingly):

keyring "aead" {
 active = false
}
keyring "aead" {
 name = "migrationaead"
 active = true
}
Step 3: Verify wrapped keys in Keystore directory
root@node-01-server:/opt/nomad/server/keystore# ls -ltr
total 4
-rw------- 1 root root 2166 Nov 21 16:12 26d55e23-58f4-ce50-8d79-39a8cab7a34e.aead.nks.json
Step 4: Restart Nomad Agent to Apply Configuration Changes
root@node-01-server:/opt/nomad/server/keystore# pkill nomad
root@node-01-server:/opt/nomad/server/keystore# nomad agent \
 -config=/nomad-sandbox/configs/agentfiles/nomad_server.hcl \
 >> /tmp/nomad.log 2>&1 &

root@node-01-server:/opt/nomad/server/keystore# ps
  PID TTY          TIME CMD
 2194 pts/0    00:00:00 nomad
Step 5: Rotate the DEK Full Re-encryption Immediately

This creates a new DEK, wraps it with the new active keyring, and decrypts/re-encrypts all variables with the new key:

root@node-01-server:/opt/nomad/server/keystore# nomad operator root keyring rotate -full -now
Key       State   Create Time           Publish Time
a371e5a3  active  2025-11-21T16:18:19Z  <none>
root@node-01-server:/opt/nomad/server/keystore# nomad operator root keyring list
Key       State     Create Time           Publish Time
26d55e23  inactive  2025-11-21T16:12:08Z  <none>
a371e5a3  active    2025-11-21T16:18:19Z  <none>

Verify that the keystore folder now contains multiple wrapped keys

root@node-01-server:/opt/nomad/server/keystore# ls -ltr
total 12
-rw------- 1 root root 2166 Nov 21 16:12 26d55e23-58f4-ce50-8d79-39a8cab7a34e.aead.nks.json
-rw------- 1 root root 2180 Nov 21 16:18 a371e5a3-1d5d-f47c-760b-a20a108da4f6.aead.migrationaead.nks.json
-rw------- 1 root root 2182 Nov 21 16:18 26d55e23-58f4-ce50-8d79-39a8cab7a34e.aead.migrationaead.nks.json
Step 6: Verify Variable Modify Time is Updated After Re-encryption
root@node-01-server:/opt/nomad/server/keystore# nomad var get -out json project/example
{
  "Namespace": "default",
  "Path": "project/example",
  "CreateIndex": 15,
  "ModifyIndex": 21,
  "CreateTime": 1763741703114019323,
  "ModifyTime": 1763741900374770653,
  "Items": {
    "password": "passw0rd1",
    "user": "me"
  }
}
root@node-01-server:/opt/nomad/server/keystore# nomad var list
Namespace  Path             Last Updated
default    project/example  2025-11-21T16:18:20Z
Step 7: Remove the Inactive (Old) Keyring and Delete Old Key Material
Once variables are fully migrated and accessible with the new keyring, remove the old keyring block from the config and delete old .nks.json files corresponding to inactive keys from keystore.
Step 9: Restart Nomad to Confirm Migration Completeness

After restart, Nomad should be able to access variables with only the new keyring’s DEK:

root@node-01-server:/tmp# nomad var get project/example
Namespace   = default
Path        = project/example
Create Time = 2025-11-21T16:18:20Z
Modify Time = 2025-11-21 16:18:20.374770653 +0000 UTC
Check Index = 21

Items
password = passw0rd1
user     = me

No errors regarding missing keys should occur.

Reference

Summary

This procedure ensures:

  • Safe coexistence of the old and new keyrings during migration.
  • No downtime or data access loss.
  • Full re-encryption of existing variables to the new keyring.
  • Clean removal of old keys after migration.

Following these steps will complete a secure and reliable keyring migration in Nomad.


 

Articles in this section