Overview
In Nomad versions 1.7.x and 1.8.x Enterprise, users have encountered an issue where the root keyring, used for signing workload identities, does not automatically rotate every 30 days as expected. The automatic rotation of the root keyring is a critical feature for maintaining security, ensuring that workload identities are always signed using a current key. However, in affected versions, the keyring remains unchanged after the initial creation.
This issue has been resolved in Nomad version 1.8.3+ent. We recommend upgrading to this version to address the problem.
Related Github Issue
https://github.com/hashicorp/nomad/pull/23577
Nomad Change Log
You can see more about this bug fix in the Nomad version 1.8.3 change log.
Issue Description
In the affected versions of Nomad, the root keyring, which should automatically rotate every 30 days according to the documentation, fails to rotate. Users will observe that the keyring remains unchanged from its creation date, and the automatic rotation process does not occur.
Steps to Reproduce:
- Deploy a Nomad cluster with version 1.7.x or 1.8.x.
- Check the root keyring creation date using the following command.
nomad operator root keyring list -verbose
- Observe that the root keyring has not rotated after 30 days.
Expected Behavior:
The root keyring should rotate automatically every 30 days, generating a new active key while deactivating the old one.
Actual Behavior:
The root keyring does not rotate automatically. Instead, it remains unchanged, with the same creation date as when it was initially configured. Manual intervention is required to rotate the keyring.
Example Output
Below is an example output demonstrating the behavior:
-
Root Keyring Creation Date (In this scenario - 36 days old when tested this article, no rotation):
$ nomad operator root keyring list -verbose
Key State Create Time
f8be3135-9d2b-0744-b0ce-f27a6253a7f7 active 2024-06-04T09:11:35+05:30
- Manual Rotation of the Root Keyring: To manually rotate the root keyring, use the following command:
$ nomad operator root keyring rotate
After rotation:
$ nomad operator root keyring list -verbose
Key State Create Time
78d56987-233f-ec41-dc76-2277323b3f02 active 2024-07-10T14:14:06+05:30
f8be3135-9d2b-0744-b0ce-f27a6253a7f7 inactive 2024-06-04T09:11:35+05:30
Workaround
Until upgrading to Nomad version 1.8.3+ent, users can manually rotate the root keyring to ensure security standards are met. To manually rotate the keyring, use the following command:
nomad operator root keyring rotate
This will generate a new keyring and mark the previous key as inactive.
Resolution
This issue has been resolved in Nomad version 1.8.3+ent. We strongly recommend upgrading to this version or later to ensure that the root keyring for workload identity signing rotates automatically every 30 days, as expected.
To upgrade, refer to the official Nomad documentation for instructions on performing a safe upgrade.