The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

The following article will provide steps on how to rotate/replace the initial_management token on a Consul Cluster. This is useful when you have lost the existing initial_management token or want to rotate the current token as part of security best practices.

NOTE: The initial_managementtoken is also known as the master token (for Consul < 1.11) or as a bootstrap token.

Expected Outcome

Successfully rotate the initial_management token on an ACL-enabled Consul Cluster. 

Prerequisites

• System Level admin access, as this will be required when performing a rolling restart of the Consul Server agents
• Access to a Consul token with operator:read permission to query the raft peer list during the operation

WARNING: This operation requires a rolling restart of Consul server agents, triggering a leadership switch when the existing leader is restarted. Based on your organizations policies, ensure that you have engaged the relavant change management process if any.

Use Cases

• Recover a lost initial_management token (by creating a new one)
• Rotating the existing one as part of security requirements

Procedure

• Create a UUID to be used as the new token secret. You may use the uuidgen utility on Unix/Linux systems.

  $ uuidgen | tr "[:upper:]" "[:lower:]"
  e95538d0-5cb1-4245-a217-338ed6f7dae8  

• Set the generated UUID in the Consul Configuration file on all consul server agents, as shown below:
  

  acls {
    tokens {
      initial_management = "e95538d0-5cb1-4245-a217-338ed6f7dae8"
    }
  }

  The same token must be set on all Consul server agents, as these tokens will only get activated when one of the agents becomes the leader after the rolling restart.
  

  NOTE: Make sure you leave all the existing configurations under the acl {} block as it is and only add/modify the initial_managment token.

• Initiate a rolling restart of the Consul Server agents, starting with the followers first and the leader last. Use the service manager in use to initiate the restart of the agents.
  • The following example uses systemd.
    

    sudo systemctl restart consul

• Using a token with operator:read privilege, query the raft peers; before restarting each server, and ensure that the previously restarted servers have their Voter status as True by running the following command.
  

  $ consul operator raft list-peers -token <privileged-token>
  Node ID Address State Voter RaftProtocol
  consul-server-0 aef36f1b-3953-7d57-cdda-e4d9d2d7ee6e 127.0.0.1:8300 leader true 3

• Once the rolling restart has been completed and a new leader is elected, the new token will be active and can be queried.
  

  $ consul acl token list -token e95538d0-5cb1-4245-a217-338ed6f7dae8 -format json | jq '.[] | select(.Policies)
      | select(.Policies[] | .ID=="00000000-0000-0000-0000-000000000001") | {AccessorID:
      .AccessorID, Description: .Description, CreateTime: .CreateTime}'
    {
      "AccessorID": "ea48401f-139f-6ede-1fdd-b6ce66dad04d",
      "Description": "Master Token",
      "CreateTime": "2023-02-14T10:59:15.643144+11:00"
    }
    {
      "AccessorID": "4be9391f-3acc-0031-7515-c0212e864193",
      "Description": "Master Token",
      "CreateTime": "2022-01-01T10:58:48.718415+11:00"
      }

  • In the above step, use the newly created token as this will provide verification that the new token is active.
    • If the above query is successful, copy the old AccessorID for the next step.
      • Determine the older AccessorID based on the CreateTime.
• Now that the new token is active, delete the old token, especially if this rotation was due to a token breach.
  

  IMPORTANT: Before deleting the old token, make sure to identifiy all areas where the old token was used and replace it with the new token.

  Alternatively, as a best practice, try to replace the old token by creating newer tokens with minimal required policies for specific usecase.

  $ consul acl token delete -token e95538d0-5cb1-4245-a217-338ed6f7dae8 -id 4be9391f-3acc-0031-7515-c0212e864193
  Token "4be9391f-3acc-0031-7515-c0212e864193" deleted successfully

  • In the above
    • The token (-token) is the new initial_management token
    • The ID (-id) is the old initial_management token.

Additional Information

• ACL Parameter initial_management token

• Raft Operator HTTP API Read Configuration