The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

This article is related to upgrading from Consul v0.7.1 to v0.8.5. The Consul Upgrade Instructions provided the path starting from 0.8.5. Versions of Consul < v0.7.1 do not have a snapshot command, which makes upgrades riskier. 

Expected Outcome

Successfully upgrade Consul servers running on v0.7.1 to v0.8.5.

Prerequisites

• Already running a Consul cluster on v.0.7.1
• Review General Upgrade Process

Procedure

Table of Contents

• Preparing Consul v0.7.1
• Install Consul binary v0.8.5
• Check Cluster state
• Recovery
• Conclusion
• Additional Information

Preparing Consul v0.7.1

1. Verify the Consul cluster is running by issuing the commandconsul members:
   

   Node      Address             Status  Type    Build  Protocol  DC
   backend   172.31.31.87:8301   alive   client  0.7.1  2         dc1
   frontend  172.31.17.72:8301   alive   client  0.7.1  2         dc1
   server-1  172.31.28.4:8301    alive   server  0.7.1  2         dc1
   server-2  172.31.31.136:8301  alive   server  0.7.1  2         dc1
   server-3  172.31.29.11:8301   alive   server  0.7.1  2         dc1

2. Outputs of consul operator raft -list-peers:
   

   Node      ID                  Address             State     Voter
   server-2  172.31.31.136:8300  172.31.31.136:8300  leader    true
   server-1  172.31.28.4:8300    172.31.28.4:8300    follower  true
   server-3  172.31.29.11:8300   172.31.29.11:8300   follower  true

3. Create a snapshot under /opt/consul/raft/snapshots folder
   

   $ consul snapshot save /opt/consul/raft/snapshots/backup.snap
   Saved and verified snapshot to index 3068

Install Consul binary v0.8.5

1. Start from one of the follower nodes, run consul leave or systemctl stop consul to unjoin the cluster

   $ consul leave 
   Graceful leave complete

2. Run consul members on the leader node
   

   Node      Address             Status  Type    Build  Protocol  DC
   backend   172.31.31.87:8301   alive   client  0.7.1  2         dc1
   frontend  172.31.17.72:8301   alive   client  0.7.1  2         dc1
   server-1  172.31.28.4:8301    left    server  0.7.1  2         dc1
   server-2  172.31.31.136:8301  alive   server  0.7.1  2         dc1
   server-3  172.31.29.11:8301   alive   server  0.7.1  2         dc1

3. On the node that has left the cluster, install Consul version 0.8.5 with the commands listed below
   

   mv /usr/bin/consul /usr/bin/consul_old
   curl -s https://releases.hashicorp.com/consul/0.8.5/<OS Architecture Type> -o consul.zip
   unzip -d /usr/bin consul.zip consul
   chmod +x /usr/bin/consul
   rm -fr consul.zip

4. Run consul version
   

   Consul v0.8.5
   Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

5. Rejoin the cluster by restarting Consul via systemctl restart consul
6. Run consul members and consul operator raft -list-peers to verify if the node joined the cluster and the Voter status is true:
   

   $ cosul members
   Node      Address             Status  Type    Build  Protocol  DC
   backend   172.31.31.87:8301   alive   client  0.7.1  2         dc1
   frontend  172.31.17.72:8301   alive   client  0.7.1  2         dc1
   server-1  172.31.28.4:8301    alive   server  0.8.5  2         dc1
   server-2  172.31.31.136:8301  alive   server  0.7.1  2         dc1
   server-3  172.31.29.11:8301   alive   server  0.7.1  2         dc1
   
   $ consul operator raft -list-peers
   Node      ID                  Address             State     Voter
   server-2  172.31.31.136:8300  172.31.31.136:8300  leader    true
   server-3  172.31.29.11:8300   172.31.29.11:8300   follower  true
   server-1  172.31.28.4:8300    172.31.28.4:8300    follower  true

   • We can see that the follower node server-1 is now running Consul v0.8.5 and has successfully joined the cluster.
7. Repeat these steps on other server nodes.
   
   NOTE: The leader node should be updated last.
8. Once all the server agents are upgraded, begin a rollout of client agents following the same process.

Check Cluster state

1. Once the consul binaries are all replaced with v0.8.5, run consul members and consul operator raft -list-peers to verify that all nodes are now running the new version:

   $ consul members
   Node      Address             Status  Type    Build  Protocol  DC
   backend   172.31.31.87:8301   alive   client  0.8.5  2         dc1
   frontend  172.31.17.72:8301   alive   client  0.8.5  2         dc1
   server-1  172.31.28.4:8301    alive   server  0.8.5  2         dc1
   server-2  172.31.31.136:8301  alive   server  0.8.5  2         dc1
   server-3  172.31.29.11:8301   alive   server  0.8.5  2         dc1
   
   $ consul operator raft -list-peers
   Node      ID                  Address             State     Voter  RaftProtocol
   server-1  172.31.28.4:8300    172.31.28.4:8300    leader    true   2
   server-3  172.31.29.11:8300   172.31.29.11:8300   follower  true   2
   server-2  172.31.31.136:8300  172.31.31.136:8300  follower  true   2

2. Read a K/V pair:

   $ consul kv get -detailed foo/bar
   CreateIndex  532
   Flags        0
   Key          foo/bar
   LockIndex    0
   ModifyIndex  532
   Session      -
   Value        foobar

3. Verify service status:

   $ curl http://localhost:8500/v1/catalog/service/frontend?pretty
   [
       {
           "ID": "10ce29ff-1c7c-710a-a1e7-d71e91bb2be2",
           "Node": "frontend",
           "Address": "172.31.17.72",
           "Datacenter": "dc1",
           "TaggedAddresses": {
               "lan": "172.31.17.72",
               "wan": "172.31.17.72"
           },
           "NodeMeta": {},
           "ServiceID": "frontend",
           "ServiceName": "frontend",
           "ServiceTags": [],
           "ServiceAddress": "127.0.0.1",
           "ServicePort": 6060,
           "ServiceEnableTagOverride": false,
           "CreateIndex": 3448,
           "ModifyIndex": 3448
       }
   ]

4. This verifies that the new version of Consul is able to pull data from the existing data-dir and Consul has now successfully been upgraded from v0.7.1 to v0.8.5.

Recovery

If for some reason the Consul version had to be rolled back to v0.7.1, restore the snapshot backup taken during these steps.

The below steps assume the binary has already been placed back on v0.7.1

1. Run the below command to restore Consul from the previously saved snapshot

$ consul snapshot restore /opt/consul/raft/snapshots/backup.snap
Restored snapshot

Conclusion

No issues were found while upgrading from 0.7.1 to 0.8.5. Consul 0.8.5 is compatible with 0.7.1. However, this article mentioned that when upgrading to Consul 0.8.x from a version lower than 0.7.0, users will need to set the -raft-protocol option to 1 in order to maintain backward compatibility with the old servers during the upgrade.

Additional Information

• Consul Upgrade Instructions

• Consul 0.7.1 GitHub page
  • Consul 0.7.1 Snapshot Command reference
• Consul 0.8.5 GitHub page