Introduction

Consul requires a majority vote from the predefined quorum of nodes to bootstrap the cluster. Server nodes inside a cluster become voters to elect a leader of the cluster, and this process is known as the leader election. The active Consul nodes participating in the leader election are known as voters and contribute to the election as a means to elect leader nodes within the cluster.

Consul uses Raft as the consensus protocol which is used during leader elections. A healthy cluster will have all server nodes available to vote in the election, however, there are times when a Consul server will be demoted or removed as a voter from the cluster.

It is important to understand that the release of Consul that is being used is important in determining reasons why a voter can get removed or demoted. In the open-source releases of Consul the cluster is maintained by the server administrator, however, Consul Enterprise edition introduces automated features that can perform operations that will cause a voter to get demoted, or removed from the cluster. When using the open-source release of Consul most of the reasons a voter will get demoted, or removed, is due to an unhealthy agent, but this is not as simple in the Enterprise version.

OSS vs Enterprise

Open Source

Consul voters will only be removed if the node fails, or leaves as determined via gossip.

• Node status observed through gossip

  • The agent leaves the cluster

  • The agent was removed from the cluster

  • Agent failure

    • Consul application crash

    • The host machine powered down

    • Networking unavailable

Enterprise

Enterprise Consul introduces automated features from Autopilot which can add more situations where a voter might be demoted or removed from the cluster. In addition, similar to Open Source voters can be removed, or demoted in the cluster if it’s determined that the node is either unhealthy or left the cluster.

• Node status observed through gossip

  • The agent leaves the cluster

  • The agent was removed from the cluster

  • Agent failure

    • Consul application crash

    • The host machine powered down

    • Networking unavailable

• Autopilot

  • Auto-Upgrade Process

    • Old version nodes will get demoted when new target version voters are promoted

      • The Add new servers section in the Automate Upgrades with Consul Enterprise tutorial explains when this happens

      • Note: If the option disable_upgrade_migration is set totrue, the voters will not be rotated out of the cluster automatically.

        • In case the feature is disabled (disable-upgrade-migration=true) in your datacenter you can enable it using theconsul operator autopilot set-config -disable-upgrade-migration=falsecommand.

  • Redundant extra voters in a zone will be demoted once a healthy replacement in another zone is available.

    • Redundancy zones will do whatever is necessary to keep 1 voter per zone.

    • Total Zone Failure

      • Autopilot will do anything necessary to ensure the right number of voters remains

        • It is possible that multiple voters will exist in one zone, however, the extra voter will be first to be demoted once the failed zone becomes healthy again.

  • Unhealthy servers in a zone will be demoted/removed

    • When AutoPilot is disabled there is a 72-hour delay until unhealthy servers are removed from the cluster.

FAQ

Node      ID                                    Address           State     Voter  RaftProtocol
consul-1  2c8e12bd-d666-7ce6-ac83-b0f1cca8eaec  192.168.0.3:8300  follower  true   3
consul-2  2e9b0e74-658d-da08-0495-304bfebd5746  192.168.0.4:8300  follower  true   3
consul-3  2dfb3902-a331-b008-4eb7-df7870f91589  192.168.0.5:8300  leader    true   3