The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.
All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.
Introduction
Multiple Consul Datacenters (DCs) can be federated using two methods:
- Mesh Gateways
- Gossip
This article applies to clusters using WAN Federation via a Mesh Gateway on Kubernetes, which is the default method for Kubernetes deployments.
Use Case
When attempting to remove a DC from the federation, when 2 or more federated DCs exist, you may run into issues with Consul attempting to contact the removed DC. Consul uses a caching mechanism to retain both LAN and WAN membership information for agents within each DC connected to the federation. The default TTL for this mechanism can be adjusted in the agent configuration by adjusting the reconnect_timeout_wan parameter.
In environments where there are only 2 DCs that are federated, we recommend disrupting the connection between the DCs on port 8302 and issuing the Consul force-leave command to remove the member.
Procedure
- Modify secondary DC Helm values to disable WAN Federation
federation:
enabled: false
meshGateway:
enabled: falseTip: There is a 72-hour window where Consul will attempt to contact the old cluster. This will disappear after the timeframe has passed.
- Deregister Mesh-gateway in secondary DC
helm upgrade <deployment-name> hashicorp/consul --namespace <namespace> -f values.yaml
You should see that the members of the secondary DC are marked as failed when reviewing the output of the consul members command from the primary DC
kubectl -n consul exec consul-server-0 -- consul members -wan - Remove the remote server(s) from the WAN pool by running the consul force-leave command on the primary DC:
kubectl -n consul exec consul-server-0 -- consul force-leave -prune -wan -token <acl-token> <secondary-dc-node> - Restart the Consul agent on Primary DC that hosts the mesh-gateway
kubectl rollout restart ds/consul-client -n consulTip: A long-term recommendation for connecting multiple Consul DCs might be to use Cluster Peering. This methodology will allow for the DCs to be cleanly removed from membership. Reference the Compared with WAN federation section of the Cluster Peering page for a capability comparison.