The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.
All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.
Multiple Consul Datacenters (DCs) can be federated using two methods:
- Mesh Gateways
This article applies to clusters using WAN Federation via a Mesh Gateway on Kubernetes, which is the default method for Kubernetes deployments.
When attempting to remove a DC from the federation, when 2 or more federated DCs exist, you may run into issues with Consul attempting to contact the removed DC. Consul uses a caching mechanism to retain both LAN and WAN membership information for agents within each DC connected to the federation. The default TTL for this mechanism can be adjusted in the agent configuration by adjusting the reconnect_timeout_wan parameter.
In environments where there are only 2 DCs that are federated, we recommend disrupting the connection between the DCs on port 8302 and issuing the Consul force-leave command to remove the member.
- Modify secondary DC Helm values to disable WAN Federation
- Deregister Mesh-gateway in secondary DC
helm upgrade <deployment-name> hashicorp/consul --namespace <namespace> -f values.yaml
You should see that the members of the secondary DC are marked as failed when reviewing the output of the consul members command from the primary DC
kubectl -n consul exec consul-server-0 -- consul members -wan
- Remove the remote server(s) from the WAN pool by running the consul force-leave command on the primary DC:
kubectl -n consul exec consul-server-0 -- consul force-leave -prune -wan -token <acl-token> <secondary-dc-node>
- Restart the Consul agent on Primary DC that hosts the mesh-gateway
kubectl rollout restart ds/consul-client -n consul