Enable Consul service discovery within different k8s clusters, using syncCatalog

Avatar
Georgi Tsvetkov
  • Updated

Introduction

We can use Consul service discovery capability in order to expose/discover Kubernetes services that are part of different Kubernetes clusters. Regardless where the Consul server resides (outside Kubernetes, or inside Kubernetes).

  • In order to sync service that are part of different Kubernetes cluster, we need to use adminPartitions, so Consul can distinguish the incoming through syncCatalog services.
  • If we attempt to build this setup without having Kubernetes clusters being part of different adminPartitions, we will see services invalidating one another and then re-added in a loop, regardless whether the service names/namespaces overlap or not: 
[DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=
[INFO]  to-k8s/source: received services from Consul: count=6
[INFO]  to-consul/sink: invalid service found, scheduling for delete:
[DEBUG] to-consul/sink: [scheduleReapServiceLocked] service being scheduled for deregistration: namespace="
[INFO]  to-consul/sink: registering services
[DEBUG] to-consul/sink: [watchReapableServices] services returned from catalog: services=
[DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=
  • The service sync feature deploys a long-running process which can run either inside or outside of a Kubernetes cluster. However, running this process within the Kubernetes cluster is generally easier since it is automated using the Helm chart. Each synced service is registered onto a Consul node called k8s-sync. This node is not a real node. Instead, the catalog sync process is monitoring Kubernetes and syncing the services to Consul.

Use Case

  • Kubernetes services synced to the Consul catalog enable Kubernetes services to be accessed by any node that is part of the Consul cluster, including other distinct Kubernetes clusters. For non-Kubernetes nodes, they can access services using the standard Consul DNS or HTTP API. This also causes all discoverable services to be part of a central service catalog in Consul for further syncing into alternate Kubernetes clusters or other platforms.
  • Using syncCatalog, in conjunction with adminPartitions, you can enable service discovery within multiple Kubernetes clusters

Procedure

  • You can have Consul Server outside Kubernetes (or inside Kubernetes).
  • You can use the following Helm chart for the Kubernetes clusters, by modifying the adminPartitions and k8sAllowNamespaces fields, as well as the external/internal Consul server/clients details accordingly 
global:
    adminPartitions:
        enabled: true
        name: <non-default Partition name>
    enableConsulNamespaces: true

    acls:
        manageSystemACLs: true

connectInject:
    enabled: false

externalServers:
    enabled: true
    hosts: ["<Consul API destination>"]
    httpsPort: 8501
    grpcPort: 8503 # The grpc_tls port on Consul Servers
    tlsServerName: "<hostname tied to the https API>"

client:
  # If true, the chart will install all
  # the resources necessary for a Consul client on every Kubernetes node. This _does not_ require
  # `server.enabled`, since the agents can be configured to join an external cluster.
  # @default: global.enabled
  # @type: boolean
  enabled: null
  join: [“<A list of valid `-retry-join` values>”]
  exposeGossipPorts: true # If true, the Helm chart will expose the clients' gossip ports as hostPorts. This is only necessary if pod IPs in the k8s cluster are not directly routable and the Consul servers are outside of the k8s cluster.

  enterpriseLicense:
    secretName: license
    secretKey: key

syncCatalog:
    enabled: true
    toConsul: true
    toK8S: false
    k8sAllowNamespaces: ["specific-namespace","specific-namespace2"]
    consulNamespaces:
        mirroringK8s: false
    addK8SNamespaceSuffix: false
    syncClusterIPServices: null
    ingress:
        enabled: null
        loadBalancerIPs: null
    nodePortSyncType: null
    aclSyncToken: null

 

Additional Information

Articles in this section

Related articles