Integrating Kubernetes and Non-Kubernetes with Consul Connect Sidecars

Avatar
Sujata Roy
  • Updated

Introduction

Consul Connect simplifies service-to-service communication by providing a secure and configurable service mesh. This document outlines how to configure Consul Connect using sidecar proxies to connect services running in Kubernetes with those running in non-Kubernetes environments. This hybrid approach allows for seamless integration and communication between different infrastructure platforms.

Expected Outcome

Successful configuration of a service mesh connecting Kubernetes and non-Kubernetes services using Consul Connect sidecar proxies. This includes:

  • Secure communication between Kubernetes and non-Kubernetes services.
  • Proper routing and discovery of services across platforms.
  • Verification of connectivity and service mesh functionality.

Prerequisites

Before proceeding with the configuration, ensure the following prerequisites are met:

  • Consul Cluster: A running and healthy Consul cluster.
  • Kubernetes Cluster: A functional Kubernetes cluster.
  • Non-Kubernetes Environment: A suitable environment for running non-Kubernetes services (e.g., VMs, bare metal).
  • Consul Clients: Consul clients installed and configured on both Kubernetes nodes and non-Kubernetes hosts.
  • Connect-Capable Services: Services designed or configured to use Consul Connect.
  • consul CLI: Access to the consul command-line interface.
  • kubectl CLI: Access to the kubectl command-line interface for Kubernetes interaction.
  • Networking: Appropriate network connectivity between Kubernetes and non-Kubernetes environments, including allowing communication on necessary ports.

Use Case

This configuration is ideal for organizations migrating applications to Kubernetes or those needing to integrate existing non-Kubernetes services with new Kubernetes-based microservices. It enables a unified service mesh across diverse infrastructures, providing consistent security and observability.

Services can be synced in two ways:

  1. Catalog Sync
  2. Connect Sidecar.

Using both methods together is not recommended. For cluster security, prioritize the Connect sidecar. In Consul clusters with Connect enabled, exclusively use the Connect injector. Using Catalog Sync in this scenario can lead to either non-functional integrations (if services are properly secured) or insecure communication (by bypassing the proxy).

For Connect, the tokens do not need to be manually passed. Because, the Connect injector uses the Consul’s Kubernetes auth method to retrieve individual service tokens before the container starts up.

 

Procedure

This setup used an ACL-enabled Kubernetes Consul client (running in Minikube) and a Consul server outside of Kubernetes.

 

  1. The authentication process necessitates the creation of a Kubernetes service account. In this particular instance, the default service account for the cluster was utilized, and, consistent with the Helm release name (sujata), it was designated sujata-consul-client.
    % kubectl get serviceaccounts
NAME                                                 SECRETS   AGE
counting                                             1         45h
default                                              1         46h
static-client                                        1         83m
static-server                                        1         84m
sujata-consul-client                                 1         46h
sujata-consul-connect-injector-webhook-svc-account   1         46h

     

    1. The same RBAC configuration described in this link was used to grant the necessary permissions to the sujata-consul-client service account, which then replaced the consul-auth-method-example.
      ~ % cat sujata_rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: review-tokens
  namespace: default
subjects:
- kind: ServiceAccount
  name: sujata-consul-client
  namespace: default
roleRef:
  kind: ClusterRole
  name: system:auth-delegator
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: service-account-getter
  namespace: default
rules:
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: get-service-accounts
  namespace: default
subjects:
- kind: ServiceAccount
  name: sujata-consul-client
  namespace: default
roleRef:
  kind: ClusterRole
  name: service-account-getter
  apiGroup: rbac.authorization.k8s.io 
  
~~ % kubectl create -f sujata_rbac.yaml
clusterrolebinding.rbac.authorization.k8s.io/review-tokens created
clusterrole.rbac.authorization.k8s.io/service-account-getter created
clusterrolebinding.rbac.authorization.k8s.io/get-service-accounts created
  2. To enable the injector to authenticate, create an ACL auth method using the command below. The CA certificate and token were extracted from the service account's secret.
     consul acl auth-method create -type "kubernetes" \
          -name "my-k8s" \
          -description "This is an example kube method" \
          -kubernetes-host "https://<kubernetes service cluster IP>:443" \
          -kubernetes-ca-file /path/to/kube.ca.crt \
          -kubernetes-service-account-jwt "<jwt token contents of the service account from step 1>"
    • The auth-method-sujata auth method was created as follows:
      % kubectl get endpoints | grep kubernetes
kubernetes                           192.xxx.xx.1x:8443   23h

~ % kubectl get sa sujata-consul-client -o yaml | grep "\- name:"
- name: sujata-consul-client-token-56q9c

Next step was to create the secret token and cert.

~ % kubectl get secret sujata-consul-client-token-56q9c -o yaml | grep token:

The secret from this output was decoded (base64) and then, the value was saved as token value.

~ % kubectl get secret sujata-consul-client-token-56q9c -o yaml | grep ca.crt:

The certificate from this output was decoded (base64) as before and then, the value was saved in a file ca.crt.

~ %consul acl auth-method create \
-type "kubernetes" \
-name "auth-method-sujata" \
-description "This is an auth method using kubernetes for the cluster sujata" \
-kubernetes-host "192.xxx.xx.1x:8443" \
-kubernetes-ca-cert=@ca.crt \
-kubernetes-service-account-jwt="<use the secret token after decoding>”
  3. With the authentication method created, the next step is to create the binding rule.
    ~ % consul acl binding-rule create -method=auth-method-sujata -bind-type=service -bind-name='${serviceaccount.name}' -selector="serviceaccount.name!=default"
  4. To use the new auth method, set connectInject.overrideAuthMethodName in the Helm chart and reinstall the chart. This configures the Connect injector to use the specified auth method when obtaining service tokens.
    • Below is an example client configuration file.
      ---
client:
  enabled: true
  exposeGossipPorts: true
  extraConfig: |-
      {
         "acl": {
          "enabled": true,
          "default_policy": "deny",
          "enable_token_persistence": true,
          "tokens": {
           "agent": "489027f3-43f8-5c6b-d2b1-7300b786ac78"
          }
         }
      }
  grpc: true
  image: "consul:latest"
  join:
    - "10.x.x.xx"
connectInject:
  enabled: true
  aclBindingRuleSelector: "serviceaccount.name!=default"

  # If not using global.bootstrapACLs and instead manually setting up an auth
  # method for Connect inject, set this to the name of your auth method.
  
  overrideAuthMethodName: "auth-method-sujata"
  
global:
  enabled: false
  image: "consul:1.7.2"
server:
  enabled: false
syncCatalog:
  enabled: false
ui:
  enabled: false
    • Because the Consul server runs outside of Kubernetes, manual creation of the auth method is necessary. When the server is deployed within the same Kubernetes environment as the clients, this step is automated. For a more streamlined approach with external servers, store the server's ACL bootstrap token in a Kubernetes secret and reference it via the global.acls.bootstrapToken value in the Helm chart. This will automatically configure all required ACLs, including the client token and the auth method. Further information is available here.
  5. The services are now ready for deployment to Kubernetes.
    • The Connect injector automatically adds sidecars and registers services with Consul, eliminating the need for manual service registration. To inject a sidecar into an existing service, restart the service after installing the Connect injector. Sidecar injection can be enabled via an annotation in the service's specification or by setting the connectInject.default Helm value to true.
    • For example, policies and tokens for static-server and static-client were created using the Consul UI (the command line could also be used). The files used in this example can be found:
    • The static-server policy is shown below:
      service "static-server" {
  policy = "write"
}
service "static-server-sidecar-proxy" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
}
    • The service can be deployed with kubectl create -f consul-helm/static-server.yaml. The services should then be visible in the Consul UI.

Additional Information

Articles in this section

Related articles