Introduction

This article is meant to provide amplifying configuration guidelines for implementing an OpenShift cluster ingress solution to Consul's service mesh using Consul's API Gateway.

Environment Assumptions

This article assumes the Consul environment is already deployed and configured on a live OpenShift cluster for purposes of leveraging:

Article Example Resources:

Service Mesh Service: frontend
- See full kubernetes service definition in Appendix: frontend-service below.
API Gateway Name: api-gateway
OpenShift Ingress Domain: apps.consul-openshift-cluster.sbx.hashidemos.io

Ingress Terms on OpenShift

Ingress Operator
- Operator that implements the IngressController API and enables external client access to the OpenShift Container Platform cluster internal services.
- Default Ingress Domain: apps.<openshift_base_domain>
OpenShift Route
- Resource responsible for providing the public-facing hostname for an internal cluster application.
- Default URL DNS: <application_name>.apps.<openshift_base_domain>
OpenShift Ingress Object
- Kubernetes resource that handles cluster applications that integrate with Ingress-type resources, but do not explicitly have OpenShift Route resources

Preparing for Consul API Gateway on Kubernetes

If not already accomplished, the Consul API Gateway CRDs should be installed in the cluster via the following helm chart override values for connectInject:

connectInject:
  enabled: true
  apiGateway:
    manageExternalCRDs: true

This override can be applied after installation using helm upgrade if necessary.

This step enables the cluster to deploy API Gateway resources on the cluster.

Configure API Gateway(s) and Application Listener(s)

The Gateway configuration CRD will:

Define the gateway for deployment
Define the exposed listener port for external ingress access

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: api-gateway
  namespace: consul
spec:
  gatewayClassName: consul
  listeners:
    - name: frontend
      protocol: HTTP # options: HTTP | HTTPS | TCP
      port: 2080     # options: OpenShift needs extra permission to set port num < 1024
      allowedRoutes:
        namespaces:
          from: All  # options: All or Same or Specific

Important Note on OpenShift:

Please note that when selecting a listener port you must select a non-privileged port for exposing a listener unless you, or another cluster admin, manage and allow permissions for this. Privileged Ports are defined as port numbers < 1024.
The API Gateway Envoy listener will fail to create if this is not respected without explicitly granting privileged port permission.

Configure API Gateway Route

The API Gateway Route resource defines the API Gateway relationship to an internal service-mesh application.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: frontend
  namespace: consul
  labels:
    app: frontend
spec:
  parentRefs:
    - name: api-gateway
      namespace: consul
  rules:
    - backendRefs:
        - kind: Service
          name: frontend
          namespace: consul
          port: 9090

This CRD tells the API Gateway (api-gateway) to establish a connection and provide listener accessibility to the backend service reference frontend on port 9090.

Configure OpenShift Ingress Object

This is the OpenShift-specific step required to properly establish secure communication to the mesh services via the API Gateway listener, that is exposed on the gateway pod itself.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-api-gateway
  namespace: consul
  labels:
    app: api-gateway
  annotations:
    route.openshift.io/termination: "edge"
spec:
  rules:
    - host: frontend-api-gateway-consul.apps.consul-openshift-cluster.sbx.hashidemos.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-gateway
                port:
                  number: 2080
  tls:
    - {}

Important Configuration Options:

metadata.labels.app & spec.rules.paths.backend.service.name:
- Notice these are both pointing to the api-gateway resource itself and not to the mesh application. Remember that we're pointing OpenShift to use the API gateway exposed listener, not the specific application listener.
spec.rules.host: This hostname is required to be explicitly defined when working with Ingress objects. See Creating a route through an Ingress object for more details on why.

Configure Consul Service Intention (API Gateway to Service Permission)

To allow the API Gateway to communicate explicitly with a particular service residing on the mesh, intentions must permit communication.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: frontend
spec:
  destination:
    name: frontend
    namespace: consul
  sources:
    - name: api-gateway
      namespace: consul
      action: allow

Verify External Connectivity

Using the Ingress object hostname specified in the resource definition, the service should now be accessible by DNS.

$ curl -sk https://frontend-api-gateway-consul.apps.consul-openshift-cluster.sbx.hashidemos.io

Example Response

{
  "name": "frontend-service",
  "uri": "/",
  "type": "HTTP",
  "ip_addresses": [
    "10.129.2.18"
  ],
  "start_time": "2024-01-29T20:07:26.269830",
  "end_time": "2024-01-29T20:07:26.278833",
  "duration": "9.00255ms",
  "body": "FRONTEND FRONTEND FRONTEND FRONTEND",
  "upstream_calls": {
    "http://backend.consul.svc.cluster.local": {
      "name": "backend-service",
      "uri": "http://backend.consul.svc.cluster.local",
      "type": "HTTP",
      "ip_addresses": [
        "10.131.0.23"
      ],
      "start_time": "2024-01-29T20:07:26.277288",
      "end_time": "2024-01-29T20:07:26.277383",
      "duration": "94.62µs",
      "headers": {
        "Content-Length": "283",
        "Content-Type": "text/plain; charset=utf-8",
        "Date": "Mon, 29 Jan 2024 20:07:26 GMT",
        "Server": "envoy",
        "X-Envoy-Upstream-Service-Time": "3"
      },
      "body": "BACKEND BACKEND BACKEND BACKEND",
      "code": 200
    }
  },
  "code": 200
}

Summary

This article briefly summarizes the additional element of providing Consul Service Mesh external traffic access to a cluster deployed on OpenShift. This doesn't cover all possible scenarios but is intended to provide insight into what a secure means of gaining access to a mesh service would look like using Consul API Gateway. Please review the OpenShift and Consul API Documentation below for further details on configuration options.

References:

Consul API Gateway: Installation
Consul API Gateway: Listener Configuration
Consul API Gateway: Route Configuration
OpenShift: Understanding Ingress Operator
OpenShift: Route Configuration

Appendix

Frontend Service Definition:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: frontend
  namespace: consul
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: consul
spec:
  selector:
    app: frontend
  ports:
    - port: 9090
      targetPort: 9090
      protocol: TCP
      name: "9090"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: consul
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      name: frontend
      namespace: consul
      labels:
        app: frontend
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
        'consul.hashicorp.com/transparent-proxy': 'true'
        'consul.hashicorp.com/enable-metrics-merging': 'false'
        'consul.hashicorp.com/transparent-proxy-overwrite-probes': 'true'
        'k8s.v1.cni.cncf.io/networks': '[{ "name":"consul-cni" }]'
    spec:
      serviceAccountName: frontend
      containers:
        - name: frontend
          image: nicholasjackson/fake-service:v0.26.0
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: NAME
              value: "frontend-service"
            - name: MESSAGE
              value: "FRONTEND FRONTEND FRONTEND FRONTEND"
            - name: LOG_LEVEL
              value: trace
            - name: LISTEN_ADDR
              value: 127.0.0.1:9090
            - name: UPSTREAM_URIS
              value: "http://backend.$(POD_NAMESPACE).svc.cluster.local"
          ports:
            - containerPort: 9090
              name: http
              protocol: TCP

Consul Service Mesh Ingress on RedHat OpenShift

Introduction

Environment Assumptions

Article Example Resources:

Ingress Terms on OpenShift

Preparing for Consul API Gateway on Kubernetes

Configure API Gateway(s) and Application Listener(s)

Configure API Gateway Route

Configure OpenShift Ingress Object

Configure Consul Service Intention (API Gateway to Service Permission)

Verify External Connectivity

Summary

References:

Appendix

Frontend Service Definition:

Articles in this section

Introduction

Environment Assumptions

Article Example Resources:

Ingress Terms on OpenShift

Preparing for Consul API Gateway on Kubernetes

Configure API Gateway(s) and Application Listener(s)

Configure API Gateway Route

Configure OpenShift Ingress Object

Configure Consul Service Intention (API Gateway to Service Permission)

Verify External Connectivity

Summary

References:

Appendix

Frontend Service Definition:

Articles in this section

Related articles