How to Run HTTPS applications in Consul Service Mesh

Introduction

Consul, by default, supports applications of type HTTPS when the service protocol defined is of type TCP. Consul does not support running applications with HTTPS when the protocol type is HTTP.

Expected Outcome

This KB article will help you use the Advanced Configuration options provided by Consul to modify the Envoy Configurations to support applications running HTTPS. You can read more about this here

Prerequisites

• These steps only apply to Consul on VMs or Nomad where you have full control over the service registration and do not apply to Consul on Kubernetes.

Use Case

Let's take two example services, dashboard and countingthat runs with HTTPS enabled. Both have service-defaults with the protocol for each set as http instead of tcp (the default).

# file: counting-defaults.hcl

Kind = "service-defaults"
Name = "counting"
Protocol = "http"

# file: dashboard-defaults.hcl

Kind = "service-defaults"
Name = "dashboard"
Protocol = "http"

When you try to access the upstream service via the downstream sidecar listener, you will get errors similar to the ones below.

# Error when accessing using Curl
$ curl localhost:5000
Client sent an HTTP request to an HTTPS server.

You will also see the following error in your downstream proxy's logs (provided you are running envoy with log level set to debug).

[debug][http] [source/common/http/conn_manager_impl.cc:1452] [C197][S8406769818210979393] encoding headers via codec (end_stream=false):
':status', '400'
'x-envoy-upstream-service-time', '10'
'date', 'Mon, 20 Sep 2021 06:15:30 GMT'
'server', 'envoy'

Procedure

You will have to use the envoy_local_cluster_json advanced configuration option in Consul to get this working.

In your service definition file, add the JSON snippet under connect.sidecar_service.proxy.config. Please note that this customisation currently is only possible where you can control the service registration (VMs, Nomad etc.) since this requires specifying a service-specific port in the load assignment config. The transport_socket section in the below example is what makes the proxy make a TLS connection to the backend.

The below example uses an example service named counting:

service {
  name = "counting",
  id = "counting-local"
  port = 8080,
  connect {
    sidecar_service {
      proxy {
        config {

          envoy_local_cluster_json = <<EOL
              {
              "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
              "name": "local_app",
              "type": "STATIC",
              "connect_timeout": "5s",
              "load_assignment": {
                  "cluster_name": "local_app",
                  "endpoints": [
                  {
                      "lb_endpoints": [
                      {
                          "endpoint": {
                          "address": {
                              "socket_address": {
                              "address": "127.0.0.1",
                              "port_value": 8080
                              }
                          }
                          }
                      }
                      ]
                  }
                  ]
              },
              "transport_socket": {
                  "name": "tls",
                  "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
                  "common_tls_context": {}
                  }
              }
              }
          EOL
        }
      }
    }
  }
}

Once you update the service registration using consul service register <filename>, Envoy will reconfigure itself and will start talking to the backend application using HTTPS.