Changes to gRPC TLS configuration at 1.14.x, causing "connection refused" errors and confusion among users

Avatar
Georgi Tsvetkov
  • Updated

Introduction

Problem

  •  Once customers that are using auto_encrypt (enableAutoEncrypt) attempt to upgrade to <= 1.14.x, without properly modifying grpc and grpc_tls, they are facing "transport: Error while dialing dial tcp x.x.x.x:8502: connect: connection refused" errors and inability to connect clients to servers.
  • Looking at documentation, it can be difficult to differentiate which port should be set for grpc and grpc_tls in both Consul VM and Consul-k8s environments.

Prerequisites:

Cause

  • Once upgrade is done and the grpc and grpc_tls ports are not being matched correctly, we will end up with servers that are not listening on the correct gRPC ports. We will observer errors like: "transport: Error while dialing dial tcp x.x.x.x:8502: connect: connection refused" and cluster instability.

  • grpc and grpc_tls ports could be mixed unintentionally and this can cause problems in environments where Consul server is outside Kubernetes cluster, since consul-k8s defaults to 8502, as noted in the Helm chart for both client and server settings, as well as in the changelog. It is recommended to 8503 to be used for grpc_tls at the Consul server side 

  • Note that Consul-k8s does not support mTLS (for now within 1.14.x)

Solutions:

  • In the Consul server config (VM), you need to to have verify_incoming = false for grpc, so that traffic coming from consul-k8s client cluster to Consul VM server won't be mTLS checked, hence it would be allowed.

  • In the Consul server config (VM), you need to to have grpcand grpc_tls ports defined (grpc = 8502and grpc_tls = 8503).

  • In Consul-k8s side, you need to have externalServers.grpcPort:8503 defined in your Helm chart.

  • It is better to use 8503 for grpc_tls to follow the consul convention, as traditionally 8502 was used for grpc, where 8502 used to inherit its TLS settings from the HTTPS settings.

Outcome

consul_server_config.hcl

server    = true
bootstrap = true
log_level = "debug"

ui_config {
  enabled = true
}

datacenter     = "dc1"
node_name      = "server-1"
bind_addr      = "192.168.64.1"
client_addr    = "0.0.0.0"
data_dir       = "./data"

tls {
  defaults {
    ca_file   = "consul-agent-ca.pem"
    cert_file = "dc1-server-consul-0.pem"
    key_file  = "dc1-server-consul-0-key.pem"
    verify_incoming = true
    verify_outgoing = true
  }
  https {
    verify_incoming = false
  }
  grpc {
    verify_incoming = false
  }
}

auto_encrypt {
  allow_tls = true
}

ports {
  https    = 8501
  grpc     = 8502
  grpc_tls = 8503
}

connect {
enabled = true
}

acl {
  enabled = true

  tokens {
    master = "root"
    agent = "root"
  }
}

values.yaml

global:
  enabled: false
  adminPartitions:
    enabled: true
    name: testis
  enableConsulNamespaces: true

  image: "hashicorp/consul-enterprise:1.14.4-ent"
  enterpriseLicense:
    secretName: consul-license
    secretKey: license
    enableLicenseAutoLoad: true

  logLevel: "debug"
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: bootstrap-token
      secretKey: token
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt

client:
  enabled: true
  join: ["10.16.64.20"]
  exposeGossipPorts: true

externalServers:
  enabled: true
  grpcPort: 8503
  hosts: ["10.16.64.20"]
  k8sAuthMethodHost: "https://10.16.64.8:6443"

connectInject:
  enabled: true

controller:
  enabled: true

Additional Information

  • grpc and grpc_tls, summarised details from our documentation:
     
    • grpc in consul-k8s defaults to 8502, as noted in the helm chart for both client and server settings, as well as per the changelog  

    • Additional details in regards to grpc vs grpc_tls:

      grpc - The gRPC API, -1 to disable. Default -1 (disabled). We recommend using 8502 for grpc as your conventional gRPC port number, as it allows some tools to work automatically. This parameter is set to 8502 by default when the agent runs in -dev mode. The grpc port only supports plaintext traffic starting in Consul 1.14. Refer to grpc_tls for more information on configuring a TLS-enabled port.

      grpc_tls - The gRPC API with TLS connections, -1 to disable. gRPC_TLS is enabled by default on port 8503 for Consul servers. We recommend using 8503 for grpc_tls as your conventional gRPC port number, as it allows some tools to work automatically. grpc_tls is always guaranteed to be encrypted. Both grpc and grpc_tls can be configured at the same time, but they may not utilize the same port number. This field was added in Consul 1.14.

    • Changes to gRPC TLS configuration at 1.14.X:

      Consul 1.14 introduces ports.grpc_tls, a new configuration for encrypting communication over gRPC. The existing ports.grpc configuration no longer supports encryption. As of version 1.14, ports.grpc_tls is the only port that serves encrypted gRPC traffic. The default value for the gRPC TLS port is 8503 for Consul servers. To disable the gRPC TLS port, use value -1.

  • Consul Auto-Encrypt Tech-mech

Articles in this section

See more

Related articles