TLS Verification Failure in Consul gRPC Streams Due to CA Certificate Mismatch

Avatar
Vaibhav Arora
  • Updated

Introduction

The issue occurs due to a certificate mismatch between HTTPS and gRPC communication in Consul.

  • The Consul client is registered with CA B on port 8501, while gRPC TLS is registered with CA A on port 8503.
  • When the customer runs consul connect envoy -sidecar-for <service>, the following sequence happens:
    1. Service details are fetched from the Consul client.
    2. Envoy bootstrap configuration is generated.
    3. Envoy proxy is launched and attempts to connect using gRPC TLS (configured with CA A).

This leads to a mismatch because Envoy is bootstrapped with the HTTPS certificate (CONSUL_CACERT) while gRPC uses certificates from tls.defaults (CONSUL_GRPC_CACERT), resulting in authentication failures and certificate verification errors.

Problem

TLS Authentication Failures in Consul due to Certificate Mismatch

When customers attempt to run consul connect envoy -sidecar-for <service>, Envoy fails to authenticate due to a mismatch in the certificates used for HTTPS and gRPC communication.

Below is an error of snippet:

Screenshot 2025-01-31 at 11.40.14 AM.png

Cause

Mismatch Between Certificates for HTTPS (CONSUL_CACERT) and gRPC (CONSUL_GRPC_CACERT):

  1. The Consul client is registered with CA B on port 8501 for HTTPS communication.
  2. gRPC TLS, however, is registered with a different certificate authority, CA A, on port 8503.
  3. Envoy is bootstrapped with the HTTPS certificate (CONSUL_CACERT) during initialization, but gRPC communication relies on the certificate configured in tls.defaults (CONSUL_GRPC_CACERT).
  4. This discrepancy causes certificate validation errors when Envoy tries to establish a connection with gRPC, leading to authentication failures.

This issue highlights the importance of consistent certificate configurations across all communication channels (HTTPS and gRPC) to avoid such conflicts.

Solution

  • Configuring the CONSUL_GRPC_CACERT Environment Variable

    Steps:

    • Set the CONSUL_GRPC_CACERT environment variable to reference the correct CA file (e.g., /etc/consul.d/certs/consul-agent-ca.pem).
    • This configuration ensures that Envoy uses the appropriate CA for gRPC communication, aligning with the tls.defaults settings to utilize the correct CA required for Envoy connections.
    • Please use these commands: export CONSUL_GRPC_CACERT=/etc/consul.d/certs/consul-agent-ca.pe

    Impact:

    • Resolves the certificate mismatch issue by ensuring the gRPC TLS certificate verification process aligns with the intended CA configuration.

Outcome

Confirmation of Problem Resolution:

  • Verify the logs after exporting the CONSUL_GRPC_CACERT variable to ensure no further certificate-related errors are present (e.g., "tls: bad certificate").
  • Confirm that Envoy successfully connects to gRPC without authentication failures.

If the Problem Persists:

  • Double-check the CONSUL_GRPC_CACERT environment variable to ensure it points to the correct CA file as specified in tls.defaults.
  • Validate the CA certificate being used for gRPC communication matches the CA trusted by the Consul client.
  • Restart the Consul agent and Envoy processes to apply the changes and reinitialize the connection.
  • Review any additional configuration discrepancies or logs for unresolved issues.

Additional Information

Envoy logs: The error will persist until the CONSUL_GRPC_CACERT variable is correctly applied.

[2025-01-28 09:46:19.831][907228][info][main] [source/server/server.cc:1001] starting main dispatch loop

[2025-01-28 09:47:01.056][907228][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:226] DeltaAggregatedResources gRPC config stream to local_agent closed since 41s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
[2025-01-28 09:47:12.404][907228][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:226] DeltaAggregatedResources gRPC config stream to local_agent closed since 52s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
[2025-01-28 09:47:33.676][907228][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:226] DeltaAggregatedResources gRPC config stream to local_agent closed since 73s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

After exporting the CONSUL_GRPC_CACERT variable, the logs no longer show any errors.

[2025-01-28 09:50:10.527][913980][info][main] [source/server/server.cc:430] initializing epoch 0 (base id=0, hot restart version=disabled)
[2025-01-28 09:50:10.528][913980][info][main] [source/server/server.cc:432] statically linked extensions:
[2025-01-28 09:50:10.528][913980][info][main] [source/server/server.cc:434] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2025-01-28 09:50:10.528][913980][info][main] [source/server/server.cc:434] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2025-01-28 09:50:10.528][913980][info][main] [source/server/server.cc:434] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2025-01-28 09:50:10.528][913980][info][main] [source/server/server.cc:434] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router

Reference documents:

 

 

 

 

 

Articles in this section

See more