Consul-k8s: Optimizing Terminating Gateway Performance with Increased Envoy Concurrency

Avatar
Nathan Mollica
  • Updated

Introduction

The Consul Terminating Gateway serves as an egress "edge proxy," responsible for the secure termination of mTLS traffic leaving a Consul service mesh. Given its function as the central point for all outgoing traffic, unlike distributed sidecars, optimizing concurrency is vital. This guide focuses on why and how increasing Envoy worker thread concurrency is essential for effectively scaling egress traffic in an mTLS-enabled cluster.

 

Expected Outcome

For Consul on Kubernetes users deploying Consul's terminating gateway as the primary egress proxy for their service mesh, properly configuring Envoy worker thread concurrency will yield significant benefits, particularly as the service mesh scales to production levels. By implementing the recommendations outlined in this guide, users can expect:

  • Improved Egress Traffic Handling: The terminating gateway will be better equipped to manage a higher volume of outbound connections and requests, preventing bottlenecks and ensuring reliable egress traffic flow.
  • Enhanced Performance and Throughput: Increasing worker thread concurrency allows Envoy to process more network events in parallel, leading to lower latency and higher throughput for egress traffic.
  • Optimized Resource Utilization: By aligning the number of worker threads with the demands of egress traffic, the terminating gateway can more efficiently utilize available CPU resources, avoiding underutilization or unnecessary resource contention.
  • Increased Scalability: As the number of services within the Consul mesh grows, the properly configured terminating gateway will be able to handle the increased egress traffic demands without significant performance degradation.
  • Greater Stability and Reliability: Efficient handling of egress traffic contributes to the overall stability and reliability of the service mesh, ensuring consistent communication with external services.

In essence, correctly configuring Envoy worker thread concurrency for Consul terminating gateways in Kubernetes environments is a crucial step towards building a robust, scalable, and performant service mesh capable of handling production-level workloads.

 

Use Case

Envoy Concurrency

This setting controls the number of worker threads the Envoy proxy will employ. If left unspecified, Envoy automatically scales to the number of hardware threads available on the machine (one thread per CPU core per Envoy proxy instance). While setting this to zero forces Envoy to run with a single worker thread, this practice can lead to unnecessary overhead in terms of thread and memory usage, as well as the persistence of idle connections. Therefore, carefully sizing this value is advisable, keeping the following in mind:

  • Low for sidecars (Dataplane Sidecars)
  • High for edge proxies (Terminating Gateways/API Gateways)

Procedure

Consul Helm Chart 

Currently, the Consul value overrides do not support configuring Envoy concurrency settings for anything outside of the consul-dataplane sidecar proxies via the connectInject.sidecarProxy.concurrencyoverride, which defaults to a value of 2:

connectInject:
  enabled: true
  sidecarProxy:
    concurrency: 2

 

This default value will get applied to all Envoy-related components within a consul-k8s cluster, including Terminating Gateways.

Configuring Terminating Gateway Envoy Concurrency

To configure the concurrency for the terminating-gateways on a Kubernetes or OpenShift cluster, users can patch their terminating-gateway deployments using kubectl patch via CLI or, if using Kustomize, apply the following overlay patch:

 

kubectl patch

kubectl patch \
    deployment \
    consul-terminating-gateway \
    --namespace consul \
    --type='json' \
    --patch '[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "-envoy-concurrency=4"}]'

 

Kustomize Overlay Patch

patches:
  - target:
      version: apps/v1
      kind: Deployment
      name: consul-terminating-gateway
      namespace: consul
    patch: |
      - op: add
      path: /spec/template/spec/containers/0/args/-
      value: "-envoy-concurrency=4"

 

Kubernetes Resources and Envoy Concurrency 

In order for Envoy to leverage the concurrency configuration set with -envoy-concurrency, users will have to configure their terminating-gateway resource CPU requests and limits set accordingly. For example if configuring a concurrency of 4, the terminating-gateway pods would require a that the available CPUs allocated to the container are at least 4.

Envoy Concurrency: "-envoy-concurrency=4"

Terminating Gateway Resourcing:

terminatingGateways:
  enabled: true
  logLevel: trace
  defaults:
    replicas: 1
    consulNamespace: default
    resources:
      requests:
      memory: '200Mi'
      cpu: '4'
    limits:
      memory: '200Mi'
      cpu: '4'

 

Diagnosing

Terminating Gateway Envoy Stats (Buffer Overload)

To confirm a buffer overload on your Consul Terminating Gateway (TGW) using Envoy, especially when deployed withconsul-k8s, you can monitor and trend specific Envoy statistics that provide insight into buffer and resource usage. Here are some key Envoy stats to observe:

  1. Downstream Buffer Statistics:

    • http.downstream_cx_overflow: Tracks the number of times Envoy’s connection buffers have been overloaded. This is a direct indicator of buffer overload on the downstream (incoming) side.
    • http.downstream_rq_overload: Counts how many requests were overloaded due to buffer constraints.

  2. Upstream Buffer Statistics:

    • cluster.cluster_name.upstream_cx_tx_overflow: If the cluster is not able to transmit data quickly enough, leading to an overflow, this stat will increment.
    • cluster.cluster_name.upstream_cx_rx_overflow: Similar to tx_overflow but for receiving data upstream. It indicates that incoming data from the upstream cluster is being throttled due to buffer constraints.

  3. Downstream/Upstream Connection Duration:

    • downstream_cx_length_ms and upstream_cx_length_ms: These stats provide the duration of downstream and upstream connections in milliseconds. High connection durations combined with other overload stats can indicate performance bottlenecks due to buffer overload.

  4. HTTP/HTTP2 Specific Stats:

    • http2.outbound_flood and http2.inbound_flood: These stats track flooding incidents in the HTTP/2 protocol, which can sometimes contribute to buffer overload.
    • http.downstream_flow_control_paused_reading_total and http.downstream_flow_control_resumed_reading_total: Track how often flow control pauses/resumes. If these values are high, it can indicate that Envoy is frequently pausing reading from downstream clients due to buffer constraints.

  5. Resource Saturation Stats:

    • server.memory_allocated: Tracks the amount of memory Envoy has allocated. Rapid increases may point to a buffer-related issue.
    • server.total_connections: Monitoring the total number of connections can provide insight into when overload conditions occur, especially during spikes.

  6. Envoy Buffer Overflow Alerts:

    • listener.listener_name.downstream_cx_overload_reject: Indicates how often the listener has rejected connections due to overload conditions.

By trending these stats over time, especially during periods of high load, you can identify whether buffer overload is occurring in the TGW pod. Envoy's concurrency settings and proper CPU/worker thread allocation in OpenShift can help mitigate these buffer constraints.

 

TCP Dump Indications of Terminating Gateway Buffer Overflow

As the number of external service-mesh services increases, users may experience transient network communication resets during normal operation due to the Terminating Gateway pod's restriction on CPU concurrency for egress traffic.

  1. TCP Retransmissions

    Buffer overloads can lead to delayed acknowledgments (ACKs) or unacknowledged segments, which will eventually cause TCP retransmissions.

    • Indications: Look for TCP retransmissions, which appear as duplicated packets with the same sequence number. You’ll see lines like TCP Retransmission or TCP Dup ACK in the tcpdump output.
    • Command:
      tcpdump -i <interface> tcp and 'tcp[13] == 0x10'
      This command filters for ACK flags in the TCP header and can help identify delayed ACKs leading to retransmissions.
  2. Zero Window Conditions

    A zero window condition occurs when the receive buffer is full, and the sender is told to stop sending data.

    • Indications: Look for TCP Zero Window packets, where the receiving side is indicating it has no buffer space available. These appear in tcpdump messages like TCP ZeroWindow.
    • Command:
      tcpdump -i <interface> 'tcp[14:2] == 0x0000'
      This command captures TCP packets where the window size is zero, indicating buffer exhaustion.
  3. Out-of-Order Packets

    Buffer overloads may cause packets to be delivered out of order, especially if the overloaded gateway cannot process traffic quickly enough.

    • Indications: TCP Out-of-order messages indicate that packets are being processed in a sequence different from their original transmission order.
    • Command:
      tcpdump -i <interface> 'tcp[13] == 0x40'
      This filters for packets marked with the push flag (PSH), which can help detect buffer issues causing out-of-order delivery.
  4. High Latency in Packet Round Trips

    Buffer overload can cause increased latency in network round-trip times (RTTs).

    • Indications: Look for abnormally long time intervals between packets, particularly between SYN, SYN-ACK, and ACK in the TCP handshake. A delay in ACK responses can be a sign of buffer issues.
    • Command:
      tcpdump -i <interface> -tttt -nn 'tcp[tcpflags] & tcp-ack != 0'
      This will show timestamps for TCP ACK packets, allowing you to manually check the time intervals for signs of latency.
  5. Packet Loss

    Dropped packets are another possible consequence of buffer overload, and this can lead to frequent retransmissions and connection resets.

    • Indications: Missing sequence numbers or TCP Retransmissions are indicators of packet loss. Look for TCP RST (reset) packets as well, which may occur when the connection times out due to packet loss.
    • Command:
      tcpdump -i <interface> 'tcp[tcpflags] & (tcp-rst) != 0'
      This command captures reset (RST) packets, which can indicate connection drops due to overload.
  6. Flow Control and Slow ACKs

    Flow control mechanisms, like TCP sliding windows, may be in place, but slow acknowledgments can indicate buffer congestion.

    • Indications: Look for delayed ACKs where a packet is acknowledged much later than expected, which could indicate that the system is struggling to keep up with buffer processing.
    • Command:
      tcpdump -i <interface> 'tcp[tcpflags] & (tcp-ack) != 0'
      Use this to observe the timing of ACK responses and see if there are significant delays.
  7. Large Number of FIN or RST Flags

    If the buffer overload leads to dropped connections, you may notice a large number of FIN (finish) or RST (reset) flags, which are indicative of connection terminations.

    • Indications: Excessive TCP FIN or TCP RST packets could indicate that the pod is closing connections due to resource constraints.
    • Command:
      tcpdump -i <interface> 'tcp[tcpflags] & (tcp-fin|tcp-rst) != 0'
  8. Large Intervals Between Packet Exchanges

    When buffers are full, Envoy or the TGW may stall, leading to long pauses in packet exchanges. You can detect these stalls by monitoring the intervals between packets.

    • Indications: Long gaps between requests and responses or between multiple packets of a single TCP stream can be an indicator of a buffering issue.
    • Command:
      tcpdump -i <interface> -tttt
      Use the -tttt option to print timestamps for each packet, making it easier to see if there are long pauses in packet transmission.

By examining these indicators during a tcpdump capture, you can gather evidence of a buffer overload situation on your Consul Terminating Gateway pod. Combining this with the relevant Envoy stats will provide a clear picture of the overload condition.

 

Additional Information

Articles in this section