Service Mesh Registration Time Tuning - Consul K8s Control Plane Resources

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

The Consul K8s control-plane default values.yaml resource defaults for connectInject Kubernetes components configures minimal resources for local testing and development.

Understanding that these initial default settings are not intended for production use at scale allows users and Kubernetes administrators to understand and set expectations for "out-of-the-box" behavior as it relates to configuring Consul's service mesh on Kubernetes.

Article Applicability

• Agent Based Consul K8s Control Plane Versions (Consul-k8s < v1.0.0)

TLDR;

• Increase consul-k8s connectInject.Resources when deploying in production to at least the values shown in Table 1-1 below for expected service registration deployments of ≤ 60.
• For service deployments with > 60 registrations, expect to increase this even further to facilitate faster registration times to the mesh.
• Ensure no additional influencing factors are contributing to slow registration times to include but not limited to:
  • Kubernetes Security Policy Scanning
  • Kubernetes Resource Quota Management Settings External to the Helm specific configurations.

Table 1-1: Production connectInject Resources

connectInject:
  resources:
    requests:
      memory: "50Mi"
      cpu: "50m" 
    limits:
      memory: "500Mi"
      cpu: "250m"
  sidecarProxy:
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 100Mi
        cpu: 100m
  initContainer:
    resources:
      requests:
        memory: "150Mi"
        cpu: "250m"
      limits:
        memory: "150Mi"
        cpu: 500m

Consul Kubernetes Resourcing - Full Discussion

Terms/Definitions

Some of the subjects in this article can be somewhat ambiguous and up for different interpretations. This section helps articulate what some of the below referenced terms mean.

Production Resource Value Increases

The discussion below helps guide Kubernetes administrators and Infrastructure engineers come to a reasonable conclusion on how to set their resource limits specifically on the connectInject related Consul K8s components.

Production Value Override Resource Increases

Reference Table 1-2 below when looking for production grade initial value settings to adjust your Consul Helm override values with respect to the connectInject resources.

connectInject.resources

This resource quota increase is necessary for ensuring you have an optimal starting point for Consul Connect service-mesh time to registration.

• This does not negate the required Kubernetes administration required in determining what resources best suite your use-case with respect to your environment.

• To avoid any undesired latency, it is highly recommended to adjust the connectInject.resources to at least those listed in Table 1-1.

Additional Considerations Deployments at Scale:

• If > 60 service applications are intended to be mass registered on the mesh at once, consider raising this resource limit beyond the recommended 250m CPU and 500Mi Memory values.
• If large service registrations to the mesh are expected, also look into service application interdependencies as they can also contribute to mesh registration time increases.
  • This falls into the category of service mesh application deployment planning. Some questions to ask:
    • Are your services being deployed to the mesh in a logical and controlled manner?
    • Could the deployment be optimized to reduce service interdependent induced health check failures?
• Investigate and correct any security scans that could play into Consul's init container or Connect Inject controller service registration process as security scans can also contribute to latency in registration.

sidecarProxy.resources

Increasing resources here ensures optimization of:

• Initial Service Registrations to Consul agent and catalog
• Periodic Service Synchronizations within Consul
• Metrics Merging Operations with respect to Consul's Envoy Sidecars.

initContainer.resources

The increase in these resources is to optimize time for initialization with regard to:

• Copying over the required consul and consul-k8s required binaries.
• Running Connect initialization prerequisite items required for service mesh communication.

Table 1-2: Consul K8s Control Plane Connect Inject Resourcing

# Consul Overrides - values.yaml
connectInject:
  resources:
    requests:
      memory: "50Mi"
      cpu: "50m" 
    limits:
      memory: "500Mi"
      cpu: "250m"
  sidecarProxy:
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 100Mi
        cpu: 100m
  initContainer:
    resources:
      requests:
        memory: "150Mi"
        cpu: "250m"
      limits:
        memory: "150Mi"
        cpu: 500m  

# Consul Overrides - values.yaml
connectInject:
  resources:
    requests:
      memory: "50Mi"
      cpu: "50m" 
    limits:
      memory: "50Mi"
      cpu: "50m"
  sidecarProxy:
    resources:
      requests:
        memory: null
        cpu: null
    limits:
      memory: null
        cpu: null
  initContainer:
    resources:
      requests:
        memory: "25Mi"
        cpu: "50m"
      limits:
        memory: "150Mi"
        cpu: null 

References:

• Consul on Kubernetes Agentless - link
• Consul K8s Default Overrides - link
• Kubernetes Pod Documentation - link
• Kubernetes Pod Readiness - link
• Google's Kubernetes Resourcing Best Practices - link