The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.
All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.
Introduction
The Consul K8s control-plane default values.yaml resource defaults for connectInject Kubernetes components configures minimal resources for local testing and development.
Understanding that these initial default settings are not intended for production use at scale allows users and Kubernetes administrators to understand and set expectations for "out-of-the-box" behavior as it relates to configuring Consul's service mesh on Kubernetes.
Article Applicability
- Agent Based Consul K8s Control Plane Versions (Consul-k8s < v1.0.0)
| Upgrade Recommendation |
| The Consul Kubernetes team at HashiCorp has diligently developed recent improvements in consul-k8s where consul client agents are eliminated in versions >= v1.0.0. The new model uses a lightweight consul-dataplane and offloads health checking and service location functions onto kubelets. Consider upgrading to this version to leverage its added benefits. |
TLDR;
- Increase consul-k8s
connectInject.Resourceswhen deploying in production to at least the values shown in Table 1-1 below for expected service registration deployments of ≤ 60. - For service deployments with > 60 registrations, expect to increase this even further to facilitate faster registration times to the mesh.
- Ensure no additional influencing factors are contributing to slow registration times to include but not limited to:
- Kubernetes Security Policy Scanning
- Kubernetes Resource Quota Management Settings External to the Helm specific configurations.
Table 1-1: Production connectInject Resources
connectInject:
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "500Mi"
cpu: "250m"
sidecarProxy:
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 100Mi
cpu: 100m
initContainer:
resources:
requests:
memory: "150Mi"
cpu: "250m"
limits:
memory: "150Mi"
cpu: 500m
Consul Kubernetes Resourcing - Full Discussion
Terms/Definitions
Some of the subjects in this article can be somewhat ambiguous and up for different interpretations. This section helps articulate what some of the below referenced terms mean.
| Service(s) |
A containerized application or service that receives direction from the Kubernetes API to be either deployed and/or configured for use. In Kubernetes, a Service is a method for exposing a network application that is running as one or more Pods in your cluster. |
| Deployment(s) |
An applied configuration or desired state to be managed and updated by the Kubernetes API for services to be exposed within the Kubernetes cluster. In Consul, a deployment is the start trigger for service mesh registration processes prior to services being authorized for mesh communication. |
| Service Registrations |
The number of replica-sets, replica-set scaling value, or number of services to be registered within a single Consul cluster's service mesh. |
| Service Registration Times |
Time from K8s deployment of the service to the time the service pod is considered ready by the K8s cluster. |
Production Resource Value Increases
The discussion below helps guide Kubernetes administrators and Infrastructure engineers come to a reasonable conclusion on how to set their resource limits specifically on the connectInject related Consul K8s components.
| IMPORTANT |
|
It is assumed that practitioners, on an individual use-case basis, incorporate Kubernetes resource stress testing as a routine part of administering their clusters. |
|
The dynamic nature of each use-case varies greatly, and baseline production resource recommendations outlined here are only to establish a starting point for a production environment. This article published by Google titled Kubernetes best practices: Resource requests and limits is a good starting point for any practitioners wondering where to start. |
Production Value Override Resource Increases
Reference Table 1-2 below when looking for production grade initial value settings to adjust your Consul Helm override values with respect to the connectInject resources.
connectInject.resources
This resource quota increase is necessary for ensuring you have an optimal starting point for Consul Connect service-mesh time to registration.
- This does not negate the required Kubernetes administration required in determining what resources best suite your use-case with respect to your environment.
- To avoid any undesired latency, it is highly recommended to adjust the
connectInject.resourcesto at least those listed inTable 1-1.
Additional Considerations Deployments at Scale:
- If > 60 service applications are intended to be mass registered on the mesh at once, consider raising this resource limit beyond the recommended
250mCPU and500MiMemory values. - If large service registrations to the mesh are expected, also look into service application interdependencies as they can also contribute to mesh registration time increases.
- This falls into the category of service mesh application deployment planning. Some questions to ask:
- Are your services being deployed to the mesh in a logical and controlled manner?
- Could the deployment be optimized to reduce service interdependent induced health check failures?
- This falls into the category of service mesh application deployment planning. Some questions to ask:
- Investigate and correct any security scans that could play into Consul's init container or Connect Inject controller service registration process as security scans can also contribute to latency in registration.
sidecarProxy.resources
Increasing resources here ensures optimization of:
- Initial Service Registrations to Consul agent and catalog
- Periodic Service Synchronizations within Consul
- Metrics Merging Operations with respect to Consul's Envoy Sidecars.
initContainer.resources
The increase in these resources is to optimize time for initialization with regard to:
- Copying over the required consul and consul-k8s required binaries.
- Running Connect initialization prerequisite items required for service mesh communication.
Table 1-2: Consul K8s Control Plane Connect Inject Resourcing
| Production Grade Env Resources | Dev/Testing Env Default Resources |
# Consul Overrides - values.yaml |
# Consul Overrides - values.yaml |