Introduction
When deploying workloads in Kubernetes with Consul service mesh, the Consul connect-injector seamlessly integrates sidecar proxies to intercept ingress and egress traffic. By default, both Kubernetes containers and the Consul sidecar proxy have a termination grace period of 30 seconds. However, certain workloads require an extended termination period, making it necessary to align the Consul sidecar’s shutdown behavior with the application's termination lifecycle.
If the Consul-Dataplane sidecar defaultShutdownGracePeriodSeconds is set to match the pod’s terminationGracePeriodSeconds, then the pod will always wait for the full duration before terminating. In cases where workloads require long termination periods (e.g., 3 hours), this can lead to excessive delays, complicating node maintenance and autoscaling operations.
Expected Outcome
The goal is to enable a more flexible termination process without statically increasing the shutdown time of pods. The solution allows applications to notify Consul when they have completed their shutdown process, prompting the sidecar to terminate immediately rather than waiting for the full grace period.
Use Case
A workload with an extended termination grace period (e.g., 3 hours) should be able to exit promptly once its shutdown tasks are complete, without forcing the Consul-Dataplane sidecar to delay termination unnecessarily. This approach is especially useful in scenarios where termination times vary significantly between instances.
Procedure
- Ensure your application properly drains connections before initiating termination.
- Once the application is ready to exit, send an HTTP POST request to the Envoy admin interface:
curl -X POST http://127.0.0.1:19000/quitquitquit
- This request signals Envoy to shut down, prompting Consul-Dataplane to exit gracefully.
Additional Information
- Pod Lifecycle: Termination of Pods (Kubernetes page)
- Helm Chart Reference: connectInject lifecycle
- Container Lifecycle Hooks (Kubernetes page)
- Connect-Inject Graceful Termination Period