The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

This guide provides instructions on customizing request timeouts within Consul's Service Mesh, specifically tailored for service sidecars. It encompasses modifying timeouts from the default 15 seconds at Envoy's local app level to accommodate services that necessitate additional processing time, effectively mitigating 504 Gateway Timeout errors. The procedure entails updating the service-default or proxy-default with adjusted timeout settings and is indispensable for operators managing services with prolonged processing requirements. This article complements the content on Configure the Request Timeout at the Service Level and can be used in conjunction with it for comprehensive timeout management within the Service Mesh architecture.

Expected Outcome

The service default local_request_timeout will override the default local app timeout, allowing the default 15 seconds to be modified. The local app request timeout is different from the service level request timeout, which can be configured by the service router or service resolver addressed in the article Configure the Request Timeout at the Service Level.

 Prerequisites

• service default file
• proxy default file
• L7 feature enabled(protocol=http)

Use Case

When receiving a 504 upstream request timeout error, the request timeout may need to be tuned in two places: the service level(service router or service resolver), and the local app level. If the timeout is tuned at the service level, and not the local app level, you'll still encounter a 504 since the local app timeout of 15s has to be overridden/tuned. The service default provides the local_request_timeout parameter to configure the local app timeout in Envoy sidecars, allowing for the adjustment of this parameter to either decrease or increase the duration of upstream requests, thereby enabling them to be prolonged or shortened as needed.

Procedure

Override local app request timeout by upstream

There are use cases where you may want specific upstreams to have a timeout different from other upstream services. You can do this by configuring the following:

1. In either yaml, hcl, or json, create the service default config. Take note the unit of time is in milliseconds.

#hcl
Kind = "service-default"
Name = "counting"  #configuring for upstream service
LocalRequestTimeoutMs = 30000 #30s

• Alternatively, you can add the local_request_timeout_ms in the proxy config section when registering the upstream service like the following:

#hcl
service {
name = "counting"
id = "counting-1"
port = 9003

connect {
  sidecar_service {
    proxy {

    config {
      local_request_timeout_ms = 30000 


   }
  }
 }
}

check {
  id = "counting-check"
  http = "http://localhost:9003/health"
  method = "GET"
  interval = "10s"
  timeout = "1s"
 }
}

Once applied, you can see the local app default timeout being overridden in the upstream Envoy sidecar(found at localhost:19000/config_dump)

    {
         "name": "envoy.filters.network.http_connection_manager",
         "typed_config": {
           "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
           "stat_prefix": "public_listener",
           "route_config": {
            "name": "public_listener",
            "virtual_hosts": [
             {
              "name": "public_listener",
              "domains": [
               "*"
              ],
              "routes": [
               {
                "match": {
                 "prefix": "/"
                 },
                 "route": {
                  "cluster": "local_app",
                  "timeout": "30s" #defaults to 15s
                 }

Override local app request timeout globally

If granular configuration for request timeouts isn't necessary, setting the local app request timeout globally is possible through the proxy default:

Procedure

1. In either yaml, hcl, or json, create the proxy default config. Take note the unit of time is in milliseconds.

#hcl
Kind = "proxy-defaults"
Name = "global"
Config {
  protocol = "http" # needed for L7 functionality
  local_request_timeout_ms = 30000
}

This will have the same effect in the Envoy sidecar configuration and logs, but now all services that do not have their own service default will use the proxy default request timeout for the local app instance. 

Additional Information

• • If you encounter issues where the request timeout persists at the default 15 seconds, you can investigate the Envoy logs to verify if the decoded x-envoy-expected-rq-timeout-ms header displays 15000 milliseconds.

    • The decoded header can be seen under "router decoding headers:" when Envoy logs are in debug mode

      • Please reference Accessing and Setting Envoy Logs for Consul for more information

    • Failing to override the default timeout implies that the service-level request timeout remains active, suggesting a potential misconfiguration in your proxy default or proxy configuration settings