The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

This guide provides instructions on customizing request timeouts within Consul's Service Mesh, specifically tailored for services, whether they have an Ingress Gateway or not. It encompasses modifying timeouts from the default 15 seconds to accommodate services that necessitate additional processing time, effectively mitigating 504 Gateway Timeout errors. The procedure entails updating the service-router or service-resolver with adjusted timeout settings and is indispensable for operators managing services with prolonged processing requirements. This article complements the content on Override Envoy's local app request timeout and can be used in conjunction with it for comprehensive timeout management within the Service Mesh architecture.

Expected Outcome

The service router or service resolver request_timeout will override the default request timeout, allowing the default 15 seconds to be modified at the service level. 

Prerequisites

• Service router file
• Service resolver file
  • The RequestTimeout field for Service Resolver was backported to 1.15+
• L7 feature enabled
  • protocol=http

Use Case

When services in the mesh have longer request times, operators may encounter more 504 Gateway timeout errors even if the local app request timeout is overridden. This is due to the downstream needing more time to wait for the entire request to finish. By default the timeout is 15 seconds, but with the following steps, this can be decreased or increased allowing upstream requests to be prolonged.

Procedure

Tune Request Timeout

1. In either YAML, HCL, or JSON, create the Service router config

   • Note: the unit of time is in milliseconds

     #hcl
     Kind = "service-router"
     Name = "counting" # upstream name
     Routes = [
      {
        Match {
        HTTP {
          PathPrefix = "/counting"
       }
      }
     
     Destination {
       Service = "counting"
       RequestTimeout = 60000 # 60 seconds
      }
      },
     ]

      

2. The upstream name is used because it needs to match the route name referenced in the Envoy config dump. Alternatively, you can use the service resolver
   • Note: the RequestTimeout field for Service Resolver was backported to Consul 1.15+
     

     Kind = "service-resolver"
     Name = "counting" # upstream name
     RequestTimeout = "60s"
     
     

3. Once applied, you can see the request timeout updated in the downstream Envoy sidecar
   • Found at localhost:19000/config_dump
     

     {
          "version_info": "6d987dda470c60b9a167e4796eefc1eb88302b2d99aa1b16d781bb60ce296a87",
          "route_config": {
           "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
           "name": "counting",
           "virtual_hosts": [
            {
             "name": "counting",
             "domains": [
              "*"
             ],
             "routes": [
              {
               "match": {
                "prefix": "/counting"
               },
               "route": {
                "cluster": "counting.default.dc1.internal.af72f29d-8d5e-409c-db36-13193caeae84.consul",
                "timeout": "60s"
     }

Tune RequestTimeout if Ingress Gateway is configured

You have the option to fine-tune the request timeout for a service at the ingress gateway level, offering a strategic approach to prevent bottlenecks. This level of control allows for precise adjustments, particularly beneficial for data-intensive services requiring additional time to complete requests.

1. Create the Service router config
   

   #hcl
   Kind = "service-router"
   Name = "dashboard" # downstream name
   Routes = [
    {
      Match {
      HTTP {
        PathPrefix = "/counting"
    }
   }
   
   Destination {
     Service = "counting"
     RequestTimeout = 60000 # 60 seconds
     }
    },
   ]

    

2. The downstream name can now be used in the service router since the Ingress Gateway provides a route config for the service.
   

   // Envoy Config Dump from Ingress Gateway(Under route.v3.RouteConfiguration)
   
   {
   "name": "dashboard",
   "domains": [
     "dashboard.ingress.*",
     "dashboard.ingress.*:8080"
   ],
   "routes": [
   {
     "match": {
       "prefix": "/counting"
   },
   "route": {
     "cluster": "counting.default.dc1.internal.af72f29d-8d5e-409c-db36-13193caeae84.consul",
       "timeout": "60s"
    }
   },
   {
    "match": {
    "prefix": "/"
   },
   "route": {
     "cluster": "dashboard.default.dc1.internal.af72f29d-8d5e-409c-db36-13193caeae84.consul"
   }

    

Additional Information

• If you encounter issues where the request timeout persists at the default 15 seconds, you can investigate the Envoy logs to verify if the decoded x-envoy-expected-rq-timeout-ms header displays 15000 milliseconds.

  • The decoded header can be seen under "router decoding headers:" when Envoy logs are in debug mode

    • Please reference Accessing and Setting Envoy Logs for Consul for more information

  • Failing to override the default timeout implies that the service-level request timeout remains active, suggesting a potential misconfiguration in your proxy default or proxy configuration settings