How to manage Consul Config Entry with Terraform

Avatar
Bruce Lok
  • Updated

Introduction

This article describes a workaround for resolving protocol mismatch errors in Consul when using Consul Terraform provider to manage config entries, such as deleting service-defaults or modifying the protocol for a downstream service referenced by the following config entries: 

  • service-router
  • service-resolver
  • ingress-gateway
  • service-splitter
  • api-gateway
  • terminating-gateway

 

Problem

From time to time, you may want to remove one or more service-defaults config entries using Terraform.  When running terraform apply after removing a service-defaults config entry resource from the Terraform code, the resource cannot be destroyed and you will see an error like this:

│ Error: failed to delete 'api-v1' config entry: Unexpected response code: 500 (discovery chain "api" uses inconsistent protocols; service "api-v1" has "tcp" which is not "http")

This happens because Consul's validation blocks the deletion to prevent a temporary protocol mismatch in the chain.

 

Cause

Consul validates config changes to ensure protocol consistency (eg: HTTP protocol throughout for path-based routing).  In Consul, services will be reverted to "tcp" protocol when their service-defaults are deleted.  If a service-router or other config entries still references/links the services with HTTP protocol, the deletion will fail.  That's why terraform apply will return error because the target Consul cluster doesn't allow such deletion. 

 

Solution

To fix the issue, update the service-router to remove the downstream service reference, so it can pass Consul's validation.  Below are the steps to achieve this using the Terraform CLI.

Assuming your Consul Terraform Provider codes have created the following three config entries as an example:

  • a service-defaults for a main service "api"

  • a service-defaults for a destination "api-v1"

  • a referencing service-router

The service-router resource may looks like this.  It configures HTTP path-based routing for "api" service, redirecting requests with a /v1 prefix to the downstream "api-v1" service.

resource "consul_config_entry" "service_router_api" {
  depends_on = [
    consul_config_entry.service_defaults_api,
    consul_config_entry.service_defaults_api_v1
  ]

  kind = "service-router"
  name = "api"

  config_json = jsonencode({
    Routes = [
      {
        Match = {
          HTTP = {
            PathPrefix = "/v1"
          }
        }
        Destination = {
          Service = "api-v1"
        }
      }
    ]
  })
}

 

Next, you want to remove the service_defaults_api_v1 resource block from your code (causing the failure), follow these steps to resolve:

 

1. Update the service_router_api resource to isolate the update.

  • Temporarily comment out or remove the depends_on block.

  • Update the config_json to remove the Routes reference associated to service "api-v1".

The updated service_router_api block may look like this:

resource "consul_config_entry" "service_router_api" {
  # depends_on = [ ... ]
  kind = "service-router"
  name = "api"
  config_json = jsonencode({
    Routes = []
  })
}

 

2.  Run a targeted apply to update only the service-router , it should only remove the reference in the Routes block without deleting any resource.  

$ terraform apply -target=consul_config_entry.service_router_api

 

3. Finally, run a full apply.  Since the service "api-v1" is removed from the Routes block, this apply should succeed to delete service_defaults_api_v1 as Consul no longer sees any protocol mismatch.

$ terraform apply

 

Outcome

The target service-defaults config entry can be removed through Terraform.

 

Reference:

 

Articles in this section

See more