Introduction

This article aims to address and troubleshoot common issues associated with configuration entries in Consul, particularly those managed through Kubernetes custom resource definitions (CRDs).

Table of Contents

• Getting Started
• Common Errors
  • Error: config entry already exists in Consul
  • Error: migration failed: Kubernetes resource does not match existing Consul config entry
  • Error: if kind is a CRD, it should be installed before calling Start OR Error: the server could not find the requested resource
• Next steps
• Additional Resources

Getting Started 

1. Identify the specific component utilized by the installed consul-k8s to manage configuration entries within the Kubernetes environment.
   • For consul-k8s <1.0.0
     • Configuration entry CRDs are managed with the controller Helm parameter.
   • For consul-k8s >= 1.0.0
     • Configuration entry CRDs are managed with the connectInject Helm parameter
2. Verify the below setting, based on the consul-k8s version, is enabled so that Consul CRDs will work.
   
   • consul-k8s <1.0.0
     

         controller:
             enabled: true

      

   • consul-k8s >= 1.0.0

       connectInject:
           enabled: true

      

3. In order to troubleshoot Consul CRD issues, access to the pod logs is required.

   • consul-k8s <1.0.0

     kubectl -n <namespace> logs consul-controller-<id>

      

   • consul-k8s >= 1.0.0
     

     kubectl -n <namespace> logs consul-connect-injector-<id>

      

Common Errors

Error: config entry already exists in Consul

Example

2023-08-23T05:46:03.407Z        ERROR   Reconciler error        {"controller": "proxydefaults", "controllerGroup": "consul.hashicorp.com", "controllerKind": "ProxyDefaults", "ProxyDefaults": {"name":"global","namespace":"default"}, "namespace": "default", "name": "global", "reconcileID": "569f5cda-5546-45a8-a8a3-f36694fcadf0", "error": "config entry already exists in Consul"}

• This error indicates that the config entry has been created/updated previously, outside of Kubernetes CRD.
• The error will also appear when running a describe command on the CRD
  • Example
    

    kubectl describe proxydefaults global
    
    Status:
      Conditions:
        Last Transition Time:  2023-08-23T05:48:09Z
        Message:               config entry already exists in Consul
        Reason:                ExternallyManagedConfigError
        Status:                False
        Type:                  Synced
      Last Synced Time:        2023-08-23T05:08:25Z

• Follow the instructions in the Migrating Config Entries document to migrate existing config entries to Kubernetes CRD.

IMPORTANT Once migrated, only apply configuration changes via Kubernetes CRD and not with Consul CLI/API to prevent the error from happening

Error: migration failed: Kubernetes resource does not match existing Consul config entry

Example

2023-08-23T06:07:46.988Z        ERROR   Reconciler error        {"controller": "proxydefaults", "controllerGroup": "consul.hashicorp.com", "controllerKind": "ProxyDefaults", "ProxyDefaults": {"name":"global","namespace":"default"}, "namespace": "default", "name": "global", "reconcileID": "84bdd1dd-8a20-40b7-9430-e10c91b79d50", "error": "migration failed: Kubernetes resource does not match existing Consul config entry: consul={\"Kind\":\"proxy-defaults\",\"Name\":\"global\",...

This error occurs when the configuration entry has different datacenter with the current datacenter, annotation 'consul.hashicorp.com/migrate-entry': 'true' is provided, and the configuration entry mismatches with current config in Consul. The solution is to make changes to the Kubernetes resource to match the Consul resource.

Error: if kind is a CRD, it should be installed before calling Start

OR

Error: the server could not find the requested resource

• If any of Consul's CRDs are deleted, the controller/connect-injector pod may contain these error messages:
  • controller
    

    2023-08-23T05:55:35.064Z        ERROR   controller-runtime.source       if kind is a CRD, it should be installed before calling Start   {"kind": "ProxyDefaults.consul.hashicorp.com", "error": "no matches for kind \"ProxyDefaults\" in version \"consul.hashicorp.com/v1alpha1\""}

  • connect-injector
    
    

    2023-08-23T05:54:17.951Z        ERROR   pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1alpha1.ProxyDefaults: failed to list *v1alpha1.ProxyDefaults: the server could not find the requested resource (get proxydefaults.consul.hashicorp.com)
    

     

• This is Kubernetes complaining that the required CRD does not exist.
• The solution is to run helm upgrade with the same values file for the required CRDs to be installed.

Next steps

If the issue persists after following the above setup and troubleshooting, please contact HashiCorp support and provide us with all of the troubleshooting data you have collected, as well as the following additional information:

• Consul Server version
• Consul Helm Chart version
• Consul Helm values file

Additional Resources

• Consul on Kubernetes
• Custom Resource Definitions (CRDs) for Consul on Kubernetes
• Manage Consul with Kubernetes custom resource definitions (CRDs)
• Migrating Config Entries