How Nomad Manages ACL Tokens/Polices for Consul Service Mesh – HashiCorp Help Center

The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

Nomad's first-class integration with Consul allows operators to design jobs that natively leverage Consul Service Mesh. However, in Consul Clusters that are ACL-enabled, a few extra steps are required to verify that your Nomad servers and clients have Consul ACL tokens with sufficient privileges to create additional services for the required sidecar proxies.

This article will describe how Nomad requests Consul to inject ACL tokens when creating Service Mesh services in an ACL-enabled Consul cluster.

Prerequisites

A Consul cluster enabled with ACL and Service Mesh (Connect)
A Nomad Cluster
Knowledge of Nomad Jobs

ACL Policy Requirements for Nomad Agents

Follow the steps in Generate Consul ACL tokens for Nomad to assign ACL tokens to Nomad agents
- ACL policy of the Nomad server token
```
agent_prefix "" {
  policy = "read"
}  
node_prefix "" {
  policy = "read"
}  
service_prefix "" {
  policy = "write"
}  
acl = "write"
```
- acl:write is required for generating Service Identity (SI) tokens for Consul services created by Nomad jobs. service_prefix"":write is also required for Nomad servers to register themselves as a Consul service as shown below.
- ACL policy for the Nomad client token
```
agent_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "write"
}
# uncomment if using Consul KV with Consul Template
# key_prefix "" {
#   policy = "read"
# }
```
- service_prefix"":write is required for these purposes:

Update Nomad's Consul configuration

Open the Nomad configuration file on all of the nodes and add a consul stanza with your token.

consul {
  address = "127.0.0.1:8500"
  grpc_address = "127.0.0.1:8502"
  token = "<<your nomad agent token>>"
}

Note: Consul and Nomad client agents run alongside each other so 127.0.0.1 can be used as the IP address of Consul in the Nomad config file.

Note: An important requirement is that each Nomad agent talks to a unique Consul agent. Nomad agents should be configured to talk to Consul agents and not Consul servers. If you are observing flapping services, you may have multiple Nomad agents talking to the same Consul agent. Therefore avoid configuring Nomad to talk to Consul via DNS such as consul.service.consul.

Case Studies

Service Mesh Job

Create a Service Mesh Job
- Follow this tutorial to create a Service Mesh job for demonstration. Two Consul services count-dashboard and count-api will be created.
Verifying the Service Identity (SI) Tokens
- In the Consul UI, go to Tokens and two SI tokens are created by Nomad. Note that the ACL policy acl:write has been granted to the Nomad server ACL token.
  
  Note: Starting in Nomad v1.3.0, Consul Service Identity ACL tokens automatically generated by Nomad on behalf of Connect-enabled services are now created in Local rather than Global scope, and are no longer replicated globally.
- During the process, the Consul server receives the SI token request from the Nomad server. Consul will generate the SI token and pass it back to the Nomad server. This action is logged in the Consul server log.
```
[DEBUG] agent.http: Request finished: method=PUT url=/v1/acl/token from=<<Nomad Server IP>>:40156 latency=2.999165ms
```
- Lastly, the SI token will be sent to the Nomod client to finish the Nomad job process.
```
 [INFO] client.alloc_runner.task_runner.task_hook.consul_si_token: derived SI token: alloc_id=24baf063-5331-53fd-2307-3b63722ef17e task=connect-proxy-count-api task=connect-proxy-count-api si_task=count-api
```

Leaf Certificate Renewal

By default, the TTL of a leaf certificate is 72 hours. The Consul client launches the renewal before the expiry. With service_prefix"":write applied in the Nomad client ACL policy, the Consul agent in the same node will be able to renew the leaf certificates.
```
[DEBUG] agent.auto_config: handling a cache update event: correlation_id=leaf
[DEBUG] agent.auto_config: leaf certificate watch fired - updating TLS certificate
```

If for some reason the service_prefix"": write is removed or changed to read, leaf certificate renewal will fail due to the lack of write permission.

[ERROR] agent.client: RPC failed to server: method=ConnectCA.Sign server=172.31.8.188:8300 error="rpc error making call: Permission denied: token with AccessorID 'dd1ba98a-13e7-c6aa-5237-a9c8ff2921ac' lacks permission 'service:write' on \"count-api\""
[WARN] agent.cache: handling error in Cache.Notify: cache-type=connect-ca-leaf error="rpc error making call: Permission denied: token with AccessorID 'dd1ba98a-13e7-c6aa-5237-a9c8ff2921ac' lacks permission 'service:write' on \"count-api\"" index=1284135

Terminating Gateway

Create Terminating Gateway Service Mesh
- Before creating a Terminating Gateway service, launch an external service in Consul by following this tutorial.
- Use this Nomad job example to create a terminating gateway service mesh.
SI Token Policy
- According to the articles Terminating Gateway Token Requirements and ACL with Terminating Gateways, service:write permission for each service the terminating gateway forwards traffic for is required.
  - Such permission allows the terminating gateway being able to request/renew leaf certificates from the Consul CA for the services it represents.
- When inspecting the terminating gateway ACL Token requested by Nomad, it was created as a regular SI token.
```
AccessorID: 6d6c3f4a-bade-7c1c-df4e-c8fb1312d77f
SecretID: 2db3122f-cef2-d339-ebb8-3a49fb81e450
Description: _nomad_si [4d1f827b-a8b9-04cb-163c-2e6d198f4891] [58d6e623-93ab-2eb7-8adc-68fd59de9aee] [connect-terminating-counting-tgw]
Local: true
Service Identities:
  counting-tgw (Datacenters: all)
```
- The SI token contains the default ACL policies.
```
service "counting-tgw" {
  policy = "write"
}
service "counting-tgw-sidecar-proxy" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
} 
```
- And the write permission to the service it represents is not in place. The counting service is always reachable from the dashboard.

Update the Nomad client ACL policy by adding service"counting":read so that it removes the write permission to the counting service.

agent_prefix "" {
      policy = "read"
}
    node_prefix "" {
      policy = "read"
}
service_prefix "" {
  policy = "write"
}
service "counting" {
      policy = "read"
}

Warnings as below start to appear that the Consul agent is unable to renew the leaf certificate for counting service due to lack the write permission.

[WARN] agent.cache: handling error in Cache.Notify: cache-type=connect-ca-leaf error="rpc error making call: Permission denied: token with AccessorID 'dd1ba98a-13e7-c6aa-5237-a9c8ff2921ac' lacks permission 'service:write' on \"counting\"" index=1308629

The counting service is now unreachable from the dashboard service.

Conclusion

In this article, we went through how Nomad implements Consul Service Mesh toward ACL-enabled Consul clusters. By configuring acl:writein Nomad server ACL policy and service_prefix"":write in the Nomad client ACL policy, Nomad job specification authors gain the ability to leverage Nomad's Consul Service Mesh integration effortlessly. This empowers them to implement Consul services without the need for direct management of services' ACL tokens and TLS certificates.

Notably, the transparent nature of this integration ensures that security features within the Service mesh persist consistently, even as the application scales up or down, or undergoes rescheduling by Nomad.