Introduction
This article demonstrates using a nomad attribute when using Vault to create a job certificate.
Problem
There may be particular use cases where injecting nomad attributes while creating certificates is needed; an example might be setting your nomad client IP as the IP SAN of the certificate.
Since Nomad collects client node information during fingerprinting, this information is available to you when using templates.
Using nomad attributes in your templates is straightforward; using attribute information when creating a Vault certificate requires a bit more effort.
For example, in the template block below, I'd like a certificate created for my job and I would like to use the Nomad client IP address for the IP SAN `ip_sans={{env attr.unique.network.ip-address}}`.
template {
data = "{{ with secret \"${var.vault_pki_path}\" \"common_name=${var.tls_domain}\" \"alt_names=${var.alt_names}\" \"ip_sans={{env attr.unique.network.ip-address}}\" }}{{ .Data.certificate }}\n{{ .Data.issuing_ca }}{{ end }}"
destination = "${NOMAD_SECRETS_DIR}/server.crt"
change_mode = "restart"
}
Unfortunately, when the job is run, the `{{env attr.unique.network.ip-address}}` is read literally and passed along to Vault, which, as you can see below, Vault is expecting IP format.
nomad[2791]: URL: PUT https://service.vault-nonprod.internal:8200/v1/pki_int/issue/nomad-cluster
nomad[2791]: Code: 400. Errors:
nomad[2791]: * the value '{{env attr.unique.network.ip-address}}' is not a valid IP address (exceeded maximum retries)
nomad[2791]: 2023-03-22T12:01:48.286Z [ERROR] agent: (runner) watcher reported error: vault.write(pki_int/issue/nomad-cluster -> 6089f095): vault.write(pki_int/issue/nomad-cluster -> 6089f095): Error making API request.
Solution
When using attributes with `secret` or `pkiCert` functions, you first need to store the attribute as a variable outside of the function; this can then be passed when calling `with secret` or `with pkiCert`.
Take the example jobspec below; we are again passing the Nomad client node IP as the IP SAN. However, we are first setting $VAR1 to `ip_sans=<NOMAD_CLIENT_IP`and then inserting $VAR1 into our certificate create call to Vault.
job "example" {
datacenters = ["dc1"]
group "cache" {
network {
port "db" {
to = 6379
}
}
task "redis" {
driver = "docker"
config {
image = "redis:7"
ports = ["db"]
auth_soft_fail = true
}
resources {
cpu = 500
memory = 256
}
vault {
policies = ["tls-policy"]
}
template {
data = <<EOH
{{- $VAR1 := (printf "ip_sans=%s" (env "attr.unique.network.ip-address")) -}}
{{- with secret "pki_int/issue/nomad-cluster" "common_name=${var.tls_domain}" "alt_names=${var.alt_names}" $VAR1 -}}
{{- .Data.certificate -}}
{{- printf "\n" -}}
{{- .Data.issuing_ca -}}
{{ end }}
EOH
destination = "local/bundle.pem"
change_mode = "restart"
}
}
}
}
variable "tls_domain" {
type = string
}
variable "alt_names" {
type = string
default = ""
}
Running the job now, we see that it is placed successfully, and no errors are seen.
root@nomad-server-1:/vagrant# nomad run -var="tls_domain=test4.global.nomad" -var="alt_names=client.global.nomad" example2.nomad
==> 2023-03-27T19:41:06Z: Monitoring evaluation "6810d403"
2023-03-27T19:41:06Z: Evaluation triggered by job "example"
2023-03-27T19:41:06Z: Allocation "7e7551a4" created: node "00b74485", group "cache"
2023-03-27T19:41:07Z: Evaluation within deployment: "1af7e392"
2023-03-27T19:41:07Z: Allocation "7e7551a4" status changed: "pending" -> "running" (Tasks are running)
2023-03-27T19:41:07Z: Evaluation status changed: "pending" -> "complete"
==> 2023-03-27T19:41:07Z: Evaluation "6810d403" finished with status "complete"
==> 2023-03-27T19:41:07Z: Monitoring deployment "1af7e392"
✓ Deployment "1af7e392" successful
2023-03-27T19:41:18Z
ID = 1af7e392
Job ID = example
Job Version = 0
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
cache 1 1 1 0 2023-03-27T19:51:17Z
Looking good so far, but how about the certificate? Was the client node IP used when creating the cert?
root@nomad-client-1:/home/vagrant# ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.45.140 netmask 255.255.255.0 broadcast 172.16.45.255
inet6 fe80::20c:29ff:fe0b:53f9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:0b:53:f9 txqueuelen 1000 (Ethernet)
RX packets 1215772 bytes 217948319 (217.9 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1217592 bytes 202373812 (202.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[12] → cat bundle.pem | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
37:52:42:7f:1c:e4:90:49:40:0b:fa:98:4d:34:f9:86:0f:a0:ee:e6
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=global.nomad Intermediate Authority
Validity
Not Before: Mar 27 19:40:37 2023 GMT
Not After : Mar 28 19:41:06 2023 GMT
Subject: CN=test4.global.nomad
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:b4:05:bb:04:5d:56:d3:ec:c9:50:62:ce:4a:2e:
b6:9c:97:8b:39:04:02:30:7d:c9:5d:65:49:f8:c9:
3c:af:71:b8:d5:9e:c2:8b:10:6f:27:0f:fe:2e:2b:
7a:c6:3f:7c:21:eb:4a:62:c2:4c:db:47:47:46:2c:
f3:1c:fc:82:6c:c2:4a:28:3e:3f:fa:f0:f3:0a:62:
c5:fa:81:a9:fc:79:1e:26:6a:fa:1c:b0:85:cb:d2:
01:5c:19:6d:2d:68:2a:65:25:95:a6:d3:64:62:7b:
a0:58:01:43:f4:60:bc:49:c2:07:42:18:c4:e6:56:
19:79:4d:29:17:55:b8:a2:c5:fb:b2:1f:ce:ff:28:
f8:04:69:47:c8:5c:40:bf:10:d6:24:be:f5:04:4b:
8e:4a:2a:17:7a:d4:50:a4:c6:82:9e:e1:91:4d:e3:
8e:b9:19:9d:f6:81:17:e5:d4:99:05:f4:a6:3f:c9:
49:cb:2e:e4:2a:a6:af:6a:4e:ea:f5:cc:4a:a3:85:
fc:02:2f:2f:ff:4c:02:42:9b:52:ad:10:00:33:56:
c4:0b:2e:e9:42:a0:db:d2:35:db:e6:cb:19:64:6c:
03:07:b2:24:78:14:72:7a:b3:0d:c8:6a:de:fd:c7:
f8:4c:31:a4:a0:f5:e0:16:a5:6a:2a:4a:53:3b:fa:
20:89
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Key Agreement
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
43:55:07:C6:F3:6B:4E:DA:1B:DF:1E:61:53:C0:5D:63:09:AF:1B:ED
X509v3 Authority Key Identifier:
keyid:76:65:6F:21:1E:ED:AB:D7:AE:CD:3C:5B:9E:3E:35:DE:B5:9C:66:7C
X509v3 Subject Alternative Name:
DNS:client.global.nomad, DNS:test4.global.nomad, IP Address:172.16.45.140
Signature Algorithm: sha256WithRSAEncryption
a0:ec:02:ce:e2:35:d2:5b:44:9e:12:ca:22:d9:88:6b:1f:31:
88:4b:60:c9:d8:fe:e1:14:59:08:c6:52:93:d3:b1:c8:02:13:
25:8b:38:d2:d4:bd:70:4b:e0:9c:c0:93:bc:7c:58:60:47:7a:
0c:00:4e:7f:07:b5:3e:78:46:52:72:ec:6c:01:95:a7:a3:cd:
10:a8:ad:0c:34:a9:a6:b8:44:52:6e:9e:58:13:f6:6c:c6:45:
f9:80:d4:5a:df:e9:aa:25:3b:fd:3c:9a:37:41:6b:e2:a1:75:
c3:3d:59:6e:02:ed:44:74:eb:33:b2:25:15:1b:1d:d8:f1:6d:
b2:1b:9a:10:ae:37:13:8b:86:10:b9:a4:74:1d:c4:34:2c:86:
1a:f1:60:c8:67:c6:88:ca:9b:58:e0:0e:00:d0:ad:0a:89:3f:
0c:68:60:23:9f:aa:9f:c2:c0:29:31:d6:2b:5f:21:da:19:ec:
38:44:ec:78:c2:9a:25:17:dc:dc:d2:ea:0b:48:c8:29:0d:74:
f4:cf:d1:73:47:ef:79:65:f5:45:18:67:34:2e:a3:a2:95:f0:
f2:ea:93:d3:f6:a2:7a:42:f5:a5:96:5d:ce:07:a4:32:da:eb:
83:9b:4a:d3:f2:21:3a:52:ff:0c:f3:2a:ec:ab:b2:11:39:c9:
4e:98:39:f4
Earlier, I noted the pkiCert function; storing the attribute as a variable "$VAR1" is also needed. The only change is the function used to create the certificate with Vault.
template {
data = <<EOH
{{- $VAR1 := (printf "ip_sans=%s" (env "attr.unique.network.ip-address")) -}}
{{- with pkiCert "pki_int/issue/nomad-cluster" "common_name=${var.tls_domain}" "alt_names=${var.alt_names}" $VAR1 -}}
{{- .Cert -}}
{{- printf "\n" -}}
{{- .CA -}}
{{ end }}
EOH
destination = "local/bundle2.pem"
change_mode = "restart"
}
Additional Information
-
Consul Template Secret function
- Consul Template pkiCert function
- Integrate Nomad with Vault