Nomad creates a working directory for each allocation on a client. This directory can be found in the Nomad data_dir at ./alloc/«alloc_id». The allocation working directory (/alloc) is where Nomad creates task directories and directories shared between tasks, write logs for tasks, and downloads artifacts or templates. Each task has a task working directory with the same name as the task (/«taskname»). Tasks in a task group can't read each other's task working directory.
An allocation with two tasks (named task1 and task2) will have an allocation directory like the one below.
├── alloc
│ ├── data
│ ├── logs
│ └── tmp
├── task1
│ ├── local
│ ├── secrets
│ └── tmp
└── task2
├── local
├── secrets
└── tmp
When we use template stanza in our nomad job file to create file and inject values in it under /secret, sometimes secrets folder is completely empty in the running Docker container as well as in Nomad allocation CLI. Same file you can see in Allocations in Nomad UI but not in Docker container or Nomad allocation (CLI). Please see below template stanza and error snippet for same -
Template Stanza -
template {
data = <<EOH
This is a sample test data content.
EOH
destination = "${NOMAD_SECRETS_DIR}/file"
change_mode = "restart"
}
After deployment of Nomad job, your allocation may look like below -
Nomad Allocation UI - (Here file has been created)
Nomad Allocation CLI - (Here file is not present)
Docker Container - (Here also file is not present)
The above scenario can be due to bad configuration of your Nomad Linux services file (nomad.service) on Nomad Client Nodes. You need to use below for your nomad.service file on your Nomad Client Nodes -
[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target
[Service]
ExecReload=/bin/kill -s HUP
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d/ -bind=0.0.0.0
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity
[Install]
WantedBy=multi-user.targetBelow are the things which is important from above nomad.service file -
Wants=:This directive is similar to Requires=, but less strict. Systemd will attempt to start any units listed here when this unit is activated. If these units are not found or fail to start, the current unit will continue to function. This is the recommended way to configure most dependency relationships. Again, this implies a parallel activation unless modified by other directives.
After=: The units listed in this directive will be started before starting the current unit. This does not imply a dependency relationship and one must be established through the above directives if this is required.
network-online.target: as a value for After and Wants parameter for [Unit] stanza in nomad.service file. This target unit (network-online.target) is intended to pull in a service that delays further execution until the network is sufficiently set up. What precisely this requires is left to the implementation of the network managing service.
Note - After making changes in your nomad.service file as shown above, please do below -
-
Run below command to reload systemd manager configuration for nomad on the nomad client node where you updated your unix service file
sudo systemctl daemon-reload -
Restart docker services on nomad client nodes.
-
Restart the agents on nomad client nodes.
-
And check the secret folders in docker container as well as Nomad UI.
Reference Document -