Configuring and Deploying a Custom VLAN CNI Plugin in Nomad – HashiCorp Help Center

Overview

This KB outlines the procedure to configure and validate a custom VLAN CNI plugin within Nomad. It also demonstrates how to deploy a Nomad job using the custom CNI network and serves as a reference for similar custom CNI setups.

Prerequisites

Docker installed and running. Refer this.
Nomad installed and running. Refer this.
A valid Nomad Enterprise license file (license.hclic) if using Nomad Enterprise.

1. Nomad Configuration File

Update the Nomad config:

sudo vi /etc/nomad.d/nomad.hcl

Example configuration:

datacenter = "dc1"
data_dir   = "/opt/nomad"

acl {
  enabled = true
}

server {
  license_path      = "/etc/nomad.d/license.hclic"
  enabled           = true
  bootstrap_expect  = 1
}

client {
  enabled         = true
  servers         = ["<PRIVATE_IP_OF_SERVER>"]
  cni_path        = "opt/cni/bin"
  cni_config_dir  = "opt/cni/config"
}

Save and exit.

Start Nomad:

sudo systemctl restart nomad
sudo systemctl status nomad

2. Install CNI Plugins

Nomad requires CNI plugins for advanced networking modes (including VLAN).

Run:

export ARCH_CNI=$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)
export CNI_PLUGIN_VERSION=v1.6.2

curl -L -o cni-plugins.tgz \
"https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGIN_VERSION}/cni-plugins-linux-${ARCH_CNI}-${CNI_PLUGIN_VERSION}.tgz" && \
sudo mkdir -p /opt/cni/bin && \
sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz

3. Create VLAN CNI Configuration

Create the CNI config directory:

sudo mkdir -p /opt/cni/config
cd /opt/cni/

Identify host interfaces:

ip link

Create the VLAN configuration file:

sudo vi /opt/cni/config/vlan.json

Example configuration (adjust master interface appropriately):

{
  "name": "mynet",
  "cniVersion": "0.3.1",
  "type": "vlan",
  "master": "ens5",
  "mtu": 1500,
  "vlanId": 5,
  "linkInContainer": false,
  "ipam": {
    "type": "host-local",
    "subnet": "10.1.1.0/24"
  },
  "dns": {
    "nameservers": ["10.1.1.1", "8.8.8.8"]
  }
}

4. Nomad Job Example

Create the job file:

vi job.hcl

Example job:

job "example" {

  group "group" {

    network {
      mode = "cni/mynet"
    }

    task "task" {
      driver = "docker"

      config {
        image   = "busybox:1"
        command = "httpd"
        args    = ["-vv", "-f", "-p", "8001", "-h", "/local"]
      }

      resources {
        cpu    = 100
        memory = 100
      }
    }
  }
}

Run the job:

nomad job run job.hcl

Check job status:

nomad job status example

5. References

https://developer.hashicorp.com/nomad/docs/networking/cni#background
https://developer.hashicorp.com/nomad/docs/job-networking/cni
https://developer.hashicorp.com/nomad/docs/job-specification/network#mode
https://github.com/containernetworking/plugins