Nomad: NFS CSI Volume

Avatar
Suyash Kumar
  • Updated

Introduction

The Container Storage Interface (CSI) plugin for Network File System (NFS) in HashiCorp Nomad enables the dynamic provisioning and management of NFS volumes for containerized workloads. This guide will walk you through the steps to configure the NFS CSI plugin in HashiCorp Nomad, allowing you to seamlessly integrate NFS storage into your containerized applications.

 

Prerequisites 

Before you begin configuring the NFS CSI plugin in HashiCorp Nomad, make sure you have the following prerequisites in place:

  1. Nomad Cluster: You should have a working Nomad cluster set up. If you haven't done this yet, follow the official Nomad documentation to set up a Nomad cluster.

  2. Nomad 1.3.0 or above.

  3. NFS Server: You need an NFS server that hosts the NFS shares you want to use as volumes in your workloads. Ensure that the NFS server is accessible from your Nomad clients and has the necessary shares configured.

  4. Enable privileged Docker jobs: CSI Node plugins must run as privileged Docker jobs because they use bidirectional mount propagation in order to mount disks to the underlying host.

    1. CSI Plugins running as node or monolith type requires root privileges (or CAP_SYS_ADMIN on Linux) to mount volumes on the host. With the Docker task driver, you can use the privileged = true configuration, but no other default task drivers currently have this option. Nomad’s default configuration does not allow privileged Docker jobs and must be edited to allow them.

    2. To enable, edit the configuration for all of your Nomad clients, and set allow_privileged to true inside of the Docker plugin’s configuration. Restart the Nomad client process to load this new configuration.

 

Deploy the NFS Plugin

Plugins for CSI are run as Nomad jobs with a plugin stanza. I am using mcr.microsoft.com/k8s/csi/nfs-csi docker image for this tutorial. It’s packaged as a Docker container that you can run with the Docker task driver.

Each CSI plugin supports one or more types: Controllers and Nodes. Node instances of a plugin need to run on every Nomad client node where you want to mount volumes. You'll probably want to run Node plugin instances as Nomad system jobs. Some plugins also require coordinating Controller instances that can run on any Nomad client node.

Create a file for the controller job called nfs-controller.nomad.hcl with the following content:

job "plugin-nfs-controller" {
  datacenters = ["dc1"]

  group "controller" {
    task "plugin" {
    driver = "docker"

    config {
      image = "mcr.microsoft.com/k8s/csi/nfs-csi:latest"
      args = [
        "--endpoint=unix://csi/csi.sock",
        "--nodeid=${attr.unique.hostname}",
        "--logtostderr",
        "-v=5",
      ]
    }

    csi_plugin {
      id = "nfs"
      type = "controller"
      mount_dir = "/csi"
    }

    resources {
      cpu = 250
      memory = 128
    }
  }
 }
}

Run the above job using the command - nomad job run nfs-controller.nomad.hcl

 

Create a file for the node job named nfs-nodes.nomad.hcl with the following content.

job "plugin-nfs-nodes" {
  datacenters = ["dc1"]

  type = "system"

  group "nodes" {
    task "plugin" {
    driver = "docker"
    config {
      image = "mcr.microsoft.com/k8s/csi/nfs-csi:latest"
      args = [
        "--endpoint=unix://csi/csi.sock",
        "--nodeid=${attr.unique.hostname}",
        "--logtostderr",
        "--v=5",
      ]
      privileged = true
    }

    csi_plugin {
      id = "nfs"
      type = "node"
      mount_dir = "/csi"
    }

    resources {
      cpu = 250
      memory = 128
    }
  }
 }
}

Run the above job using the command - nomad job run nfs-nodes.nomad.hcl

 

Validate the plugin status for NFS

It will take a few moments for the plugins to register themselves as healthy with Nomad after the job itself is running. You can check the plugin status with the nomad plugin status command. Below is the sample output - 

$ nomad plugin status
Container Storage Interface
ID   Provider        Controllers Healthy/Expected  Nodes Healthy/Expected
nfs  nfs.csi.k8s.io  1/1                           1/1

To check more in detail, you can use the command - nomad plugin status nfs . Below is the sample output - 

$ nomad plugin status nfs
ID                   = nfs
Provider             = nfs.csi.k8s.io
Version              = v4.5.0
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1

Allocations
ID       Node ID  Task Group  Version  Desired  Status   Created    Modified
a8d82cf9 f4ca0795 nodes       0        run      running  4m15s ago  4m13s ago
41d27e96 f4ca0795 controller  0        run      running  4m28s ago  4m16s ago

 

Register the volume for NFS CSI

Use the below file to register your NFS volume in Nomad -

$ cat nfs-volume.hcl
type = "csi"
id = "nfs"
name = "nfs"
plugin_id = "nfs"

capability {
  access_mode = "multi-node-multi-writer"
  attachment_mode = "file-system"
}

capability {
  access_mode = "single-node-writer"
  attachment_mode = "file-system"
}

context {
  server = "10.211.55.5"
  share = "/mnt/nfs_share/vault"
}

mount_options {
  fs_type = "nfs"
}

Please note - in the above hcl file for volume register, we need to mention the NFS server address and filesystem which is configured on all nomad client nodes as NFS file system. 

Now, run the command - nomad volume register nfs-volume.hcl to register the volume. You can validate the same using the command - nomad volume status nfs. You will get output something like the below - 

$ nomad volume status nfs
ID                   = nfs
Name                 = nfs
Namespace            = default
External ID          = <none>
Plugin ID            = nfs
Provider             = nfs.csi.k8s.io
Version              = v4.5.0
Schedulable          = true
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1
Access Mode          = <none>
Attachment Mode      = <none>
Mount Options        = fs_type: nfs
Namespace            = default

Allocations
No allocations placed

The volume status output above indicates that the volume is ready to be scheduled, but has no allocations currently using it.

 

Deploy Nomad Job using NFS Volume

Here, a job file is created that will deploy the vault docker image using certificates and a custom vault configuration file on the Nomad cluster using the NFS volume that we registered in the previous step. Please note - before deploying this job, we need to place the certificate file and key file for the vault server along with the vault configuration file inside the NFS file system. These will be used inside the job by referencing the NFS volume that we registered.

Create job file - 

Create a file called vault-server.nomad.hcl and provide it with the following contents.

job "vault-server" {
  datacenters = ["dc1"]

  group "vault-server" {
    volume "vault" {
      type = "csi"
      read_only = false
      source = "nfs"
      attachment_mode = "file-system"
      access_mode = "multi-node-multi-writer"
    }

    network {
      port "http" {
        to = 8200
        static = 9090
      }

      port "https" {
        to = 8201
        static = 9091
      }
    }

    task "vault-server" {
      volume_mount {
        volume = "vault"
        destination = "/vault/config.d"
        read_only = false
      }

      driver = "docker"

      config {
        image = "hashicorp/vault"
        privileged = "true"
        ports = ["http", "https"]

        command = "vault"
        args = ["server", "-config", "/vault/config.d/vault.hcl", "-client-key", "/vault/config.d/vault.key", "-client-cert", "/vault/config.d/vault-server.crt"]
      }

      env {
        VAULT_ADDR = "https://127.0.0.1:8201"
        VAULT_CLIENT_CERT = "/vault/config.d/vault-server.crt"
        VAULT_CLIENT_KEY = "/vault/config.d/vault.key"
        VAULT_CACERT = "/vault/config.d/vault-server.crt"
      }
    }
  }
}

 

Run the job

Register the job file we created in the previous step with the following command - nomad job run vault-server.nomad.hcl. We will get the output like below - 

==> 2023-11-18T14:16:27+05:30: Monitoring evaluation "5709a3df"
2023-11-18T14:16:27+05:30: Evaluation triggered by job "vault-server"
2023-11-18T14:16:28+05:30: Evaluation within deployment: "187d9a9c"
2023-11-18T14:16:28+05:30: Allocation "9ab25570" created: node "f4ca0795", group "vault-server"
2023-11-18T14:16:28+05:30: Evaluation status changed: "pending" -> "complete"
==> 2023-11-18T14:16:28+05:30: Evaluation "5709a3df" finished with status "complete"

The allocation status will have a section for the CSI volume, and the volume status will show the allocation claiming the volume. You need to run the command nomad alloc status <allocation_id> and search for the "CSI Volumes" section in the same.

CSI Volumes:
ID      Read Only
vault   false

 

Validate the volume

Run the nomad volume status nfs command to validate whether the Nomad jobs's allocation is showing or not. If the allocation is successfully running and using the nfs volume, you will see the allocation ID under the Allocation section, something like below - 

$ nomad volume status nfs
ID                   = nfs
Name                 = nfs
Namespace            = default
External ID          = <none>
Plugin ID            = nfs
Provider             = nfs.csi.k8s.io
Version              = v4.5.0
Schedulable          = true
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1
Access Mode          = multi-node-multi-writer
Attachment Mode      = file-system
Mount Options        = fs_type: nfs
Namespace            = default

Allocations
ID        Node ID   Task Group    Version  Desired  Status   Created     Modified
9ab25570  f4ca0795  vault-server  0        run      running  18m44s ago  17m55s ago

Please note - once user perform any operation inside the job's allocation related to the vault, then all its data will be stored in the NFS volume because it is being configured inside the job. Even if user stop the allocation and start it again, this data should still be intact.

 

Conclusion

Configuring the NFS CSI plugin in HashiCorp Nomad allows you to leverage NFS storage in your containerized workloads. By following the steps in this guide, one can seamlessly integrate NFS volumes into his Nomad jobs, enabling his applications to access shared storage easily. Remember to adjust the configurations and job files to match your specific environment and requirements.

 

Reference

Nomad Storage Plugins

Nomad volume block

Nomad volume_mount block

Container Storage Interface (CSI) Specification

Configure NFS Linux

Articles in this section

Related articles