How to correctly handle/attach EBS-backed volumes and NVMe on Linux instances

Introduction

EBS volumes are exposed as NVMe block devices on instances built on the Nitro System (a lot of nowadays ). The device names are /dev/nvme0n1, /dev/nvme1n1, and so on. The device names that you specify in a block device mapping are renamed using NVMe device names (/dev/nvme[0-26]n1). The block device driver *can assign NVMe device names in a different order than you specified for the volumes in the block device mapping*. And that’s how, for example, assigned data-volume for the TFE got suddenly lost during the restart - because often scripts of provision instance specifying either mounts or some other operations based on the absolute device name. In the worst-case scenario - this can lead to the production outage, or broken backup restore (when the absolute device had been specified to LVM operation commands).

We are going to look at the situation from the perspective of the Terraform and Linux systems, see section [Additional Information] towards the end of this document.

Recommendation

To avoid unpleasant surprises, we recommend identifying devices corresponding to the EBS volumes by the EBS volume ID: vol-01234567890abcdefand not by absolute number.

Different AMIs (and different operation systems distros) allow you to do that in different ways, the most frequently used ones are listed below :

Amazon Linux AMIs

With Amazon Linux AMI 2017.09.01 or later (including Amazon Linux 2), you can run the ebsnvme-id command as follows to map the NVMe device name to a volume ID and device name:

[ec2-user ~]$ sudo /sbin/ebsnvme-id /dev/nvme1n1
Volume ID: vol-01324f611e2463981
/dev/sdf

So in this case you are straight-forward aware that device /dev/sdf corresponds to some EBS volume with id : vol-01324f611e2463981 - use this information in you provision scripts. For example you can assign this device name to a shell variable DEVICE and format and find UUID for the device to be used later in a way :

sudo mkfs -t xfs -q $DEVICE
PUUID=$(sudo blkid $DEVICE | grep -E -o "[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}")

Debian derivatives (Uincluding Ubuntu)

You will need a Linux kernel at least 4.2. Most distributives (AMIs) in this branch do not have the nvme id-ctrl command by default installed. So you will need to fix this by issuing :

sudo apt install -q -y nvme-cli

Afterward, you can either use nvme list to get the list of all the NVMe devices with corresponding volume IDs or nvme id-ctrl as below :

The following example gets the volume ID and device name. The device name is available through the NVMe controller vendor-specific extension (bytes 384:4095 of the controller identification):

[ec2-user ~]$ sudo nvme id-ctrl -v /dev/nvme1n1
NVME Identify Controller:
vid     : 0x1d0f
ssvid   : 0x1d0f
sn      : vol01234567890abcdef
mn      : Amazon Elastic Block Store
...
0000: 2f 64 65 76 2f 73 64 6a 20 20 20 20 20 20 20 20 "/dev/sdf..."

Warning: As you can see above, standart nvme-cli tools present the EBS volume ID without dash “vol01234567890abcdef”, pay attention to this in your provisioning scripts.
One could think of stripping dash from the EBS id in variable EBS_ID like this in BASH :

VOLUME="${EBS_ID//-}"  # strip dash from volume ID as nvme list return them without dash

And later - the lsblk command lists available devices and their mount points (if applicable). This helps you determine the correct device name to use. In this example, /dev/nvme0n1p1 is already mounted as the root device and /dev/nvme1n1 is attached but not mounted.

[ec2-user ~]$ sudo lsblk
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1       259:3   0  100G  0 disk
nvme0n1       259:0   0    8G  0 disk
  nvme0n1p1   259:1   0    8G  0 part /
  nvme0n1p128 259:2   0    1M  0 part

RedHat and derivatives (CentOS, Fedora, Scientifc Linux…)

All of them coming by default without nvme-cli.

For CentOS/Scientific Linux systems you can install just by using: sudo yum install nvme-cli
For RedHat Enterprise Linux NVMe CLI available as snap: sudo snap install nvme-cli --candidate

For some other free derivatives of RedHat or older systems you may need to use REMI repository: from https://rpms.remirepo.net/

Further usage going to be the same as with Debian-based systems.

Additional Information

1. You can look up Amazon’s recommendations for NVMe-EBS volumes in details here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html (including information for other UNIX-based systems)
2. General information about device naming could be found here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html
3. Home-page of NVMe-CLI tools: https://github.com/linux-nvme/nvme-cli