How to Resize Terraform Enterprise Volumes When Disk Space Is Full

Problem

When Terraform Enterprise is running in mounted disk operational mode, a volume containing application data may reach its capacity. This can cause performance issues and errors similar to the following.

docker[4541]: ERRO taskqueue/task.go:319 write /var/lib/replicated/db/goleveldb_data/000001.log: no space left on device...

Cause

HashiCorp provides a recommended base size for the backing storage for mounted disk operational mode. If an instance does not meet these size requirements, or if an organization's usage outgrows the configured storage capacity, Terraform Enterprise may experience performance degradation or downtime.

Solution

You can resolve this issue by increasing the size of the affected disk and then resizing the relevant partition and file system. Because Terraform Enterprise can be deployed in multiple cloud or on-premises environments, the implementation of these steps may differ slightly. This guide outlines a general procedure. Where appropriate, you should consult the provided links to external cloud provider documentation for details on their respective storage solutions.

Procedure

1. Identify which filesystem is out of space using the df utility. The output will show the disk usage and mounted filesystems.

   # df -hT
   Filesystem   Type    Size  Used Avail Use% Mounted on
   devtmpfs     tmpfs   2.0G     0  2.0G   0% /dev
   tmpfs        tmpfs   2.0G     0  2.0G   0% /dev/shm
   tmpfs        tmpfs   2.0G  2.7M  2.0G   1% /run
   tmpfs        tmpfs   2.0G     0  2.0G   0% /sys/fs/cgroup
   /dev/xvda1   xfs      50G   50G  651M  99% /
   tmpfs        tmpfs   392M     0  392M   0% /run/user/1000
   overlay      overlay  50G   50G  651M  99% /var/lib/docker/overlay2/3e0c95ad8522c3b22991abc051527e44f7a592279df39c7010c5336a955caa94/merged
   ## ...

   In the example above, /dev/xvda1 has exhausted its 50G of disk space.

2. In mounted disk operational mode, Terraform Enterprise stores application data in the directory specified by the disk_path setting. Identify this directory.

   # replicatedctl app-config export --template '{{ .disk_path.Value }}'
   /opt/terraform-enterprise

3. Inspect the disk usage of that directory to confirm that Terraform Enterprise usage has outgrown the provisioned disk size.

   # du -hs /opt/terraform-enterprise/
   41G /opt/terraform-enterprise/

4. Before making changes, take a snapshot of the disk to enable a rollback if necessary. Refer to your cloud provider's documentation for instructions.

   • AWS: Create Amazon EBS snapshots
   • GCP: Create and manage disk snapshots
   • Azure: Create a snapshot of a virtual hard disk

   For example, in AWS, you can create a snapshot from the EC2 console.

   

5. Increase the size of the mounted disk using your cloud provider's console, CLI, or API. Refer to the relevant documentation for specific steps.

   • AWS: Request modifications to your EBS volumes
   • GCP: Increase size of persistent disk
   • Azure: Expand virtual hard disks on a Linux VM

   In AWS, you can modify the volume from the EC2 console and increase its size.

   

6. After increasing the disk size in your provider's console, connect to the Terraform Enterprise instance and confirm the new disk size is visible to the operating system.

   # lsblk
   NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   xvda    202:0    0  100G  0 disk 
   └─xvda1 202:1    0   50G  0 part /

7. Extend the partition using the growpart or parted utilities.

   # growpart /dev/xvda 1
   CHANGED: partition=1 start=4096 old: size=104853471 end=104857567 new: size=209711071 end=209715167

8. Confirm the partition was extended.

   # lsblk
   NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   xvda    202:0    0  100G  0 disk 
   └─xvda1 202:1    0  100G  0 part /

9. Expand the size of the mounted filesystem. Use the appropriate utility for your filesystem type, which you can confirm with df -T.

   • xfs: xfs_growfs -d <MOUNTPOINT>
   • ext4: resize2fs <DEVICE>

   The following example is for an xfs filesystem.

   # xfs_growfs -d /
   meta-data=/dev/xvda1             isize=512    agcount=26, agsize=524159 blks
            =                       sectsz=512   attr=2, projid32bit=1
            =                       crc=1        finobt=1, sparse=0, rmapbt=0
            =                       reflink=0    bigtime=0 inobtcount=0
   data     =                       bsize=4096   blocks=13106683, imaxpct=25
            =                       sunit=0      swidth=0 blks
   naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
   log      =internal log           bsize=4096   blocks=2560, version=2
            =                       sectsz=512   sunit=0 blks, lazy-count=1
   realtime =none                   extsz=4096   blocks=0, rtextents=0
   data blocks changed from 13106683 to 26213883

10. View the filesystem disk space usage to confirm the change was successful.

    # df -h
    Filesystem      Size  Used Avail Use% Mounted on
    devtmpfs        2.0G     0  2.0G   0% /dev
    tmpfs           2.0G     0  2.0G   0% /dev/shm
    tmpfs           2.0G  2.7M  2.0G   1% /run
    tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
    /dev/xvda1      100G   50G   51G  50% /
    tmpfs           392M     0  392M   0% /run/user/1000
    overlay         100G   50G   51G  50% /var/lib/docker/overlay2/3e0c95ad8522c3b22991abc051527e44f7a592279df39c7010c5336a955caa94/merged
    ## ...

Additional Information

While these steps provide a temporary solution, you should evaluate your organization's usage patterns to implement long-term solutions. For example, migrating to an external services operational mode can improve scalability.

Additionally, installing system-level monitoring or configuring alerts in your cloud provider's monitoring service (such as AWS CloudWatch, Azure Monitor, or Google Cloud Monitoring) can help you anticipate low disk space. This allows your organization to perform necessary system maintenance and avoid downtime.