How to run tfc-agent binary with memory limits

Expected Outcome

This guide explains how to allocate specific memory limits to the tfc-agent binary process using two different methods with systemd.

Prerequisites

• You have access to download the tfc-agent binary package or have a local copy.
• The following Linux programs are installed: unzip, screen.
• You are using a Linux operating system with systemd version 219 or later.

Procedure

There are two primary methods to apply memory limits to the tfc-agent process. The first method is for agents run ephemerally using systemd-run, and the second is for agents configured as a persistent systemd service.

Method 1: Using systemd-run for an Ephemeral Agent

This method is suitable for running the agent on-demand without creating a permanent service file. Connect to the instance using SSH and follow these steps as a non-root user.

1. Create a directory for the agent, download the binary, and extract it. This example uses version 1.10.1.

   $ ## Create a new directory for the agent and change into it
   $ mkdir TFC_AGENT; cd TFC_AGENT
   
   $ ## Download the tfc-agent binary package
   $ wget https://releases.hashicorp.com/tfc-agent/1.10.1/tfc-agent_1.10.1_linux_amd64.zip
   
   $ ## Extract the package
   $ unzip tfc-agent_1.10.1_linux_amd64.zip -d ./

2. Create a tfc-agent.env file with the required environment variables. Replace the placeholder values with your configuration.

   $ cat > tfc-agent.env <<EOF
   TFC_ADDRESS=https://tfe.example.net
   TFC_AGENT_TOKEN=REPLACE_WITH_YOUR_AGENT_TOKEN
   TFC_AGENT_NAME=ubuntu-aws
   TFC_AGENT_LOG_LEVEL=INFO
   TFC_AGENT_DATA_DIR=/home/ubuntu/TFC_AGENT
   EOF

3. Create a new screen session to run the agent process in the background.

   $ screen -S tfc-agent

4. Launch the tfc-agent binary using systemd-run to apply memory limits. Adjust MemoryHigh and MemoryMax as needed. If your instance does not have swap space, omit the MemorySwapMax parameter. You will be prompted for your user password to authorize the action.

   $ env $(cat ~/TFC_AGENT/tfc-agent.env) systemd-run \
       --unit=tfc-agent-binary \
       --scope \
       -p MemoryHigh=600M \
       -p MemoryMax=800M \
       -p MemorySwapMax=1 \
       ~/TFC_AGENT/tfc-agent >> ~/TFC_AGENT/tfc-agent.log 2>&1

   You will see an authentication prompt.

   ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
   Authentication is required to manage system services or other units.
   Authenticating as: Ubuntu (ubuntu)
   Password:
   ==== AUTHENTICATION COMPLETE ===

5. Detach from the screen session by pressing Ctrl+a followed by d.

6. Verify the agent is running and that the memory limits have been applied. The systemd scope unit is ephemeral and will be removed when the process stops.

   $ ## Check the status of the ephemeral scope unit
   $ systemctl status tfc-agent-binary.scope

   The output shows the applied memory limits.

   ● tfc-agent-binary.scope - /home/ubuntu/TFC_AGENT/tfc-agent
      Loaded: loaded (/run/systemd/transient/tfc-agent-binary.scope; transient)
   Transient: yes
      Active: active (running) since Mon 2023-07-31 15:10:22 UTC; 31min ago
       Tasks: 16 (limit: 4686)
      Memory: 66.0M (high: 600.0M max: 800.0M swap max: 1B) <-- Memory limits
      CGroup: /system.slice/tfc-agent-binary.scope
              ├─13047 /home/ubuntu/TFC_AGENT/tfc-agent
              └─13070 /home/ubuntu/TFC_AGENT/tfc-agent-core

7. Check the agent's log file for operational messages.

   $ tail -f ~/TFC_AGENT/tfc-agent.log

   ##...
   [INFO] core: Agent registered successfully with Terraform Cloud: agent_id=agent-fbxBEfR6RaScVkhW agent_pool_id=apool-NGTvRq5FrUo1RyLu
   [INFO] agent: Core version is up to date: version=1.12.0
   [INFO] core: Waiting for next job
   ##...

Method 2: Configuring a Persistent Systemd Service

If you have configured the tfc-agent to run as a persistent service, you can add memory limits directly to its systemd unit file.

1. Open the tfc-agent service unit file for editing. The path may vary based on your setup.

   $ ## Example using vim; use your preferred text editor
   $ sudo vim /etc/systemd/system/tfc-agent.service

2. In the [Service] section of the file, add the MemoryHigh, MemoryMax, and MemorySwapMax attributes. Omit MemorySwapMax if your system does not have swap enabled.

   [Unit]
   Description=Service to automatically start TFC/E Agent
   After=network.target
   
   [Install]
   WantedBy=multi-user.target
   
   [Service]
   EnvironmentFile=/opt/tfc_agent/tfc-agent.env
   Type=simple
   ExecStart=/opt/tfc_agent/tfc-agent
   KillSignal=SIGINT
   MemoryHigh=600M
   MemoryMax=800M
   MemorySwapMax=1
   WorkingDirectory=/opt/tfc_agent
   Restart=always
   RestartSec=5
   StandardOutput=syslog
   StandardError=syslog
   SyslogIdentifier=%n

3. Save the changes to the file, then reload the systemd daemon and restart the tfc-agent service to apply the new configuration.

   $ ## Reload the systemd daemon to read the new configuration
   $ sudo systemctl daemon-reload
   
   $ ## Restart the tfc-agent service
   $ sudo systemctl restart tfc-agent

4. Confirm that systemd loaded the memory limit changes.

   $ systemctl show tfc-agent | grep -E 'MemoryHigh|MemoryMax|MemorySwapMax'

   The output displays the memory values in bytes.

   MemoryHigh=629145600
   MemoryMax=838860800
   MemorySwapMax=1

Additional Information

• For more details on setting up the agent as a service, refer to the guide on How to run tfc-agent binary as a Service with Systemd.