How to run tfc-agent binary with memory limits

Expected Outcome

Allocate memory for the tfc-agent binary process

Prerequisites

•  Access to download the tfc-agent binary package or local copy.
•  Installed version of the following Linux programs: unzip, screen.
•  Linux OS with Systemd version 219 or later.

Procedure

• SSH into the instance and follow the steps below as a regular user:

  # Create a new directory for the agent and move into it
  mkdir TFC_AGENT; cd TFC_AGENT
  
  # Download the tfc-agent binary package, example:
  wget https://releases.hashicorp.com/tfc-agent/1.10.1/tfc-agent_1.10.1_linux_amd64.zip
  
  # Extract the package
  unzip tfc-agent_1.10.1_linux_amd64.zip -d ./
  
  # Create a tfc-agent.env file with the tfc-agent variables,example:
  cat > tfc-agent.env <<EOF
  TFC_ADDRESS=https://tfe.example.net        <-- Skip this for TFC
  TFC_AGENT_TOKEN=Q4jKvOMx0RHzMA.atlasv1.qiyyzbGbANsKQmMi9MmBFhoMIa8dXSjH1VEJo6Pj9XVRKDp0YtKZbylcaGjUfPEh0YE
  TFC_AGENT_NAME=ubuntu-aws
  TFC_AGENT_LOG_LEVEL=INFO
  TFC_AGENT_DATA_DIR=/home/ubuntu/TFC_AGENT  <-- This is not required
  EOF
  
  # Create a new screen session to launch the agent
  screen -S tfc-agent
  
  # Launch the tfc-agent binary and enter the user's password when prompted. Adjust MemoryHigh and MemoryMax as needed. If your instance does not have swap skip MemorySwapMax.
  env $(cat ~/TFC_AGENT/tfc-agent.env) systemd-run --unit=tfc-agent-binary --scope -p MemoryHigh=600M -p MemoryMax=800M -p MemorySwapMax=1 ~/TFC_AGENT/tfc-agent >> ~/TFC_AGENT/tfc-agent.log 2>&1 
  ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
  Authentication is required to manage system services or other units.
  Authenticating as: Ubuntu (ubuntu)
  Password:
  ==== AUTHENTICATION COMPLETE ===
  
  # Detach from the screen session using hotkey
  Ctrl+a,b

• The tfc-agent will continue to run and you can check the status and logs: 

  # Systemd will create an ephemeral scope unit
  systemctl status tfc-agent-binary.scope
  ● tfc-agent-binary.scope - /home/ubuntu/TFC_AGENT/tfc-agent
  Loaded: loaded (/run/systemd/transient/tfc-agent-binary.scope; transient)
  Transient: yes
  Active: active (running) since Mon 2023-07-31 15:10:22 UTC; 31min ago
  Tasks: 16 (limit: 4686)
  Memory: 66.0M (high: 600.0M max: 800.0M swap max: 1B) <-- Memory limits
  CGroup: /system.slice/tfc-agent-binary.scope
  ├─13047 /home/ubuntu/TFC_AGENT/tfc-agent
  └─13070 /home/ubuntu/TFC_AGENT/tfc-agent-core
  
  systemd[1]: Started /home/ubuntu/TFC_AGENT/tfc-agent
  
  # Check the tfc-agent log
  tail -f ~/TFC_AGENT/tfc-agent.log
  [INFO] core: Agent registered successfully with Terraform Cloud: agent_id=agent-PY2vBKoHtwpruAuj agent_pool_id=apool-NGTvRq5FrUo1RyLu
  [INFO] agent: Newer core versions are available, but are ignored due to the auto-update configuration: version=1.12.0-rc.1
  [INFO] agent: Core update is available: version=1.12.0
  [INFO] agent: Core successfully updated: version=1.12.0
  [INFO] agent: Core plugin is shutting down
  [INFO] core: Shutdown complete
  [INFO] core: Starting: version=1.12.0
  [INFO] core: Agent registered successfully with Terraform Cloud: agent_id=agent-fbxBEfR6RaScVkhW agent_pool_id=apool-NGTvRq5FrUo1RyLu
  [INFO] agent: Core version is up to date: version=1.12.0
  [INFO] core: Waiting for next job
  [INFO] core: Job received: job_type=plan job_id=run-MtkvifRkpPRM6Fqu
  [INFO] terraform: Handling run: run_id=run-MtkvifRkpPRM6Fqu run_operation=plan organization_name=havananet workspace_name=tf-large-state-generator-a
  [INFO] terraform: Extracting Terraform from release archive
  [INFO] terraform: Terraform CLI details: version=1.4.0
  [INFO] terraform: Downloading Terraform configuration
  [INFO] terraform: Running terraform init
  [INFO] terraform: Running terraform plan
  [INFO] terraform: Generating and uploading plan JSON
  [INFO] terraform: Generating and uploading provider schemas JSON
  [INFO] terraform: Skipping plan redaction, no upload URL found.
  [INFO] terraform: Persisting filesystem to remote storage
  [INFO] terraform: Finished handling run
  [INFO] core: Waiting for next job

• If the tfc-agent binary was setup as a Systemd Unit, follow the steps below:

  # Edit the unit file for the tfc-agent, example:
  vim /etc/systemd/system/tfc-agent.service
  # Add the memory attibutes. Ignore MemorySwapMax if the system has no swap

  [Unit]
  Description=Service to automatically start TFC/E Agent
  After=network.target
  
  [Install]
  WantedBy=multi-user.target
  
  [Service]
  EnvironmentFile=/opt/tfc_agent/tfc-agent.env
  Type=simple
  ExecStart=/opt/tfc_agent/tfc-agent
  KillSignal=SIGINT
  MemoryHigh=600M
  MemoryMax=800M
  MemorySwapMax=1
  WorkingDirectory=/opt/tfc_agent
  Restart=always
  RestartSec=5
  StandardOutput=syslog
  StandardError=syslog
  SyslogIdentifier=%n

• Save the changes made to the file, reload the Systemd Daemon and restart the tfc-agent service:

  # Reload the Systemd Daemon
  sudo systemctl daemon-reload
  
  # Restart the tfc-agent service
  sudo systemctl restart tfc-agent
  
  # Confirm changes were loaded
  systemctl show tfc-agent|grep -E 'MemoryHigh|MemoryMax|MemorySwapMax'
  MemoryHigh=629145600
  MemoryMax=838860800
  MemorySwapMax=1

Additional Information

• How to run tfc-agent binary as a Service with Systemd