How-to use OpenTelemetry with Agent metrics

Introduction

The Terraform Cloud Agent has the option to export metrics to OpenTelemetry Collector as documented here

This article shows an example of using the OpenTelemetry fileexporter (see here for more details) to write the telemetry data to a file. 

Expected Outcome

At the end you have a working example of using the Terraform Cloud Agent metrics and the possibilities. 

Prerequisites

• Terraform Cloud/Terraform Enterprise
• Configured Agent Pool within Terraform Cloud/Terraform Enterprise

Procedure

Below is an example configuration to use 

• Connect to the server where the OpenTelemetry Collector container will be started
  
  
• Create a file called telemetry.yaml. Use the following content example

extensions:
  memory_ballast:
    size_mib: 128

receivers:
  otlp/tfe_agent:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
    spike_limit_percentage: 20

exporters:
  file/no_rotation:
    path: /exported_data/telemetry_output.json
  logging:
  file:
    path: /dev/null

service:
  extensions: [memory_ballast]
  pipelines: 
    traces: 
      receivers: [otlp/tfe_agent]
      exporters: [file]
    metrics/tfe_agent:
      receivers: [otlp/tfe_agent]
      processors: [memory_limiter, batch]
      exporters: [file/no_rotation]
telemetry:
  logs:
    level: "debug"

• Create a directory called exported_data. This is the directory where the output file will be written to
  
  
• The structure should now look like this

.
├── exported_data
└── telemetry.yaml

• Start the OpenTelemetry container with the following configuration

docker run -d \
  --name collector \
  --volume $(pwd)/telemetry.yaml:/etc/otel/config.yaml \
  --volume $(pwd)/exported_data:/exported_data \
  -p 4317:4317 \
  -p 55679:55679 \
  --rm otel/opentelemetry-collector-contrib:0.73.0 \
  --config /etc/otel/config.yaml

• Start the Terraform Cloud agent with your agent pool configuration and the OpenTelemtry configuration.
  
  TFC_AGENT_OTLP_ADDRESS needs to be the address where your OpenTelemetry container is running. 

export TFC_ADDRESS="https://<TFE_fqdn>" 
export TFC_AGENT_TOKEN=<agent_token>
export TFC_AGENT_NAME=<name_of_the_agent>
export TFC_AGENT_OTLP_ADDRESS=<ipaddress_or_fqdn>>:4317

docker run -e TFC_ADDRESS -e TFC_AGENT_TOKEN -e TFC_AGENT_NAME -e TFC_AGENT_OTLP_ADDRESS hashicorp/tfc-agent:1.14.1

• When you do a run now from Terraform Cloud/Terraform Enterprise with this agent the metrics should be send to OpenTelemetry. OpenTelemetry should write these metrics to the file called telemetry_output.json in the exported_data directory
  
  
• The file should contain the following for example

{"key":"agent_name","value":{"stringValue":"agent-tele"}},{"key":"agent_pool_id","value":{"stringValue":"apool-RjP1Tazzy1HU1gGE"}}

Additional Information

• Terraform Cloud Agent metrics can be found here

• OpenTelemetry Collector can be found here
• OpenTelemetry fileexporter details can be found here