How-to use OpenTelemetry with Agent metrics - Kubernetes

Introduction

The HCP Terraform agent can export metrics to an OpenTelemetry Collector, as documented in the agent telemetry documentation.

This guide provides an example configuration for using the OpenTelemetry fileexporter to write agent telemetry data to a file within a Kubernetes container.

Expected Outcome

After completing this guide, you will have a functional example of an OpenTelemetry Collector receiving metrics from an HCP Terraform agent and writing them to a file.

Prerequisites

• An active HCP Terraform or Terraform Enterprise environment with agents configured in Kubernetes.

Procedure

Follow these steps to configure and verify the OpenTelemetry integration.

1. Create a Kubernetes ConfigMap to store the OpenTelemetry Collector configuration.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: otel-config
     namespace: terraform-enterprise-agents
   data:
     config.yaml: |
       extensions:
         memory_ballast:
           size_mib: 128
   
       receivers:
         otlp/tfe_agent:
           protocols:
             grpc:
               endpoint: 0.0.0.0:4317
   
       processors:
         batch:
         memory_limiter:
           check_interval: 1s
           limit_mib: 1024
           spike_limit_percentage: 20
   
       exporters:
         file/no_rotation:
           path: /exported_data/telemetry_output.json
         logging:
         file:
           path: /dev/null
   
       service:
         extensions: [memory_ballast]
         pipelines:
           traces:
             receivers: [otlp/tfe_agent]
             exporters: [file]
           metrics/tfe_agent:
             receivers: [otlp/tfe_agent]
             processors: [memory_limiter, batch]
             exporters: [file/no_rotation]
         telemetry:

2. Define a Kubernetes Pod that includes two containers: one for the OpenTelemetry Collector and another busybox container for inspecting the output file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: otel-collector
     namespace: terraform-enterprise-agents
     labels:
       app: otel-collector
   spec:
     containers:
       - name: busybox
         image: busybox
         command: [ "sleep","infinity" ]
         volumeMounts:
           - name: data-volume
             mountPath: /exported_data
       - name: collector
         image: otel/opentelemetry-collector-contrib:0.73.0
         args: ["--config", "/etc/otel/config.yaml"]
         ports:
           - containerPort: 4317
         volumeMounts:
           - name: config-volume
             mountPath: /etc/otel/config.yaml
             subPath: config.yaml
           - name: data-volume
             mountPath: /exported_data
     volumes:
       - name: config-volume
         configMap:
           name: otel-config
       - name: data-volume
         emptyDir: {}

3. Expose the OpenTelemetry Collector pod with a Kubernetes Service to make it accessible to the agents.

   apiVersion: v1
   kind: Service
   metadata:
     name: otel-collector
     namespace: terraform-enterprise-agents
   spec:
     selector:
       app: otel-collector
     ports:
       - name: otlp-grpc
         port: 4317
         targetPort: 4317

4. Configure the Terraform Enterprise agent to send metrics to the OpenTelemetry Collector service. Add the following environment variable to your overrides.yaml file for the Terraform Enterprise Helm installation.

   agentWorkerPodTemplate:
     spec:
       containers:
       - env:
         - name: TFC_AGENT_OTLP_ADDRESS
           value: otel-collector.terraform-enterprise-agents.svc.cluster.local:4317

5. Verify that the agent is sending metrics after a run. Execute a shell in the busybox container to inspect the output file.

   $ kubectl -n terraform-enterprise-agents exec -it otel-collector -c busybox -- sh

6. After accessing the shell, view the contents of the output file.

   $ cat /exported_data/telemetry_output.json

7. The output file should contain JSON data similar to the following example.

   {
     "key": "agent_name",
     "value": {
       "stringValue": "agent-tele"
     }
   }, {
     "key": "agent_pool_id",
     "value": {
       "stringValue": "apool-RjP1Tazzy1HU1gGE"
     }
   }

Additional Information

• For more details on available agent metrics, refer to the HCP Terraform Agent Metrics documentation.
• The OpenTelemetry Collector Contrib repository contains the collector and its components.
• Detailed information on the fileexporter is available in the OpenTelemetry documentation.