Fluent-bit log forwarding issues on the Terraform Enterprise versions before v202111-1

Problem

When using a Terraform Enterprise version prior to v202111-1, you may experience issues with Fluent-bit log forwarding, such as connection timeouts or proxy environment variables not being correctly applied to the fluent-bit container. Error messages may include:

[error] [net] connection timeout after 10 seconds to <hostname>

Additionally, you may observe that HTTP_PROXY and HTTPS_PROXY variables are not injected into the fluent-bit Docker container.

Cause

Terraform Enterprise versions before v202111-1 had limitations in how they handled proxy environment variables and custom CA certificates for the Fluent-bit service. The v202111-1 release introduced two key fixes:

1. Proper injection of HTTP proxy environment variables into the Fluent-bit container.
2. A dedicated script to inject custom CA certificates into the container's trust store.

Solution

To resolve this issue, you must upgrade your Terraform Enterprise instance to version v202111-1 or newer and ensure the environment is correctly configured. The following procedure outlines the required steps.

Procedure

1. Stop the Terraform Enterprise application.

   $ replicatedctl app stop

2. Stop the Replicated services.

   $ sudo systemctl stop replicated replicated-ui replicated-operator

3. Update the docker and libseccomp packages on the Terraform Enterprise server to meet the minimum version requirements.

   • docker version must be 20.10.0 or newer.
   • libseccomp version must be 2.4.4 or newer.

   Note: Failure to update these dependencies before upgrading Terraform Enterprise may result in startup issues.

4. Start the Replicated services.

   $ sudo systemctl start replicated replicated-ui replicated-operator

5. Start the Terraform Enterprise application.

   $ replicatedctl app start

6. Perform the upgrade to Terraform Enterprise release v202111-1 or a more recent version. Follow the official documentation for guidance:
   • Upgrading Terraform Enterprise guide
   • Terraform Enterprise Upgrades documentation

7. After the upgrade, configure proxy and CA passthrough to the fluent-bit container. This feature allows you to inject custom CA certificates into a container's system CA bundle.

   For example, update the container's configuration to call the /usr/bin/setup-ca-certificates.sh script. If a container previously had the following configuration:

   cmd: >-
     ["/usr/bin/app-start.sh"]
   config_files:
     - filename: /usr/bin/app-start.sh
       file_mode: "0755"
       contents: $read-template:app-start.sh.erb

   Update it to inject custom CA certificates:

   cmd: >-
     ["/usr/bin/setup-ca-certificates.sh", "/usr/bin/app-start.sh"]
   config_files:
     - *CUSTOM_CA_CERTIFICATES_FILE
     - *SETUP_CA_CERTIFICATES
     - filename: /usr/bin/app-start.sh
       file_mode: "0755"
       contents: $read-template:app-start.sh.erb

8. Ensure that the required proxy environment variables are set in uppercase so they are forwarded to the fluent-bit container.

   HTTP_PROXY
   HTTPS_PROXY
   NO_PROXY

9. Perform a full restart of Terraform Enterprise to apply all changes.

   $ replicatedctl app stop

   $ sudo systemctl stop replicated replicated-ui replicated-operator

   $ sudo systemctl start replicated replicated-ui replicated-operator

   $ replicatedctl app start