Fluent-bit log forwarding issues on the Terraform Enterprise versions before v202111-1

Avatar
Anton Akulov
  • Updated

Intro

This KB is dedicated to resolve Fluent-bit log forwarding issues on the Terraform Enterprise versions before v202111-1

Problem description

Customer is experiencing Fluent-bit log forwarding issues like:

  • [error] [net] connection timeout after 10 seconds to
  • HTTP_PROXY and HTTPS_PROXY variables are not injected to the fluent-bit docker container

Solution

There were 2 fixes for the fluent-bit log forwarding added in the latest Terraform Enterprise release v202111-1:

  • Injecting the HTTP proxy env vars into Fluent Bit

  • Use of dedicated common script to inject CA certificates

To make log forwarding work perform following:

Stop the Terraform Enterprise

  • replicatedctl app stop
  • systemctl stop replicated replicated-ui-operator replicated

Update docker and libseccomp on the Terraform Enterprise Server

  • ensure that docker is updated to version at least >= 20.10.0
  • ensure that libseccomp >= 2.4.4

Warning: If docker and libseccomp are not updated - Terraform Enterprise v202111-1 will face issues with startup

Start the Terraform Enterprise

  • systemctl start replicated replicated-operator replicated-ui
  • replicatedctl app start

Perform Terraform Enterprise upgrade to the release v202111-1

Configure Proxy and CA passthrough to the fluent-bit docker container. This feature is introduced in v202111-1

To inject custom CA certificates into a container's system CA bundle:

Configure the CUSTOM_CA_CERTIFICATES_FILE YAML anchor within a container's config_files directive.Configure the SETUP_CA_CERTIFICATES YAML anchor within a container's config_files directive.Update the container's entrypoint or cmd directive to call the /usr/bin/setup-ca-certificates.sh script, passing in the previous entrypoint or cmd as an argument. For example, if a container previously had the following configuration:

 

cmd: >-
["/usr/bin/app-start.sh"]

config_files:
- filename: /usr/bin/app-start.sh
file_mode: "0755"
contents: $read-template:app-start.sh.erb
It should be updated to the following configuration in order to inject custom CA certificates:

cmd: >-
["/usr/bin/setup-ca-certificates.sh", "/usr/bin/app-start.sh"]

config_files:
- *CUSTOM_CA_CERTIFICATES_FILE
- *SETUP_CA_CERTIFICATES
- filename: /usr/bin/app-start.sh
file_mode: "0755"
contents: $read-template:app-start.sh.erb

Keep in mind the following when attempting to inject custom CA certificates using the above approach:

Some container images have their own logic for injecting custom CA certificates into the system CA bundle. A good example is the hashicorp/atlas image which will inject all certificates in /tmp/cust-ca-certificates.crt into its system CA bundle using this script. A container may not need custom CA certificates injected if it does not make network calls. A container may not have a shell in order to execute the /usr/bin/setup-ca-certificates.sh script. For example, containers built from the distroless images have no shell. An alternative method, such as mounting a volume or using a static binary, should be used to inject the CA certificates in this case.

Ensure that you have following environment variables set up in uppercase so they are forwarded to fluent-bit docker container

HTTP_PROXY 
HTTPS_PROXY 
NO_PROXY

Restart the Terraform Enterprise

replicatedctl app stop

systemctl stop replicated replicated-ui-operator replicated

systemctl start replicated  replicated-operator replicated-ui

replicatedctl app start

 

Articles in this section

See more

Related articles