Intro
This KB is dedicated to resolve Fluent-bit log forwarding issues on the Terraform Enterprise versions before v202111-1
Problem description
Customer is experiencing Fluent-bit log forwarding issues like:
- [error] [net] connection timeout after 10 seconds to
- HTTP_PROXY and HTTPS_PROXY variables are not injected to the fluent-bit docker container
Solution
There were 2 fixes for the fluent-bit log forwarding added in the latest Terraform Enterprise release v202111-1
:
-
Injecting the HTTP proxy env vars into Fluent Bit
-
Use of dedicated common script to inject CA certificates
To make log forwarding work perform following:
Stop the Terraform Enterprise
replicatedctl app stop
systemctl stop replicated replicated-ui-operator replicated
Update docker and libseccomp on the Terraform Enterprise Server
- ensure that
docker
is updated to version at least >=20.10.0
- ensure that libseccomp >=
2.4.4
Warning: If docker and libseccomp are not updated - Terraform Enterprise v202111-1 will face issues with startup
Start the Terraform Enterprise
systemctl start replicated replicated-operator replicated-ui
replicatedctl app start
Perform Terraform Enterprise upgrade to the release v202111-1
- Follow https://support.hashicorp.com/hc/en-us/articles/360059165254-Upgrading-Terraform-Enterprise
- Then follow https://www.terraform.io/docs/enterprise/admin/upgrades.html
Configure Proxy and CA passthrough to the fluent-bit docker container. This feature is introduced in v202111-1
To inject custom CA certificates into a container's system CA bundle:
Configure the CUSTOM_CA_CERTIFICATES_FILE YAML anchor within a container's config_files directive.Configure the SETUP_CA_CERTIFICATES YAML anchor within a container's config_files directive.Update the container's entrypoint or cmd directive to call the /usr/bin/setup-ca-certificates.sh script, passing in the previous entrypoint or cmd as an argument. For example, if a container previously had the following configuration:
cmd: >-
["/usr/bin/app-start.sh"]
config_files:
- filename: /usr/bin/app-start.sh
file_mode: "0755"
contents: $read-template:app-start.sh.erb
It should be updated to the following configuration in order to inject custom CA certificates:
cmd: >-
["/usr/bin/setup-ca-certificates.sh", "/usr/bin/app-start.sh"]
config_files:
- *CUSTOM_CA_CERTIFICATES_FILE
- *SETUP_CA_CERTIFICATES
- filename: /usr/bin/app-start.sh
file_mode: "0755"
contents: $read-template:app-start.sh.erb
Keep in mind the following when attempting to inject custom CA certificates using the above approach:
Some container images have their own logic for injecting custom CA certificates into the system CA bundle. A good example is the hashicorp/atlas image which will inject all certificates in /tmp/cust-ca-certificates.crt into its system CA bundle using this script. A container may not need custom CA certificates injected if it does not make network calls. A container may not have a shell in order to execute the /usr/bin/setup-ca-certificates.sh script. For example, containers built from the distroless images have no shell. An alternative method, such as mounting a volume or using a static binary, should be used to inject the CA certificates in this case.
Ensure that you have following environment variables set up in uppercase so they are forwarded to fluent-bit docker container
HTTP_PROXY
HTTPS_PROXY
NO_PROXY
Restart the Terraform Enterprise
replicatedctl app stop
systemctl stop replicated replicated-ui-operator replicated
systemctl start replicated replicated-operator replicated-ui
replicatedctl app start