Disable Huge Logging in Nomad Allocations with the disable_log_collection Parameter in the Podman Driver

Avatar
Suyash Kumar
  • Updated

Overview:

Nomad is a powerful and flexible cluster manager and scheduler designed for deploying and managing applications and containers at scale. While logging is essential for debugging and monitoring applications, excessive log data can sometimes overwhelm your system and lead to performance issues and increased storage costs. To address this concern, Nomad provides the disable_log_collection parameter in the Podman driver, allowing you to control log collection on a per-allocation basis. This KB article explains how to use this feature and discusses the benefits of disabling log collection in certain scenarios.

 

Table of Contents:

  • Introduction

  • Why Disable Log Collection?

  • How to Disable Log Collection in Nomad Allocations

  • Monitor the allocation after deployment

  • Conclusion

 

Introduction:

Nomad, developed by HashiCorp, is widely used for orchestrating and managing containerized workloads across a cluster of machines. It supports various driver types, including the Podman driver, which is used for launching containers using the Podman container engine. The disable_log_collection parameter in the Podman driver allows you to control the collection of log data generated by your containers during execution.

 

Why Disable Log Collection?

Logging is crucial for monitoring the health and behavior of applications. However, in certain scenarios, collecting excessive log data can have adverse effects:

  • Resource Consumption: Large volumes of log data consume CPU, memory, and disk space on the nodes where your containers run. This can lead to resource contention and performance degradation.
  • Increased Costs: Storing vast amounts of log data can be expensive, especially in cloud environments where storage costs are based on usage. Disabling log collection can help reduce these costs.
  • Security and Privacy: Some applications may produce sensitive information in their logs, such as passwords or personally identifiable information (PII). Disabling log collection can help mitigate the risk of exposing sensitive data.
  • Performance Impact: Collecting and forwarding logs in real time can introduce latency and impact the performance of your applications.

 

How to Disable Log Collection in Nomad Allocations

To disable log collection in Nomad allocations using the disable_log_collection parameter, follow these steps:

1. You can follow another article on how to Configure nomad-podman-driver with a Nomad Cluster and Run a Nomad Job Using the Podman Driver. 

2. Configure the Nomad Driver Plugin stanza under the Nomad configuration directory like below with:

plugin "nomad-driver-podman" {
  config {
    socket_path = "unix:///run/podman/podman.sock"
    disable_log_collection = true
    gc {
      container = false
    }
  }
}

Here, I have set the value of disable_log_collection to true, so this will disable Nomad logs collection of Podman tasks. This means, nomad alloc logs <alloc_id> will not show any data, and also respective allocation will not store logs locally on the allocation directory.

3. After making the necessary changes to your configuration, restart the respective Nomad client nodes, and submit or update the job using the Nomad CLI or API. Nomad will schedule the task with log collection disabled.

 

Monitor the allocation after deployment

Below is the sample job - 

job "redis-example" {
  datacenters = ["dc1"]
  type        = "service"

  group "redis-example" {
    count = 1

    network {
      port "http" {
        to = 6379
      }
    }

    task "redis-example" {
      driver = "podman"

      config {
        image = "docker.io/library/redis:7"
        ports = ["http"]
      }

      env {
        MESSAGE = "Hello Podman!🦦"
      }
    }

    service {
      name = "redis-example"
      port = "http"

      check {
        name     = "alive"
        type     = "http"
        path     = "/"
        interval = "10s"
        timeout  = "2s"
      }
    }
  }
}

 

Allocation for the job -

$ nomad job allocs redis-example
ID        Node ID   Task Group     Version  Desired  Status   Created    Modified
099c8277  d75431f8  redis-example  0        run      running  2m34s ago  2m34s ago

 
No logs were found when running <nomad alloc logs <alloc_id> 

$ nomad alloc logs 099c8277
$

 
While checking in Podman log, we can see the logs present -  

$ podman ps
CONTAINER ID  IMAGE                      COMMAND       CREATED        STATUS        PORTS                                                       NAMES
b824d22166ab  docker.io/library/redis:7  redis-server  2 minutes ago  Up 2 minutes  172.31.54.22:20133->6379/tcp, 172.31.54.22:20133->6379/udp  redis-example-099c8277-19fd-91c2-f3d4-756ff156576a
$ podman logs b824d22166ab
1:C 14 Sep 2023 08:46:02.206 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:C 14 Sep 2023 08:46:02.207 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance.
1:C 14 Sep 2023 08:46:02.207 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 14 Sep 2023 08:46:02.207 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 14 Sep 2023 08:46:02.207 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 14 Sep 2023 08:46:02.207 * monotonic clock: POSIX clock_gettime
1:M 14 Sep 2023 08:46:02.208 * Running mode=standalone, port=6379.
1:M 14 Sep 2023 08:46:02.208 * Server initialized
1:M 14 Sep 2023 08:46:02.208 * Ready to accept connections tcp

 
Nomad logs for the same podman container id - b824d22166ab 

Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.499Z [DEBUG] client.driver_mgr.nomad-driver-podman: Checking for recoverable task: driver=podman task=redis-example taskid=099c8277-19fd-91c2-f3d4-756ff156576a/redis-example/f11ba5c8 @module=podman container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 timestamp=2023-09-14T08:53:44.499Z
Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.502Z [DEBUG] client.driver_mgr.nomad-driver-podman: Found a stopped container, removing it: driver=podman container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 @module=podman timestamp=2023-09-14T08:53:44.502Z
Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.518Z [WARN]  client.driver_mgr.nomad-driver-podman: Recovery cleanup failed: driver=podman @module=podman container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 task=099c8277-19fd-91c2-f3d4-756ff156576a/redis-example/f11ba5c8 timestamp=2023-09-14T08:53:44.518Z
Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.518Z [DEBUG] client.driver_mgr.nomad-driver-podman: Recovered container handle: driver=podman @module=podman container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 timestamp=2023-09-14T08:53:44.518Z
Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.518Z [DEBUG] client.driver_mgr.nomad-driver-podman: Monitoring container: driver=podman container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 @module=podman.podmanHandle timestamp=2023-09-14T08:53:44.518Z
Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.521Z [DEBUG] client.driver_mgr.nomad-driver-podman: Container is not running anymore: driver=podman error="No such Container" @module=podman.podmanHandle container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 timestamp=2023-09-14T08:53:44.520Z
Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.521Z [WARN]  client.driver_mgr.nomad-driver-podman: Could not remove container: driver=podman @module=podman container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 error="cannot delete container, status code: 404" timestamp=2023-09-14T08:53:44.521Z
Sep 14 08:53:44 ip-172-31-54-22.ec2.internal nomad[5562]:     2023-09-14T08:53:44.521Z [ERROR] client.driver_mgr.nomad-driver-podman: Failed to inspect stopped container, can not get exit code: driver=podman @module=podman.podmanHandle container=b824d22166ab99acb7c49d113e9d5b7eddc554719a0e5c3fa033debb8e602d60 error="No such Container" timestamp=2023-09-14T08:53:44.521Z

 

Conclusion

The disable_log_collection parameter in the Nomad Podman driver provides a valuable feature for controlling log data collection in your containerized workloads. By selectively disabling log collection for specific tasks or groups, you can optimize resource usage, reduce costs, enhance security and privacy, and minimize performance overhead. When using this feature, it's essential to strike a balance between the benefits of reduced logging and the need for effective monitoring and debugging of your applications.

 

Reference:

https://www.nomadproject.io/

https://podman.io/

 

Articles in this section

See more

Related articles