Consul Out of Memory: Cannot Allocate

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

Problem

In previous Consul deployments, a 32-bit binary might have been utilized initially. However, as usage grows, Consul's resource consumption increases accordingly, potentially leading to memory exhaustion and subsequent out-of-memory (OOM) crashes due to the inherent 4GB memory limit on 32-bit architectures. This challenge may not be readily apparent since Consul's 32-bit binary can run on operating systems with 64-bit architecture.

Prerequisites

• Root access to the server running Consul
• Management token if ACLs are enabled

Cause

• The following error will be evident in the logs if Consul cannot allocate the required memory for the workload.

consul-server-1 consul[1572]: runtime: out of memory: cannot allocate 1499136-byte block (2816507904 in use)
consul-server-1 consul[1572]: fatal error: out of memory

• • Depending on the Consul systemd configuration, the Consul service may restart on failure and can cause Consul leadership instability.

Solution

Determine the Consul architecture in use and consider transitioning to a 64-bit architecture if a 32-bit one is detected.

1. There are two ways to identify the arch of Consul on a given system
   
   The “arch” values you might see are:
        386 - 32-bit intel arch
        amd64 - 64-bit intel arch
        arm64 - 64-bit arm arch
        arm - 32-bit arm arch
        s390x - 64 bit S390/IBM System Z arch
   • Option 1: Run the consul info command
     

     consul info | grep runtime -A5
     runtime:
             arch = 386
             cpu_count = 2
             goroutines = 238
             max_procs = 2
             os = linux

   • Option 2: Run the consul debug command and review agent.json file
     

         "runtime": {
             "arch": "386",
             "cpu_count": "16",
             "goroutines": "764",
             "max_procs": "16",
             "os": "linux",
             "version": "go1.13.7"
         },

2. Verify the operating system architecture
   

   uname -a | awk '{print $NF}'

3. Download the 64-bit Consul binary on each node.
4. Follow the General Upgrade Process steps
   
   NOTE: Update the followers first and leave the leader for last

Outcome

The Consul administrator can ascertain the architecture of Consul and, if necessary, transition binaries with no impact on the cluster by following the outlined procedure.

Additional Information

• Consul Releases