Vault cluster node join failure due to NO_PROXY behaviour change in Go 1.23.7+

Summary:

A breaking change introduced in Go version 1.23.7 affected the behavior of the NO_PROXY environment variable, causing Vault versions 1.19.2 and above to fail when attempting to join a cluster. This issue stems from a change in how NO_PROXY matches IP addresses.

Affected Versions:

• Vault Versions: 1.19.2 and above.

Root Cause:

In Vault 1.19.2, the Go version was bumped to version 1.23.7. This version of Go introduced a breaking change in the behaviour of the NO_PROXY environment variable:

• Previous Behaviour: IP suffixes in NO_PROXY which includes .1 in the list, were matched.(Ex: 10.110.150.1, 1.1.1.1)

• New Behaviour (Go 1.23.7+): IP suffix which includes .1 is no longer matched as IP suffix, as detailed in Go Issue #72123.

The above change in Go is an intended change to remediate vulnerabilities with previous matching patterns. This causes Vault to ignore NO_PROXY settings that relied on suffix-based IP matching.

Symptoms:

• Vault attempts to connect through the configured proxy (e.g., Squid) despite NO_PROXY settings.

• Cluster join fails during bootstrap with a 403 Forbidden error(example below) :

Apr 29 02:40:15 ip-10-110-150-1.non.prod.hashicorp.test.com start.sh[2550]: 2025-04-29T02:40:15.511Z [ERROR] core: failed to get raft challenge: leader_addr=https://10.140.190.108:8200 error="error during raft bootstrap init call: Put \"https://10.140.190.108:8200/v1/sys/storage/raft/bootstrap/challenge\": Forbidden"

• Vault 1.19.1 and earlier do not exhibit this behaviour.

• Unsetting HTTP_PROXY/HTTPS_PROXY in the environment allows the node to join the cluster.

Resolution:

• Update the NO_PROXY settings to ensure compatibility with the new behaviour in Go 1.23.7 and above .

• Use explicit IPs or CIDRs instead of suffixes.

References

• Go Issue #72123: net/http: no_proxy no longer matches an IP suffix