Vault Experiencing a Stack Dump at the end of each Calendar Month

Problem

At the end of each calendar month (for example, at midnight - 00:00 from the 30th of November to the 1st of December), Vault experiences a panic (stack dump) and writes it to the operational logs.

The issue, if encountered, happens while loading ClientIDs to memory

Example stack trace written to the Vault Operational Logs:

Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: goroutine 5 [select]:
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/main.go:13 +0x47
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: main.main()
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/command/main.go:147
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/command.Run(...)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/command/main.go:245 +0x9a6
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/command.RunCustom({0xc0000b4ca0?, 0x2?, 0x2?}, 0xc000002380?)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /go-mod-cache/github.com/hashicorp/cli@v1.1.7/cli.go:265 +0x4de
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/cli.(*CLI).Run(0xc000da6140)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/command/server.go:1657 +0x48d0
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc00120a008, {0xc0000b4cb0, 0x1, 0x1})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: goroutine 1 [select, 1294 minutes]:
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:1247 +0x2d2
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: created by github.com/hashicorp/vault/vault.(*Core).setupActivityLogLocked in goroutine 121
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:1644 +0x446
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/vault.(*ActivityLog).activeFragmentWorker(0xc00367a000, {0xfe05408, 0xc00110da40})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:1721 +0x399
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/vault.(*ActivityLog).HandleEndOfMonth(0xc00367a000, {0xfe05408, 0xc00110da40}, {0x179f, 0xee0bed180, 0x0})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:3648 +0x73
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/vault.(*ActivityLog).cleanupEphemeralClients(0xc00367a000)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /opt/go/src/internal/runtime/maps/table.go:683 +0x86
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: internal/runtime/maps.(*Iter).Next(0xc01433fd00?)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /opt/go/src/runtime/panic.go:1058 +0x18
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: internal/runtime/maps.fatal({0xddea9d5?, 0xc0160d5a40?})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: goroutine 106699603 [running]:
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: fatal error: concurrent map iteration and map write
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: 2025-12-01T01:00:00.014+0100 [INFO]  activity: finished loading segments to store client IDs in memory
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: 2025-12-01T01:00:00.014+0100 [INFO]  activity: starting to load clientIDS to memory: start_time="2025-11-01 00:00:00 +0000 UTC" end_time="2025-11-30 23:59:59 +0000 UTC" months_included=["2025-11-01 00:00:00 +0000 UTC"]

Typically Vault automatically recovers from this condition, but it is possible that a service interruption may occur.

Cause

The cause is linked to concurrent map iteration and write in the cleanupEphemeralClients() function.

Overview of possible solutions (if applicable)

Currently there is no workaround. Engineering are actively working on a fix that is anticipated to be in the next release of Vault Enterprise.