Vault Experiencing a Stack Dump at the end of each Calendar Month

Problem

At the end of each calendar month (for example, at midnight - 00:00 from the 30th of November to the 1st of December), Vault experiences a panic (stack dump) and writes it to the operational logs.

The issue, if encountered, happens while loading ClientIDs to memory.

Example stack trace written to the Vault Operational Logs:

Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: goroutine 5 [select]:
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/main.go:13 +0x47
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: main.main()
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/command/main.go:147
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/command.Run(...)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/command/main.go:245 +0x9a6
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/command.RunCustom({0xc0000b4ca0?, 0x2?, 0x2?}, 0xc000002380?)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /go-mod-cache/github.com/hashicorp/cli@v1.1.7/cli.go:265 +0x4de
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/cli.(*CLI).Run(0xc000da6140)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/command/server.go:1657 +0x48d0
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc00120a008, {0xc0000b4cb0, 0x1, 0x1})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: goroutine 1 [select, 1294 minutes]:
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:1247 +0x2d2
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: created by github.com/hashicorp/vault/vault.(*Core).setupActivityLogLocked in goroutine 121
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:1644 +0x446
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/vault.(*ActivityLog).activeFragmentWorker(0xc00367a000, {0xfe05408, 0xc00110da40})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:1721 +0x399
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/vault.(*ActivityLog).HandleEndOfMonth(0xc00367a000, {0xfe05408, 0xc00110da40}, {0x179f, 0xee0bed180, 0x0})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /build/vault/activity_log.go:3648 +0x73
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: github.com/hashicorp/vault/vault.(*ActivityLog).cleanupEphemeralClients(0xc00367a000)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /opt/go/src/internal/runtime/maps/table.go:683 +0x86
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: internal/runtime/maps.(*Iter).Next(0xc01433fd00?)
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]:         /opt/go/src/runtime/panic.go:1058 +0x18
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: internal/runtime/maps.fatal({0xddea9d5?, 0xc0160d5a40?})
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: goroutine 106699603 [running]:
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: fatal error: concurrent map iteration and map write
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: 2025-12-01T01:00:00.014+0100 [INFO]  activity: finished loading segments to store client IDs in memory
Dec 01 01:00:00 fqdn.appid.env.test.domain.local vault[1747815]: 2025-12-01T01:00:00.014+0100 [INFO]  activity: starting to load clientIDS to memory: start_time="2025-11-01 00:00:00 +0000 UTC" end_time="2025-11-30 23:59:59 +0000 UTC" months_included=["2025-11-01 00:00:00 +0000 UTC"]

Typically Vault automatically recovers from this condition, however it is possible that a service interruption may occur requiring a restart of the Vault process.

Overview of possible solutions (if applicable)

The defect has been resolved in Vault Enterprise 1.20.7 and Vault Enterprise 1.21.2.  Upgrade to a version containing the fix.