goroutine Stack Trace Gathering
For certain issues where the vault process is panicking, “hung”, or otherwise crashing, the Vault Technical Support Engineering team could request a live goroutine stack trace output and pprof files from the relevant vault process for further debugging and troubleshooting.
You can obtain this output in the Vault operational logging by sending a USR2 signal to the relevant vault process. In other words, on the server where Vault is hanging/crashed, use the following example command.
Example Command and Output
To send a running process a SIGUSR2 signal is to re-execute the process:
- USR2 - reexecute the running binary. A separate QUIT should be sent to the original process once the child is verified to be up and running.
You can send the vault process a SIGUSR2 signal like this:
$ sudo kill -SIGUSR2 $(pidof vault)Note that although the kill command is used, the Vault process is not actually terminated as you could expected from other signals, so this is safe to use on any running vault process.
The resulting output will appear in the Vault operational logging, and will contain stack trace information which resembles the following example:
Stack Trace Example (Click to expand)
2019-06-18T21:31:26.031Z [INFO] goroutine trace: stack="goroutine 1 [running]:
github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc0006ff900, 0xc0000380a0, 0x2, 0x2, 0x0)
/gopath/src/github.com/hashicorp/vault/command/server.go:1333 +0x42e1
github.com/hashicorp/vault/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc000499180, 0xc000499180, 0xc0004954a0, 0xc0004953a0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/mitchellh/cli/cli.go:255 +0x207
github.com/hashicorp/vault/command.RunCustom(0xc000038090, 0x3, 0x3, 0xc0004a8eb0, 0x0)
/gopath/src/github.com/hashicorp/vault/command/main.go:177 +0x5e4
github.com/hashicorp/vault/command.Run(0xc000038090, 0x3, 0x3, 0xc000082058)
/gopath/src/github.com/hashicorp/vault/command/main.go:87 +0x48
main.main()
/gopath/src/github.com/hashicorp/vault/main.go:10 +0x62
goroutine 17 [syscall, 1 minutes]:
os/signal.signal_recv(0x2f0a160)
/goroot/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/goroot/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/goroot/src/os/signal/signal_unix.go:29 +0x41
goroutine 18 [chan receive, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x4d2f8a0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/golang/glog/glog.go:882 +0x8b
created by github.com/hashicorp/vault/vendor/github.com/golang/glog.init.0
/gopath/src/github.com/hashicorp/vault/vendor/github.com/golang/glog/glog.go:410 +0x203
goroutine 8 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0001c2500)
/gopath/src/github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdd
created by github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view.init.0
/gopath/src/github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57
goroutine 13 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc0006b5da0, 0xc0006ea4e0)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 12 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc0006b46c0, 0xc000083bc0)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 11 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc000079e00, 0xc000082c00)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 14 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeSighupCh.func1(0xc0006b5e00, 0xc0006ea540)
/gopath/src/github.com/hashicorp/vault/command/commands.go:626 +0x34
created by github.com/hashicorp/vault/command.MakeSighupCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:624 +0xd8
goroutine 15 [chan send, 15 minutes]:
github.com/hashicorp/vault/command.MakeSigUSR2Ch.func1(0xc0006b5ec0, 0xc0006ea5a0)
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:22 +0x4c
created by github.com/hashicorp/vault/command.MakeSigUSR2Ch
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:19 +0xd8
goroutine 16 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc000712f00, 0xc0006eaea0)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 33 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeSighupCh.func1(0xc000712f60, 0xc0006eaf00)
/gopath/src/github.com/hashicorp/vault/command/commands.go:626 +0x34
created by github.com/hashicorp/vault/command.MakeSighupCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:624 +0xd8
goroutine 34 [chan receive, 1 minutes]:
github.com/hashicorp/vault/command.MakeSigUSR2Ch.func1(0xc000712fc0, 0xc0006eaf60)
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:21 +0x34
created by github.com/hashicorp/vault/command.MakeSigUSR2Ch
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:19 +0xd8
goroutine 35 [select, 17 minutes]:
github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc0001c3180)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/inmem_signal.go:64 +0xb3
created by github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.NewInmemSignal
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/inmem_signal.go:38 +0x13e
goroutine 37 [sleep, 1 minutes]:
time.Sleep(0x3b9aca00)
/goroot/src/runtime/time.go:105 +0x14f
github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc0002140f0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/metrics.go:230 +0x2f
created by github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.New
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/start.go:79 +0x170
goroutine 300 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*RollbackManager).run(0xc000666750)
/gopath/src/github.com/hashicorp/vault/vault/rollback.go:107 +0x16b
created by github.com/hashicorp/vault/vault.(*RollbackManager).Start
/gopath/src/github.com/hashicorp/vault/vault/rollback.go:84 +0x3f
goroutine 42 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2f00, 0x72, 0xc00075a870)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001e0798, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001e0798, 0xc000be5000, 0x1000, 0x1000)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001e0780, 0xc000be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0001e0780, 0xc000be5000, 0x1000, 0x1000, 0x40875b, 0xc00002e000, 0x26aa540)
/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001da370, 0xc000be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0001d0360, 0x2efe8a0, 0xc0001da370, 0x5, 0xc0001da370, 0x0)
/goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc00022c000, 0x2c63d17, 0xc00022c120, 0xc00073c700)
/goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc00022c000, 0xc0006a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/goroot/src/crypto/tls/conn.go:1145 +0xf1
bufio.(*Reader).Read(0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0x797f7b, 0x247c1e0, 0xc00075ac80)
/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x2efaaa0, 0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0x9, 0xc000a1c900, 0xc00075ad30, 0x4051ff)
/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x2efaaa0, 0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0xc00075ad10, 0xc000a829d8, 0x795d58)
/goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc000453df8, 0x9, 0x9, 0x2efaaa0, 0xc0006b4de0, 0x0, 0x0, 0xc00075adf8, 0x797436)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000453dc0, 0xc000a3a0c0, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00075afb8, 0x2c62300, 0xc000246fb8)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1658 +0x9e
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc000234fc0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1586 +0x76
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).newClientConn
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:660 +0x636
goroutine 25 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc000123c10, 0xc0000382c0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9
goroutine 265 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).periodicLeaderRefresh(0xc00066e000, 0x0, 0xc0002196e0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:635 +0xe2
github.com/hashicorp/vault/vault.(*Core).runStandby.func5(0xc0006eb260, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:359 +0x3c
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a30f80, 0xc000a30fa0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 27 [select, 1 minutes]:
github.com/hashicorp/vault/physical/consul.(*ConsulBackend).runEventDemuxer(0xc0002141e0, 0xc00060b4b0, 0xc0006eaea0, 0xc000042044, 0x16, 0xc0006d65f0, 0xc0006d6600, 0xc0006d6610)
/gopath/src/github.com/hashicorp/vault/physical/consul/consul.go:689 +0x26d
created by github.com/hashicorp/vault/physical/consul.(*ConsulBackend).RunServiceDiscovery
/gopath/src/github.com/hashicorp/vault/physical/consul/consul.go:655 +0xfa
goroutine 28 [IO wait, 15 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2e30, 0x72, 0x0)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000126198, 0x72, 0xc0001c2d00, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000126198, 0xffffffffffffff00, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000126180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000126180, 0x308, 0x348, 0x203000)
/goroot/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000c108, 0x7fe650f1e000, 0x3, 0xc000046070)
/goroot/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).AcceptTCP(0xc00000c108, 0x42a9f1, 0x2c63d60, 0xc000757e90)
/goroot/src/net/tcpsock.go:247 +0x47
github.com/hashicorp/vault/command/server.TCPKeepAliveListener.Accept(0xc00000c108, 0xc03706584c, 0x3706584c00757eb8, 0x5d09549a, 0xc000757eb8)
/gopath/src/github.com/hashicorp/vault/command/server/listener_tcp.go:117 +0x2f
crypto/tls.(*listener).Accept(0xc00073ba40, 0xc000757f08, 0x18, 0xc000024900, 0x6bb885)
/goroot/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc000473a00, 0x2f0dc60, 0xc00073ba40, 0x0, 0x0)
/goroot/src/net/http/server.go:2826 +0x22f
created by github.com/hashicorp/vault/command.(*ServerCommand).Run
/gopath/src/github.com/hashicorp/vault/command/server.go:1208 +0x5043
goroutine 377 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc000e0a500, 0xc000515a80)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9
goroutine 266 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*Core).waitForLeadership(0xc00066e000, 0x0, 0xc0006eb500, 0xc000219740)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:535 +0xd56
github.com/hashicorp/vault/vault.(*Core).runStandby.func7(0x2c63b58, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:371 +0x45
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc00013c8d0, 0xc000a30fc0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 264 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).periodicCheckKeyUpgrade(0xc00066e000, 0x2f1cba0, 0xc00003c038, 0xc000219620)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:679 +0xca
github.com/hashicorp/vault/vault.(*Core).runStandby.func3(0xc0006eb320, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:347 +0x4b
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a30f20, 0xc000a30f40)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 263 [chan receive, 15 minutes]:
github.com/hashicorp/vault/vault.(*Core).runStandby.func1(0xc0006eb3e0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:338 +0x33
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a5ce00, 0x2c5e1c8)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 275 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2c90, 0x72, 0x0)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00047cd98, 0x72, 0xc000515100, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00047cd98, 0xffffffffffffff00, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc00047cd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc00047cd80, 0xfd65c0555ec, 0x1dccdd10, 0xc000c09d70)
/goroot/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000c8b8, 0xf4453cbb29, 0x4d2f4e0, 0xe9)
/goroot/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc00000c8b8, 0xbf3a73eb89fb3660, 0xf4453cbb29, 0x4d2f4e0, 0x0)
/goroot/src/net/tcpsock.go:260 +0x47
crypto/tls.(*listener).Accept(0xc0009e4cc0, 0xbf3a73eb89fb3660, 0xf4453cbb29, 0x4d2f4e0, 0x0)
/goroot/src/crypto/tls/tls.go:52 +0x37
github.com/hashicorp/vault/vault.(*ClusterListener).Run.func1(0xc00047cd00, 0xc00000c8b8, 0x2f1cba0, 0xc00003c038, 0xc0006eb440, 0x2f0dc60, 0xc0009e4cc0)
/gopath/src/github.com/hashicorp/vault/vault/cluster.go:493 +0xf8
created by github.com/hashicorp/vault/vault.(*ClusterListener).Run
/gopath/src/github.com/hashicorp/vault/vault/cluster.go:475 +0x486
goroutine 276 [chan receive, 15 minutes]:
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run(0xc000a30f00, 0xc000a30fc0, 0xc000219740)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:43 +0xf0
github.com/hashicorp/vault/vault.(*Core).runStandby(0xc00066e000, 0xc0006eb4a0, 0xc0006eb500, 0xc0006eb560)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:380 +0x4a8
created by github.com/hashicorp/vault/vault.(*Core).unsealInternal
/gopath/src/github.com/hashicorp/vault/vault/core.go:1113 +0x25c
goroutine 284 [select, 5 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*ClientConn).roundTrip(0xc000234fc0, 0xc000a2d500, 0x0, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1060 +0x7db
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0001e8370, 0xc000a2d500, 0xc000c0d600, 0x70000000045136d, 0x41195c, 0x2b4fa19)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:434 +0x150
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).RoundTrip(0xc0001e8370, 0xc000a2d500, 0xc000fc33b0, 0x4d110b0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:396 +0x3a
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0001e8370, 0xc000a2d500, 0xc0006282c1, 0x5, 0xc000471b88)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:2515 +0x39
net/http.(*Transport).roundTrip(0xc0004718c0, 0xc000a2d500, 0x203000, 0xc000c0da90, 0x6a81dd)
/goroot/src/net/http/transport.go:415 +0xd4c
net/http.(*Transport).RoundTrip(0xc0004718c0, 0xc000a2d500, 0xc0004718c0, 0x0, 0x0)
/goroot/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc000a2d500, 0x2efe960, 0xc0004718c0, 0x0, 0x0, 0x0, 0xc00000c8a0, 0x7fe650f1e000, 0xc000c0db20, 0x1)
/goroot/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc0005e38c0, 0xc000a2d500, 0x0, 0x0, 0x0, 0xc00000c8a0, 0x0, 0x1, 0x0)
/goroot/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0005e38c0, 0xc000a2d500, 0x0, 0x0, 0x0)
/goroot/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc0005e38c0, 0xc000a2d500, 0x4d2f4e0, 0x0, 0xc000c0dd40)
/goroot/src/net/http/client.go:509 +0x35
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Client).doRequest(0xc00019a300, 0xc00095e960, 0xc000c0ddf0, 0xc00110aee0, 0x16, 0xc00095e960)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/api.go:765 +0xb8
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*KV).getInternal(0xc000c0df20, 0xc000733360, 0xf, 0x0, 0xc000c0df28, 0xc000666990, 0xc00110d780, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/kv.go:165 +0x284
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*KV).Get(0xc000c0df20, 0xc000733360, 0xf, 0xc000c0df28, 0x0, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/kv.go:99 +0x7f
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).monitorLock(0xc000514880, 0xc00098ce70, 0x24, 0xc0006eb740)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:368 +0x12c
created by github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).Lock
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:242 +0x572
goroutine 279 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Session).RenewPeriodic(0xc0001da480, 0x2b4c782, 0x3, 0xc00098ce70, 0x24, 0x0, 0xc000219ec0, 0x413554, 0x7fe64eb1b4b0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/session.go:163 +0x20b
created by github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).Lock
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:154 +0xa0b
goroutine 1174 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc0006b28a0, 0xc0006aef80)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9
goroutine 381 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func1(0xc0008ae240, 0xc0006eb440, 0xc000515b00, 0xc0001c5c00, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:164 +0xd0
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:163 +0xe7
goroutine 382 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve(0xc000235340)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:826 +0x52f
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Server).ServeConn(0xc000761a40, 0x2f2b480, 0xc0001c5c00, 0xc000d4bf98)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:438 +0x704
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func2(0xc000515b00, 0xc0001c5c00, 0xc0008ae240, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:174 +0xd3
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:173 +0x127
goroutine 449 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2d60, 0x72, 0xc00075b970)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00047df98, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00047df98, 0xc0008b0800, 0x400, 0x400)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00047df80, 0xc0008b0800, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00047df80, 0xc0008b0800, 0x400, 0x400, 0x40875b, 0xc00002e000, 0x26aa540)
/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000cd98, 0xc0008b0800, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0008d2a20, 0x2efe8a0, 0xc00000cd98, 0x5, 0xc00000cd98, 0x0)
/goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc0001c5c00, 0x2c63d17, 0xc0001c5d20, 0x0)
/goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/crypto/tls/conn.go:1145 +0xf1
io.ReadAtLeast(0x7fe64ecc0030, 0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x9, 0xc00075bf28, 0x2, 0x2)
/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x7fe64ecc0030, 0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc00010a038, 0x9, 0x9, 0x7fe64ecc0030, 0xc0001c5c00, 0x0, 0xc000000000, 0xc000712d20, 0xc0006fae08)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00010a000, 0xc00075bee0, 0x2, 0x0, 0x1)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).readFrames(0xc000235340)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:713 +0xa4
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:818 +0x30c
goroutine 478 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2bc0, 0x72, 0xc000c96970)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001e0418, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001e0418, 0xc00051cc00, 0x400, 0x400)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001e0400, 0xc00051cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0001e0400, 0xc00051cc00, 0x400, 0x400, 0x40875b, 0xc00002e000, 0x26aa540)
/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000c968, 0xc00051cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0005e3320, 0x2efe8a0, 0xc00000c968, 0x5, 0xc00000c968, 0x0)
/goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc000be7880, 0x2c63d17, 0xc000be79a0, 0x0)
/goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/crypto/tls/conn.go:1145 +0xf1
io.ReadAtLeast(0x7fe64ecc0030, 0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x9, 0xc000c96f28, 0x2, 0x2)
/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x7fe64ecc0030, 0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc000c7a038, 0x9, 0x9, 0x7fe64ecc0030, 0xc000be7880, 0x0, 0xc000000000, 0xc0006eea80, 0xc0006faf88)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000c7a000, 0xc000c96ee0, 0x2, 0x0, 0x1)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).readFrames(0xc000668540)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:713 +0xa4
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:818 +0x30c
goroutine 487 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func1(0xc0000824e0, 0xc0006eb440, 0xc000515b00, 0xc000be7880, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:164 +0xd0
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:163 +0xe7
goroutine 488 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve(0xc000668540)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:826 +0x52f
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Server).ServeConn(0xc000761a40, 0x2f2b480, 0xc000be7880, 0xc000c9af98)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:438 +0x704
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func2(0xc000515b00, 0xc000be7880, 0xc0000824e0, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:174 +0xd3
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:173 +0x127
goroutine 378 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).emitMetrics(0xc00066e000, 0xc000dc8960)
/gopath/src/github.com/hashicorp/vault/vault/core.go:1656 +0x119
created by github.com/hashicorp/vault/vault.(*Core).postUnseal
/gopath/src/github.com/hashicorp/vault/vault/core.go:1566 +0x35f
PPROF & STACKTRACE Environment Variables
Support might request some, or all of the following environment variables be set to aid in the collection of data from issuing a SIGUSR2 starting Vault 1.11.11 and beyond...
Vault 1.17 and later releases
Starting in Vault 1.17, PPROF files generate automatically, without the need to set any environment variables. If no environment variables are set, PPROF files write to /tmp, and stack trace outputs to the operational logs. If the following adjustments are requested or needed, please set the following:
Related Environment Variables
VAULT_STACKTRACE_WRITE_TO_FILE
VAULT_STACKTRACE_FILE_PATH (Optional; string - filepath)
TMPDIR (Optional; string - filepath for PPROF files)
Ex:
VAULT_STACKTRACE_WRITE_TO_FILE=true
VAULT_STACKTRACE_FILE_PATH=/vault
TMPDIR=/vaultPlease note the file paths must exist and the user running vault process has permissions to write to the file paths specified above
Vault 1.11.11, 1.12.7, 1.13.3, 1.14.0 - 1.16.x releases
In these releases, to generate PPROF data VAULT_PPROF_WRITE_TO_FILE environment variable must be set. If no environment variables are set, stack trace outputs to the operational logs. If the following adjustments are requested or needed, please set the following:
Related Environment Variables
VAULT_STACKTRACE_WRITE_TO_FILE
VAULT_STACKTRACE_FILE_PATH (Optional; string - filepath)
VAULT_PPROF_WRITE_TO_FILE
VAULT_PPROF_FILE_PATH (Optional; string - filepath)
Ex:
VAULT_STACKTRACE_WRITE_TO_FILE=true
VAULT_STACKTRACE_FILE_PATH=/vault
VAULT_PPROF_WRITE_TO_FILE=true
VAULT_PPROF_FILE_PATH=/vaultPlease note the file paths must exist and the user running vault process has permissions to write to the file paths specified above
Process recap for data collection
- Set environment variables (if requested by Support); ie -
VAULT_PPROF_WRITE_TO_FILE=true&VAULT_STACKTRACE_WRITE_TO_FILE=true - Add the environment variables to the Vault process
- When an issue is suspected with the vault process, send
SIGUSR2 - Any time a SIGUSR2 is issued, files will be written to specified directories (exercise awareness in environment)
If aforementioned environment variable(s) are set, the SIGUSR2 results in the expected stack trace being written to operational logs (or file if environment variable is set), but it also generates pprof data to the specified path, or to /tmp directory. Please note, the location of the files is written to the operational logs, as such:
2024-05-02T20:44:44.305Z [INFO] Wrote pprof files to: /vault/vault-pprof
2024-05-02T20:44:44.288Z [INFO] Wrote stacktrace to: /vault/stacktrace121255954
systemd - PrivateTmp
note: in systemd, if filepath is not specified and PrivateTmp is set to yes in the unit file, this can create a directory in /tmp containing the files owned by root. The subdirectories should contain the data...
Ex filepath not specified (click to expand):
2023-10-23T21:23:11.667Z [INFO] Wrote pprof files to: /tmp/vault-pprof3151861046
[root@ip-10-13-13-62 vault-pprof3151861046]# pwd
/tmp/systemd-private-8dcd2a5dc9ef4fff93fa6fcde7d04c07-vault.service-zH7RIm/tmp/vault-pprof3151861046
[root@ip-10-13-13-62 vault-pprof3151861046]# ls -ltrah
total 52K
drwxrwxrwt. 3 root root 35 Oct 23 21:23 ..
-rw-r--r--. 1 vault vault 5.9K Oct 23 21:23 goroutine
-rw-r--r--. 1 vault vault 17K Oct 23 21:23 heap
-rw-r--r--. 1 vault vault 17K Oct 23 21:23 allocs
drwx------. 2 vault vault 69 Oct 23 21:23 .
-rw-r--r--. 1 vault vault 460 Oct 23 21:23 threadcreate
[root@ip-10-13-13-62 vault-pprof3151861046]#
When requested by HashiCorp Technical Support, it is helpful to archive these logs and the stack traces and share them in the related ticket.
All file attachments are stored securely and removed when no longer relevant to the associated support issue.