goroutine Stack Trace Gathering
For certain issues where the vault process is panicking, “hung”, or otherwise crashing, the Vault Technical Support Engineering team could request a live goroutine stack trace output from the relevant vault process for further debugging and troubleshooting.
You can obtain this output in the Vault operational logging by sending a USR2 signal to the relevant vault process. In other words, on the server where Vault is hanging/crashed, use the following example command.
Example Command and Output
To send a running process a SIGUSR2 signal is to re-execute the process:
- USR2 - reexecute the running binary. A separate QUIT should be sent to the original process once the child is verified to be up and running.
You can send the vault process a SIGUSR2 signal like this:
$ sudo kill -SIGUSR2 $(pidof vault)Note that although the kill command is used, the Vault process is not actually terminated as you could expected from other signals, so this is safe to use on any running vault process.
The resulting output will appear in the Vault operational logging, and will contain stack trace information which resembles the following example:
2019-06-18T21:31:26.031Z [INFO] goroutine trace: stack="goroutine 1 [running]:
github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc0006ff900, 0xc0000380a0, 0x2, 0x2, 0x0)
/gopath/src/github.com/hashicorp/vault/command/server.go:1333 +0x42e1
github.com/hashicorp/vault/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc000499180, 0xc000499180, 0xc0004954a0, 0xc0004953a0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/mitchellh/cli/cli.go:255 +0x207
github.com/hashicorp/vault/command.RunCustom(0xc000038090, 0x3, 0x3, 0xc0004a8eb0, 0x0)
/gopath/src/github.com/hashicorp/vault/command/main.go:177 +0x5e4
github.com/hashicorp/vault/command.Run(0xc000038090, 0x3, 0x3, 0xc000082058)
/gopath/src/github.com/hashicorp/vault/command/main.go:87 +0x48
main.main()
/gopath/src/github.com/hashicorp/vault/main.go:10 +0x62
goroutine 17 [syscall, 1 minutes]:
os/signal.signal_recv(0x2f0a160)
/goroot/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/goroot/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/goroot/src/os/signal/signal_unix.go:29 +0x41
goroutine 18 [chan receive, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x4d2f8a0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/golang/glog/glog.go:882 +0x8b
created by github.com/hashicorp/vault/vendor/github.com/golang/glog.init.0
/gopath/src/github.com/hashicorp/vault/vendor/github.com/golang/glog/glog.go:410 +0x203
goroutine 8 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0001c2500)
/gopath/src/github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdd
created by github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view.init.0
/gopath/src/github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57
goroutine 13 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc0006b5da0, 0xc0006ea4e0)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 12 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc0006b46c0, 0xc000083bc0)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 11 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc000079e00, 0xc000082c00)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 14 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeSighupCh.func1(0xc0006b5e00, 0xc0006ea540)
/gopath/src/github.com/hashicorp/vault/command/commands.go:626 +0x34
created by github.com/hashicorp/vault/command.MakeSighupCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:624 +0xd8
goroutine 15 [chan send, 15 minutes]:
github.com/hashicorp/vault/command.MakeSigUSR2Ch.func1(0xc0006b5ec0, 0xc0006ea5a0)
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:22 +0x4c
created by github.com/hashicorp/vault/command.MakeSigUSR2Ch
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:19 +0xd8
goroutine 16 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc000712f00, 0xc0006eaea0)
/gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5
goroutine 33 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeSighupCh.func1(0xc000712f60, 0xc0006eaf00)
/gopath/src/github.com/hashicorp/vault/command/commands.go:626 +0x34
created by github.com/hashicorp/vault/command.MakeSighupCh
/gopath/src/github.com/hashicorp/vault/command/commands.go:624 +0xd8
goroutine 34 [chan receive, 1 minutes]:
github.com/hashicorp/vault/command.MakeSigUSR2Ch.func1(0xc000712fc0, 0xc0006eaf60)
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:21 +0x34
created by github.com/hashicorp/vault/command.MakeSigUSR2Ch
/gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:19 +0xd8
goroutine 35 [select, 17 minutes]:
github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc0001c3180)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/inmem_signal.go:64 +0xb3
created by github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.NewInmemSignal
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/inmem_signal.go:38 +0x13e
goroutine 37 [sleep, 1 minutes]:
time.Sleep(0x3b9aca00)
/goroot/src/runtime/time.go:105 +0x14f
github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc0002140f0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/metrics.go:230 +0x2f
created by github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.New
/gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/start.go:79 +0x170
goroutine 300 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*RollbackManager).run(0xc000666750)
/gopath/src/github.com/hashicorp/vault/vault/rollback.go:107 +0x16b
created by github.com/hashicorp/vault/vault.(*RollbackManager).Start
/gopath/src/github.com/hashicorp/vault/vault/rollback.go:84 +0x3f
goroutine 42 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2f00, 0x72, 0xc00075a870)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001e0798, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001e0798, 0xc000be5000, 0x1000, 0x1000)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001e0780, 0xc000be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0001e0780, 0xc000be5000, 0x1000, 0x1000, 0x40875b, 0xc00002e000, 0x26aa540)
/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001da370, 0xc000be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0001d0360, 0x2efe8a0, 0xc0001da370, 0x5, 0xc0001da370, 0x0)
/goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc00022c000, 0x2c63d17, 0xc00022c120, 0xc00073c700)
/goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc00022c000, 0xc0006a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/goroot/src/crypto/tls/conn.go:1145 +0xf1
bufio.(*Reader).Read(0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0x797f7b, 0x247c1e0, 0xc00075ac80)
/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x2efaaa0, 0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0x9, 0xc000a1c900, 0xc00075ad30, 0x4051ff)
/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x2efaaa0, 0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0xc00075ad10, 0xc000a829d8, 0x795d58)
/goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc000453df8, 0x9, 0x9, 0x2efaaa0, 0xc0006b4de0, 0x0, 0x0, 0xc00075adf8, 0x797436)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000453dc0, 0xc000a3a0c0, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00075afb8, 0x2c62300, 0xc000246fb8)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1658 +0x9e
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc000234fc0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1586 +0x76
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).newClientConn
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:660 +0x636
goroutine 25 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc000123c10, 0xc0000382c0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9
goroutine 265 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).periodicLeaderRefresh(0xc00066e000, 0x0, 0xc0002196e0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:635 +0xe2
github.com/hashicorp/vault/vault.(*Core).runStandby.func5(0xc0006eb260, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:359 +0x3c
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a30f80, 0xc000a30fa0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 27 [select, 1 minutes]:
github.com/hashicorp/vault/physical/consul.(*ConsulBackend).runEventDemuxer(0xc0002141e0, 0xc00060b4b0, 0xc0006eaea0, 0xc000042044, 0x16, 0xc0006d65f0, 0xc0006d6600, 0xc0006d6610)
/gopath/src/github.com/hashicorp/vault/physical/consul/consul.go:689 +0x26d
created by github.com/hashicorp/vault/physical/consul.(*ConsulBackend).RunServiceDiscovery
/gopath/src/github.com/hashicorp/vault/physical/consul/consul.go:655 +0xfa
goroutine 28 [IO wait, 15 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2e30, 0x72, 0x0)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000126198, 0x72, 0xc0001c2d00, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000126198, 0xffffffffffffff00, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000126180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000126180, 0x308, 0x348, 0x203000)
/goroot/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000c108, 0x7fe650f1e000, 0x3, 0xc000046070)
/goroot/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).AcceptTCP(0xc00000c108, 0x42a9f1, 0x2c63d60, 0xc000757e90)
/goroot/src/net/tcpsock.go:247 +0x47
github.com/hashicorp/vault/command/server.TCPKeepAliveListener.Accept(0xc00000c108, 0xc03706584c, 0x3706584c00757eb8, 0x5d09549a, 0xc000757eb8)
/gopath/src/github.com/hashicorp/vault/command/server/listener_tcp.go:117 +0x2f
crypto/tls.(*listener).Accept(0xc00073ba40, 0xc000757f08, 0x18, 0xc000024900, 0x6bb885)
/goroot/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc000473a00, 0x2f0dc60, 0xc00073ba40, 0x0, 0x0)
/goroot/src/net/http/server.go:2826 +0x22f
created by github.com/hashicorp/vault/command.(*ServerCommand).Run
/gopath/src/github.com/hashicorp/vault/command/server.go:1208 +0x5043
goroutine 377 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc000e0a500, 0xc000515a80)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9
goroutine 266 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*Core).waitForLeadership(0xc00066e000, 0x0, 0xc0006eb500, 0xc000219740)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:535 +0xd56
github.com/hashicorp/vault/vault.(*Core).runStandby.func7(0x2c63b58, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:371 +0x45
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc00013c8d0, 0xc000a30fc0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 264 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).periodicCheckKeyUpgrade(0xc00066e000, 0x2f1cba0, 0xc00003c038, 0xc000219620)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:679 +0xca
github.com/hashicorp/vault/vault.(*Core).runStandby.func3(0xc0006eb320, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:347 +0x4b
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a30f20, 0xc000a30f40)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 263 [chan receive, 15 minutes]:
github.com/hashicorp/vault/vault.(*Core).runStandby.func1(0xc0006eb3e0, 0x0)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:338 +0x33
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a5ce00, 0x2c5e1c8)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe
goroutine 275 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2c90, 0x72, 0x0)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00047cd98, 0x72, 0xc000515100, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00047cd98, 0xffffffffffffff00, 0x0, 0x0)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc00047cd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc00047cd80, 0xfd65c0555ec, 0x1dccdd10, 0xc000c09d70)
/goroot/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000c8b8, 0xf4453cbb29, 0x4d2f4e0, 0xe9)
/goroot/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc00000c8b8, 0xbf3a73eb89fb3660, 0xf4453cbb29, 0x4d2f4e0, 0x0)
/goroot/src/net/tcpsock.go:260 +0x47
crypto/tls.(*listener).Accept(0xc0009e4cc0, 0xbf3a73eb89fb3660, 0xf4453cbb29, 0x4d2f4e0, 0x0)
/goroot/src/crypto/tls/tls.go:52 +0x37
github.com/hashicorp/vault/vault.(*ClusterListener).Run.func1(0xc00047cd00, 0xc00000c8b8, 0x2f1cba0, 0xc00003c038, 0xc0006eb440, 0x2f0dc60, 0xc0009e4cc0)
/gopath/src/github.com/hashicorp/vault/vault/cluster.go:493 +0xf8
created by github.com/hashicorp/vault/vault.(*ClusterListener).Run
/gopath/src/github.com/hashicorp/vault/vault/cluster.go:475 +0x486
goroutine 276 [chan receive, 15 minutes]:
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run(0xc000a30f00, 0xc000a30fc0, 0xc000219740)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:43 +0xf0
github.com/hashicorp/vault/vault.(*Core).runStandby(0xc00066e000, 0xc0006eb4a0, 0xc0006eb500, 0xc0006eb560)
/gopath/src/github.com/hashicorp/vault/vault/ha.go:380 +0x4a8
created by github.com/hashicorp/vault/vault.(*Core).unsealInternal
/gopath/src/github.com/hashicorp/vault/vault/core.go:1113 +0x25c
goroutine 284 [select, 5 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*ClientConn).roundTrip(0xc000234fc0, 0xc000a2d500, 0x0, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1060 +0x7db
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0001e8370, 0xc000a2d500, 0xc000c0d600, 0x70000000045136d, 0x41195c, 0x2b4fa19)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:434 +0x150
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).RoundTrip(0xc0001e8370, 0xc000a2d500, 0xc000fc33b0, 0x4d110b0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:396 +0x3a
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0001e8370, 0xc000a2d500, 0xc0006282c1, 0x5, 0xc000471b88)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:2515 +0x39
net/http.(*Transport).roundTrip(0xc0004718c0, 0xc000a2d500, 0x203000, 0xc000c0da90, 0x6a81dd)
/goroot/src/net/http/transport.go:415 +0xd4c
net/http.(*Transport).RoundTrip(0xc0004718c0, 0xc000a2d500, 0xc0004718c0, 0x0, 0x0)
/goroot/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc000a2d500, 0x2efe960, 0xc0004718c0, 0x0, 0x0, 0x0, 0xc00000c8a0, 0x7fe650f1e000, 0xc000c0db20, 0x1)
/goroot/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc0005e38c0, 0xc000a2d500, 0x0, 0x0, 0x0, 0xc00000c8a0, 0x0, 0x1, 0x0)
/goroot/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0005e38c0, 0xc000a2d500, 0x0, 0x0, 0x0)
/goroot/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc0005e38c0, 0xc000a2d500, 0x4d2f4e0, 0x0, 0xc000c0dd40)
/goroot/src/net/http/client.go:509 +0x35
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Client).doRequest(0xc00019a300, 0xc00095e960, 0xc000c0ddf0, 0xc00110aee0, 0x16, 0xc00095e960)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/api.go:765 +0xb8
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*KV).getInternal(0xc000c0df20, 0xc000733360, 0xf, 0x0, 0xc000c0df28, 0xc000666990, 0xc00110d780, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/kv.go:165 +0x284
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*KV).Get(0xc000c0df20, 0xc000733360, 0xf, 0xc000c0df28, 0x0, 0x0, 0x0, 0x0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/kv.go:99 +0x7f
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).monitorLock(0xc000514880, 0xc00098ce70, 0x24, 0xc0006eb740)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:368 +0x12c
created by github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).Lock
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:242 +0x572
goroutine 279 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Session).RenewPeriodic(0xc0001da480, 0x2b4c782, 0x3, 0xc00098ce70, 0x24, 0x0, 0xc000219ec0, 0x413554, 0x7fe64eb1b4b0)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/session.go:163 +0x20b
created by github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).Lock
/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:154 +0xa0b
goroutine 1174 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc0006b28a0, 0xc0006aef80)
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
/gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9
goroutine 381 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func1(0xc0008ae240, 0xc0006eb440, 0xc000515b00, 0xc0001c5c00, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:164 +0xd0
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:163 +0xe7
goroutine 382 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve(0xc000235340)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:826 +0x52f
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Server).ServeConn(0xc000761a40, 0x2f2b480, 0xc0001c5c00, 0xc000d4bf98)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:438 +0x704
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func2(0xc000515b00, 0xc0001c5c00, 0xc0008ae240, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:174 +0xd3
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:173 +0x127
goroutine 449 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2d60, 0x72, 0xc00075b970)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00047df98, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00047df98, 0xc0008b0800, 0x400, 0x400)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00047df80, 0xc0008b0800, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00047df80, 0xc0008b0800, 0x400, 0x400, 0x40875b, 0xc00002e000, 0x26aa540)
/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000cd98, 0xc0008b0800, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0008d2a20, 0x2efe8a0, 0xc00000cd98, 0x5, 0xc00000cd98, 0x0)
/goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc0001c5c00, 0x2c63d17, 0xc0001c5d20, 0x0)
/goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/crypto/tls/conn.go:1145 +0xf1
io.ReadAtLeast(0x7fe64ecc0030, 0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x9, 0xc00075bf28, 0x2, 0x2)
/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x7fe64ecc0030, 0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc00010a038, 0x9, 0x9, 0x7fe64ecc0030, 0xc0001c5c00, 0x0, 0xc000000000, 0xc000712d20, 0xc0006fae08)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00010a000, 0xc00075bee0, 0x2, 0x0, 0x1)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).readFrames(0xc000235340)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:713 +0xa4
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:818 +0x30c
goroutine 478 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2bc0, 0x72, 0xc000c96970)
/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001e0418, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001e0418, 0xc00051cc00, 0x400, 0x400)
/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001e0400, 0xc00051cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0001e0400, 0xc00051cc00, 0x400, 0x400, 0x40875b, 0xc00002e000, 0x26aa540)
/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000c968, 0xc00051cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
/goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0005e3320, 0x2efe8a0, 0xc00000c968, 0x5, 0xc00000c968, 0x0)
/goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc000be7880, 0x2c63d17, 0xc000be79a0, 0x0)
/goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/crypto/tls/conn.go:1145 +0xf1
io.ReadAtLeast(0x7fe64ecc0030, 0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x9, 0xc000c96f28, 0x2, 0x2)
/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x7fe64ecc0030, 0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x0, 0x0, 0x0)
/goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc000c7a038, 0x9, 0x9, 0x7fe64ecc0030, 0xc000be7880, 0x0, 0xc000000000, 0xc0006eea80, 0xc0006faf88)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000c7a000, 0xc000c96ee0, 0x2, 0x0, 0x1)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).readFrames(0xc000668540)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:713 +0xa4
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:818 +0x30c
goroutine 487 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func1(0xc0000824e0, 0xc0006eb440, 0xc000515b00, 0xc000be7880, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:164 +0xd0
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:163 +0xe7
goroutine 488 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve(0xc000668540)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:826 +0x52f
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Server).ServeConn(0xc000761a40, 0x2f2b480, 0xc000be7880, 0xc000c9af98)
/gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:438 +0x704
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func2(0xc000515b00, 0xc000be7880, 0xc0000824e0, 0xc0001dc220)
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:174 +0xd3
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
/gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:173 +0x127
goroutine 378 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).emitMetrics(0xc00066e000, 0xc000dc8960)
/gopath/src/github.com/hashicorp/vault/vault/core.go:1656 +0x119
created by github.com/hashicorp/vault/vault.(*Core).postUnseal
/gopath/src/github.com/hashicorp/vault/vault/core.go:1566 +0x35fWhen requested by HashiCorp Technical Support, it is helpful to archive these logs and the stack traces and share them in the related ticket.
All file attachments are stored securely and removed when no longer relevant to the associated support issue.