goroutine Stack Trace and PPROF Collection

Avatar
Brian Shumate
  • Updated

goroutine Stack Trace Gathering

For certain issues where the vault process is panicking, “hung”, or otherwise crashing, the Vault Technical Support Engineering team could request a live goroutine stack trace output from the relevant vault process for further debugging and troubleshooting.

You can obtain this output in the Vault operational logging by sending a USR2 signal to the relevant vault process. In other words, on the server where Vault is hanging/crashed, use the following example command.

Example Command and Output

To send a running process a SIGUSR2 signal is to re-execute the process:

  • USR2 - reexecute the running binary. A separate QUIT should be sent to the original process once the child is verified to be up and running.

You can send the vault process a SIGUSR2 signal like this:

$ sudo kill -SIGUSR2 $(pidof vault)

Note that although the kill command is used, the Vault process is not actually terminated as you could expected from other signals, so this is safe to use on any running vault process.

The resulting output will appear in the Vault operational logging, and will contain stack trace information which resembles the following example:

Stack Trace Example (Click to expand) 
2019-06-18T21:31:26.031Z [INFO]  goroutine trace: stack="goroutine 1 [running]:
github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc0006ff900, 0xc0000380a0, 0x2, 0x2, 0x0)
    /gopath/src/github.com/hashicorp/vault/command/server.go:1333 +0x42e1
github.com/hashicorp/vault/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc000499180, 0xc000499180, 0xc0004954a0, 0xc0004953a0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/mitchellh/cli/cli.go:255 +0x207
github.com/hashicorp/vault/command.RunCustom(0xc000038090, 0x3, 0x3, 0xc0004a8eb0, 0x0)
    /gopath/src/github.com/hashicorp/vault/command/main.go:177 +0x5e4
github.com/hashicorp/vault/command.Run(0xc000038090, 0x3, 0x3, 0xc000082058)
    /gopath/src/github.com/hashicorp/vault/command/main.go:87 +0x48
main.main()
    /gopath/src/github.com/hashicorp/vault/main.go:10 +0x62

goroutine 17 [syscall, 1 minutes]:
os/signal.signal_recv(0x2f0a160)
    /goroot/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
    /goroot/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /goroot/src/os/signal/signal_unix.go:29 +0x41

goroutine 18 [chan receive, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x4d2f8a0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/golang/glog/glog.go:882 +0x8b
created by github.com/hashicorp/vault/vendor/github.com/golang/glog.init.0
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/golang/glog/glog.go:410 +0x203

goroutine 8 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0001c2500)
    /gopath/src/github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdd
created by github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view.init.0
    /gopath/src/github.com/hashicorp/vault/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57

goroutine 13 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc0006b5da0, 0xc0006ea4e0)
    /gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
    /gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5

goroutine 12 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc0006b46c0, 0xc000083bc0)
    /gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
    /gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5

goroutine 11 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc000079e00, 0xc000082c00)
    /gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
    /gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5

goroutine 14 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeSighupCh.func1(0xc0006b5e00, 0xc0006ea540)
    /gopath/src/github.com/hashicorp/vault/command/commands.go:626 +0x34
created by github.com/hashicorp/vault/command.MakeSighupCh
    /gopath/src/github.com/hashicorp/vault/command/commands.go:624 +0xd8

goroutine 15 [chan send, 15 minutes]:
github.com/hashicorp/vault/command.MakeSigUSR2Ch.func1(0xc0006b5ec0, 0xc0006ea5a0)
    /gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:22 +0x4c
created by github.com/hashicorp/vault/command.MakeSigUSR2Ch
    /gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:19 +0xd8

goroutine 16 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeShutdownCh.func1(0xc000712f00, 0xc0006eaea0)
    /gopath/src/github.com/hashicorp/vault/command/commands.go:610 +0x34
created by github.com/hashicorp/vault/command.MakeShutdownCh
    /gopath/src/github.com/hashicorp/vault/command/commands.go:609 +0xf5

goroutine 33 [chan receive, 17 minutes]:
github.com/hashicorp/vault/command.MakeSighupCh.func1(0xc000712f60, 0xc0006eaf00)
    /gopath/src/github.com/hashicorp/vault/command/commands.go:626 +0x34
created by github.com/hashicorp/vault/command.MakeSighupCh
    /gopath/src/github.com/hashicorp/vault/command/commands.go:624 +0xd8

goroutine 34 [chan receive, 1 minutes]:
github.com/hashicorp/vault/command.MakeSigUSR2Ch.func1(0xc000712fc0, 0xc0006eaf60)
    /gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:21 +0x34
created by github.com/hashicorp/vault/command.MakeSigUSR2Ch
    /gopath/src/github.com/hashicorp/vault/command/commands_nonwindows.go:19 +0xd8

goroutine 35 [select, 17 minutes]:
github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc0001c3180)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/inmem_signal.go:64 +0xb3
created by github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.NewInmemSignal
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/inmem_signal.go:38 +0x13e

goroutine 37 [sleep, 1 minutes]:
time.Sleep(0x3b9aca00)
    /goroot/src/runtime/time.go:105 +0x14f
github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc0002140f0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/metrics.go:230 +0x2f
created by github.com/hashicorp/vault/vendor/github.com/armon/go-metrics.New
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/armon/go-metrics/start.go:79 +0x170

goroutine 300 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*RollbackManager).run(0xc000666750)
    /gopath/src/github.com/hashicorp/vault/vault/rollback.go:107 +0x16b
created by github.com/hashicorp/vault/vault.(*RollbackManager).Start
    /gopath/src/github.com/hashicorp/vault/vault/rollback.go:84 +0x3f

goroutine 42 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2f00, 0x72, 0xc00075a870)
    /goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001e0798, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
    /goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001e0798, 0xc000be5000, 0x1000, 0x1000)
    /goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001e0780, 0xc000be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0001e0780, 0xc000be5000, 0x1000, 0x1000, 0x40875b, 0xc00002e000, 0x26aa540)
    /goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001da370, 0xc000be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0001d0360, 0x2efe8a0, 0xc0001da370, 0x5, 0xc0001da370, 0x0)
    /goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc00022c000, 0x2c63d17, 0xc00022c120, 0xc00073c700)
    /goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc00022c000, 0xc0006a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /goroot/src/crypto/tls/conn.go:1145 +0xf1
bufio.(*Reader).Read(0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0x797f7b, 0x247c1e0, 0xc00075ac80)
    /goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x2efaaa0, 0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0x9, 0xc000a1c900, 0xc00075ad30, 0x4051ff)
    /goroot/src/io/io.go:310 +0x88
io.ReadFull(0x2efaaa0, 0xc0006b4de0, 0xc000453df8, 0x9, 0x9, 0xc00075ad10, 0xc000a829d8, 0x795d58)
    /goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc000453df8, 0x9, 0x9, 0x2efaaa0, 0xc0006b4de0, 0x0, 0x0, 0xc00075adf8, 0x797436)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000453dc0, 0xc000a3a0c0, 0x0, 0x0, 0x0)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00075afb8, 0x2c62300, 0xc000246fb8)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1658 +0x9e
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc000234fc0)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1586 +0x76
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).newClientConn
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:660 +0x636

goroutine 25 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc000123c10, 0xc0000382c0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9

goroutine 265 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).periodicLeaderRefresh(0xc00066e000, 0x0, 0xc0002196e0)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:635 +0xe2
github.com/hashicorp/vault/vault.(*Core).runStandby.func5(0xc0006eb260, 0x0)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:359 +0x3c
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a30f80, 0xc000a30fa0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe

goroutine 27 [select, 1 minutes]:
github.com/hashicorp/vault/physical/consul.(*ConsulBackend).runEventDemuxer(0xc0002141e0, 0xc00060b4b0, 0xc0006eaea0, 0xc000042044, 0x16, 0xc0006d65f0, 0xc0006d6600, 0xc0006d6610)
    /gopath/src/github.com/hashicorp/vault/physical/consul/consul.go:689 +0x26d
created by github.com/hashicorp/vault/physical/consul.(*ConsulBackend).RunServiceDiscovery
    /gopath/src/github.com/hashicorp/vault/physical/consul/consul.go:655 +0xfa

goroutine 28 [IO wait, 15 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2e30, 0x72, 0x0)
    /goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000126198, 0x72, 0xc0001c2d00, 0x0, 0x0)
    /goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000126198, 0xffffffffffffff00, 0x0, 0x0)
    /goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000126180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000126180, 0x308, 0x348, 0x203000)
    /goroot/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000c108, 0x7fe650f1e000, 0x3, 0xc000046070)
    /goroot/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).AcceptTCP(0xc00000c108, 0x42a9f1, 0x2c63d60, 0xc000757e90)
    /goroot/src/net/tcpsock.go:247 +0x47
github.com/hashicorp/vault/command/server.TCPKeepAliveListener.Accept(0xc00000c108, 0xc03706584c, 0x3706584c00757eb8, 0x5d09549a, 0xc000757eb8)
    /gopath/src/github.com/hashicorp/vault/command/server/listener_tcp.go:117 +0x2f
crypto/tls.(*listener).Accept(0xc00073ba40, 0xc000757f08, 0x18, 0xc000024900, 0x6bb885)
    /goroot/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc000473a00, 0x2f0dc60, 0xc00073ba40, 0x0, 0x0)
    /goroot/src/net/http/server.go:2826 +0x22f
created by github.com/hashicorp/vault/command.(*ServerCommand).Run
    /gopath/src/github.com/hashicorp/vault/command/server.go:1208 +0x5043

goroutine 377 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc000e0a500, 0xc000515a80)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9

goroutine 266 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*Core).waitForLeadership(0xc00066e000, 0x0, 0xc0006eb500, 0xc000219740)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:535 +0xd56
github.com/hashicorp/vault/vault.(*Core).runStandby.func7(0x2c63b58, 0x0)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:371 +0x45
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc00013c8d0, 0xc000a30fc0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe

goroutine 264 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).periodicCheckKeyUpgrade(0xc00066e000, 0x2f1cba0, 0xc00003c038, 0xc000219620)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:679 +0xca
github.com/hashicorp/vault/vault.(*Core).runStandby.func3(0xc0006eb320, 0x0)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:347 +0x4b
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a30f20, 0xc000a30f40)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe

goroutine 263 [chan receive, 15 minutes]:
github.com/hashicorp/vault/vault.(*Core).runStandby.func1(0xc0006eb3e0, 0x0)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:338 +0x33
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run.func1(0xc000713500, 0xc000a5ce00, 0x2c5e1c8)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:37 +0xbe

goroutine 275 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2c90, 0x72, 0x0)
    /goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00047cd98, 0x72, 0xc000515100, 0x0, 0x0)
    /goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00047cd98, 0xffffffffffffff00, 0x0, 0x0)
    /goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc00047cd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc00047cd80, 0xfd65c0555ec, 0x1dccdd10, 0xc000c09d70)
    /goroot/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000c8b8, 0xf4453cbb29, 0x4d2f4e0, 0xe9)
    /goroot/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc00000c8b8, 0xbf3a73eb89fb3660, 0xf4453cbb29, 0x4d2f4e0, 0x0)
    /goroot/src/net/tcpsock.go:260 +0x47
crypto/tls.(*listener).Accept(0xc0009e4cc0, 0xbf3a73eb89fb3660, 0xf4453cbb29, 0x4d2f4e0, 0x0)
    /goroot/src/crypto/tls/tls.go:52 +0x37
github.com/hashicorp/vault/vault.(*ClusterListener).Run.func1(0xc00047cd00, 0xc00000c8b8, 0x2f1cba0, 0xc00003c038, 0xc0006eb440, 0x2f0dc60, 0xc0009e4cc0)
    /gopath/src/github.com/hashicorp/vault/vault/cluster.go:493 +0xf8
created by github.com/hashicorp/vault/vault.(*ClusterListener).Run
    /gopath/src/github.com/hashicorp/vault/vault/cluster.go:475 +0x486

goroutine 276 [chan receive, 15 minutes]:
github.com/hashicorp/vault/vendor/github.com/oklog/run.(*Group).Run(0xc000a30f00, 0xc000a30fc0, 0xc000219740)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/oklog/run/group.go:43 +0xf0
github.com/hashicorp/vault/vault.(*Core).runStandby(0xc00066e000, 0xc0006eb4a0, 0xc0006eb500, 0xc0006eb560)
    /gopath/src/github.com/hashicorp/vault/vault/ha.go:380 +0x4a8
created by github.com/hashicorp/vault/vault.(*Core).unsealInternal
    /gopath/src/github.com/hashicorp/vault/vault/core.go:1113 +0x25c

goroutine 284 [select, 5 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*ClientConn).roundTrip(0xc000234fc0, 0xc000a2d500, 0x0, 0x0, 0x0, 0x0)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:1060 +0x7db
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0001e8370, 0xc000a2d500, 0xc000c0d600, 0x70000000045136d, 0x41195c, 0x2b4fa19)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:434 +0x150
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Transport).RoundTrip(0xc0001e8370, 0xc000a2d500, 0xc000fc33b0, 0x4d110b0, 0x0)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:396 +0x3a
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0001e8370, 0xc000a2d500, 0xc0006282c1, 0x5, 0xc000471b88)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/transport.go:2515 +0x39
net/http.(*Transport).roundTrip(0xc0004718c0, 0xc000a2d500, 0x203000, 0xc000c0da90, 0x6a81dd)
    /goroot/src/net/http/transport.go:415 +0xd4c
net/http.(*Transport).RoundTrip(0xc0004718c0, 0xc000a2d500, 0xc0004718c0, 0x0, 0x0)
    /goroot/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc000a2d500, 0x2efe960, 0xc0004718c0, 0x0, 0x0, 0x0, 0xc00000c8a0, 0x7fe650f1e000, 0xc000c0db20, 0x1)
    /goroot/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc0005e38c0, 0xc000a2d500, 0x0, 0x0, 0x0, 0xc00000c8a0, 0x0, 0x1, 0x0)
    /goroot/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0005e38c0, 0xc000a2d500, 0x0, 0x0, 0x0)
    /goroot/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc0005e38c0, 0xc000a2d500, 0x4d2f4e0, 0x0, 0xc000c0dd40)
    /goroot/src/net/http/client.go:509 +0x35
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Client).doRequest(0xc00019a300, 0xc00095e960, 0xc000c0ddf0, 0xc00110aee0, 0x16, 0xc00095e960)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/api.go:765 +0xb8
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*KV).getInternal(0xc000c0df20, 0xc000733360, 0xf, 0x0, 0xc000c0df28, 0xc000666990, 0xc00110d780, 0x0, 0x0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/kv.go:165 +0x284
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*KV).Get(0xc000c0df20, 0xc000733360, 0xf, 0xc000c0df28, 0x0, 0x0, 0x0, 0x0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/kv.go:99 +0x7f
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).monitorLock(0xc000514880, 0xc00098ce70, 0x24, 0xc0006eb740)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:368 +0x12c
created by github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).Lock
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:242 +0x572

goroutine 279 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Session).RenewPeriodic(0xc0001da480, 0x2b4c782, 0x3, 0xc00098ce70, 0x24, 0x0, 0xc000219ec0, 0x413554, 0x7fe64eb1b4b0)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/session.go:163 +0x20b
created by github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api.(*Lock).Lock
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/consul/api/lock.go:154 +0xa0b

goroutine 1174 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc0006b28a0, 0xc0006aef80)
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache.runJanitor
    /gopath/src/github.com/hashicorp/vault/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9

goroutine 381 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func1(0xc0008ae240, 0xc0006eb440, 0xc000515b00, 0xc0001c5c00, 0xc0001dc220)
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:164 +0xd0
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:163 +0xe7

goroutine 382 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve(0xc000235340)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:826 +0x52f
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Server).ServeConn(0xc000761a40, 0x2f2b480, 0xc0001c5c00, 0xc000d4bf98)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:438 +0x704
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func2(0xc000515b00, 0xc0001c5c00, 0xc0008ae240, 0xc0001dc220)
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:174 +0xd3
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:173 +0x127

goroutine 449 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2d60, 0x72, 0xc00075b970)
    /goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00047df98, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
    /goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00047df98, 0xc0008b0800, 0x400, 0x400)
    /goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00047df80, 0xc0008b0800, 0x400, 0x400, 0x0, 0x0, 0x0)
    /goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00047df80, 0xc0008b0800, 0x400, 0x400, 0x40875b, 0xc00002e000, 0x26aa540)
    /goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000cd98, 0xc0008b0800, 0x400, 0x400, 0x0, 0x0, 0x0)
    /goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0008d2a20, 0x2efe8a0, 0xc00000cd98, 0x5, 0xc00000cd98, 0x0)
    /goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc0001c5c00, 0x2c63d17, 0xc0001c5d20, 0x0)
    /goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x0, 0x0, 0x0)
    /goroot/src/crypto/tls/conn.go:1145 +0xf1
io.ReadAtLeast(0x7fe64ecc0030, 0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x9, 0xc00075bf28, 0x2, 0x2)
    /goroot/src/io/io.go:310 +0x88
io.ReadFull(0x7fe64ecc0030, 0xc0001c5c00, 0xc00010a038, 0x9, 0x9, 0x0, 0x0, 0x0)
    /goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc00010a038, 0x9, 0x9, 0x7fe64ecc0030, 0xc0001c5c00, 0x0, 0xc000000000, 0xc000712d20, 0xc0006fae08)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00010a000, 0xc00075bee0, 0x2, 0x0, 0x1)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).readFrames(0xc000235340)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:713 +0xa4
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:818 +0x30c

goroutine 478 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fe64ecb2bc0, 0x72, 0xc000c96970)
    /goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001e0418, 0x72, 0xffffffffffffff00, 0x2f01040, 0x4bf7cc0)
    /goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001e0418, 0xc00051cc00, 0x400, 0x400)
    /goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001e0400, 0xc00051cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
    /goroot/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0001e0400, 0xc00051cc00, 0x400, 0x400, 0x40875b, 0xc00002e000, 0x26aa540)
    /goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000c968, 0xc00051cc00, 0x400, 0x400, 0x0, 0x0, 0x0)
    /goroot/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0005e3320, 0x2efe8a0, 0xc00000c968, 0x5, 0xc00000c968, 0x0)
    /goroot/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc000be7880, 0x2c63d17, 0xc000be79a0, 0x0)
    /goroot/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x0, 0x0, 0x0)
    /goroot/src/crypto/tls/conn.go:1145 +0xf1
io.ReadAtLeast(0x7fe64ecc0030, 0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x9, 0xc000c96f28, 0x2, 0x2)
    /goroot/src/io/io.go:310 +0x88
io.ReadFull(0x7fe64ecc0030, 0xc000be7880, 0xc000c7a038, 0x9, 0x9, 0x0, 0x0, 0x0)
    /goroot/src/io/io.go:329 +0x58
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.readFrameHeader(0xc000c7a038, 0x9, 0x9, 0x7fe64ecc0030, 0xc000be7880, 0x0, 0xc000000000, 0xc0006eea80, 0xc0006faf88)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000c7a000, 0xc000c96ee0, 0x2, 0x0, 0x1)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).readFrames(0xc000668540)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:713 +0xa4
created by github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:818 +0x30c

goroutine 487 [select, 15 minutes]:
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func1(0xc0000824e0, 0xc0006eb440, 0xc000515b00, 0xc000be7880, 0xc0001dc220)
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:164 +0xd0
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:163 +0xe7

goroutine 488 [select, 1 minutes]:
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*serverConn).serve(0xc000668540)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:826 +0x52f
github.com/hashicorp/vault/vendor/golang.org/x/net/http2.(*Server).ServeConn(0xc000761a40, 0x2f2b480, 0xc000be7880, 0xc000c9af98)
    /gopath/src/github.com/hashicorp/vault/vendor/golang.org/x/net/http2/server.go:438 +0x704
github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff.func2(0xc000515b00, 0xc000be7880, 0xc0000824e0, 0xc0001dc220)
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:174 +0xd3
created by github.com/hashicorp/vault/vault.(*requestForwardingHandler).Handoff
    /gopath/src/github.com/hashicorp/vault/vault/request_forwarding.go:173 +0x127

goroutine 378 [select, 1 minutes]:
github.com/hashicorp/vault/vault.(*Core).emitMetrics(0xc00066e000, 0xc000dc8960)
    /gopath/src/github.com/hashicorp/vault/vault/core.go:1656 +0x119
created by github.com/hashicorp/vault/vault.(*Core).postUnseal
    /gopath/src/github.com/hashicorp/vault/vault/core.go:1566 +0x35f

 

PPROF & STACKTRACE Environment Variables

Support might request some, or all of the following environment variables be set to aid in the collection of data from issuing a SIGUSR2.

Related Environment Variables

VAULT_STACKTRACE_WRITE_TO_FILE
VAULT_STACKTRACE_FILE_PATH (Optional; string - filepath)
VAULT_PPROF_WRITE_TO_FILE
VAULT_PPROF_FILE_PATH (Optional; string - filepath)

Ex:
VAULT_PPROF_WRITE_TO_FILE=true

Process recap for log collection:

  • set environment variables specified by Support; ie -  VAULT_PPROF_WRITE_TO_FILE=true
  • Add it to the Vault process
  • When the issue is suspected, send SIGUSR2
  • Back out environment variables after collection, if needed (exercise awareness in environment)

If aforementioned environment variable(s) are set, the SIGUSR2 results in the expected stack trace being written to operational logs (or file if environment variable is set), but it also generates pprof data to the (optionally) specified path, or to /tmp directory. Please note, the location of the pprof files is written to the operational logs, as such:

2023-10-23T21:23:11.667Z [INFO]  Wrote pprof files to: /tmp/vault-pprof3151861046

systemd

note: in systemd, this can create a directory in /tmp containing the files, owned by root. The subdirectories should contain the data
 
Ex filepath not specified:

[root@ip-10-13-13-62 vault-pprof3151861046]# pwd
/tmp/systemd-private-8dcd2a5dc9ef4fff93fa6fcde7d04c07-vault.service-zH7RIm/tmp/vault-pprof3151861046
[root@ip-10-13-13-62 vault-pprof3151861046]# ls -ltrah
total 52K
drwxrwxrwt. 3 root  root    35 Oct 23 21:23 ..
-rw-r--r--. 1 vault vault 5.9K Oct 23 21:23 goroutine
-rw-r--r--. 1 vault vault  17K Oct 23 21:23 heap
-rw-r--r--. 1 vault vault  17K Oct 23 21:23 allocs
drwx------. 2 vault vault   69 Oct 23 21:23 .
-rw-r--r--. 1 vault vault  460 Oct 23 21:23 threadcreate
[root@ip-10-13-13-62 vault-pprof3151861046]#

 

When requested by HashiCorp Technical Support, it is helpful to archive these logs and the stack traces and share them in the related ticket.

All file attachments are stored securely and removed when no longer relevant to the associated support issue.

Resources

  1. Where are My Vault Logs and How do I Share Them with HashiCorp Support?
  2. Package signal
  3. Adding Environment Variables to a Vault Process
  4. Data Collection during Vault Incident

Articles in this section

See more

Related articles