There is a tool called ksniff that uses wireshark and tcpdump to capture network activity at the pod level.
With this tool installed on a Kubernetes environment, you would be able to see the packets sent, if they’re being rejected, and possibly more information that could help support troubleshoot Consul related issues.
Installing and testing out Ksniff
- In order to use ksniff, you’ll need to download wireshark if you don’t have this installed already. The wireshark GUI and command line will need to be installed for you to utilize ksniff. The best way to install both the wireshark GUI and command line(tshark) is here.
- Once installed, you’ll be prompted with a few options. You will want choose both “Add Wireshark to the system path.pkg” and “Install ChmodBPF.pkg”
- After wireshark is successfully installed, you can follow the instructions to install ksniff in your environment using this link https://kubesandclouds.com/2021-01-20-ksniff/. Once done, you can test this out on your pods. It will look like the following(pod terminal on the left, wireshark on the right):
Execute Ksniff by container
Ksniff allows granularity when it comes to packet capturing by container. For example, if you have a pod with an envoy sidecar container, you can check envoy network activity with the following command:
kubectl sniff static-client-869c6fd9f7-7pbcv -c envoy-sidecar -p
The -c flag is used to specify the container name.
When executed with the -p flag, ksniff will create a new pod on the remote kubernetes cluster that will have access to the node docker daemon.
ksniff will then use that pod to execute a container attached to the target container network namespace and perform the actual network capture. The -p flag needs to be used with the kubectl sniff command to be able to capture packets.
Verifying mTLS in Consul Connect Service Mesh using Ksniff
Lets say you have two pods, a dashboard service and a counting service communicating over mTLS(through Envoy). We can check that the TLS handshake is working by checking packet activity in the dashboard service. You would use the following command to have wireshark opened via ksniff:
kubectl sniff dashboard-55d8d5b447-pfgrq -p
- dashboard-55d8d5b447-pfgrq is the name given to my dashboard service, so you'll need to fill in your pod name there. Once ksniff is executed, it will show INFO logs of the ksniff pod being created and an attempt to open wireshark.
- When wireshark opens, you can filter to only see TLS handshakes by using
tls.record.content_type == 22.
- Depending on how/when your service makes requests, you may not initially see any packets with this filter. You can manually make a request from pod to pod to see this process. In my example, I can call the counting service(the upstream) from the dashboard pod with the following command:
kubectl exec po/dashboard-55d8d5b447-pfgrq -- wget -qO- localhost:9003
- The TLS handshake is shown as successful in wireshark. The Certificate, Server Key Exchange, Certificate Request, and Client equivalents indicate that an mTLS session is being started. Note that you'll only see the lines referencing those if you have the packet capture already running when the two services first communicate. Those actions only happen on their first interaction and subsequent requests will not contain the initial handshake that contains the certificates.
We have shown what a successful TLS handshake looks like, but what about an unsuccessful one? Any TLS handshake interrupted with a Fatal level alert indicates that Envoy is most likely running into issues with communication. Take a look at the following screenshot for example:
- Before the Client key exchange is attempted, the Client(downstream) does not accept the Certificate from the server(Envoy upstream) due to an expired certificate. The connection is reset due to that. In this situation, it's important to check the envoy certificates and make sure it is being auto renewed. This scenario is one of a few TLS handshake issues that could occur. Ksniff is an awesome tool that helps with troubleshooting these issues.