Troubleshooting OpenJDK applications on Kubernetes at the command line with jattach
Migrating Java applications to OpenJDK and Kubernetes can pose challenges. Here, we demonstrate using kubectl and jattach for interactive troubleshooting.
There are a lot of things to love about Kubernetes. It's flexible, scalable, and has a robust ecosystem. However, battle-scarred engineers may find themselves missing one thing in particular: the command line. In the old days of bare-metal servers, troubleshooting a misbehaving application was an ssh login away. However, Kubernetes - with its abstractions of nodes, pods and containers - makes that approach somewhat less straightforward. Furthermore, our applications now run in spartan, containerized environments, often lacking many familiar libraries and tools we took for granted.
To facilitate efficient deploys, Kubernetes encourages using the lightest Linux distribution possible, and recent licensing changes have caused many organizations to adopt OpenJDK as their Java runtime. Hence, Alpine Linux with OpenJDK has become a standard base image for many Java applications. In bare-metal environments running Oracle Java, we relied on jcmd and jstack for troubleshooting. However, it turns out that jcmd doesn't work so great on Alpine Linux. So, engineers supporting these production environments have a few hurdles to overcome.
jattach to rescue
Fortunately, there is a solution: jattach is an open source project that replicates the functionality of common Java SDK tools. These tools, such as jcmd and jmap implement the well documented Hotspot Attach API. So, it is able to reproduce from scratch the functionality of jcmd and friends - awesome!
For jattach to be available, we need to add it to our docker image. Here is an example Dockerfile snippet that installs jattach using the Alpine Linux package manager, apk
...where $JAVA_ALPINE_VERSION and $JATTACH_VERSION are the versions of each you wish to install, defined earlier in your Dockerfile.
Now, our deployed application will have jattach available for troubleshooting. After connecting to the container with kubectl exec, we can grab a thread dump from the Java application and see what it's up to.
First, we connect to the pod, running /bin/sh to obtain an interactive shell:
Note that in this example, we assume our Kubernetes pod hosts a single container (see the kubectl exec documentation for more about this topic). In such deployments, our Java process will have a pid of 1. We can confirm this with the familiar ps command:
We further confirm that jattach is available, per our Dockerfile:
Next, we inspect the available jcmd commands provided by OpenJDK...
...and using the familiar jcmd Thread.print command, we can peek inside the Java application, piping output to less for paging goodness:
Dumping JVM threads to a file
Running jattach from an interactive shell is familiar and comforting, but perhaps we need to quickly capture the threads from a JVM for later analysis. Again, we can accomplish this with a single kubectl exec command, this time invoking jattach directly and redirecting output to a file:
The file threads.txt will contain the desired Java thread dump.
Triggering bulk JVM garbage collection
In an earlier DevOps era, we might have used Chef and knife to force JVM garbage collection across all applications in a production environment. With Kubernetes, can accomplish the same result using kubectl, jattach and xargs.
First, we can obtain a list of pods by label (or other attribute specific to our application):
...and then using xargs we execute jcmd GC.run across these pods, specifying the -P flag to expedite the operation with parallel execution:
Bonus content: k9s
While kubectl is certainly versatile, recalling all its permuations can be somewhat burdensome. Happily, Fernand Galiana (@kitesurfer) has developed an extraordinarily useful tool for working with Kubernetes clusters: k9s
After using k9s, it's rather difficult (if not impossible!) to go back to using only kubectl. With k9s, we can quickly browse, filter and sort all our cluster resources, including pods, cronjobs, events and logs. Even better, k9s can obtain a container shell with a stroke of the "s" key - extremely convenient and a huge upgrade from the days of juggling multiple ssh and screen sessions.
Like kubectl, k9s is cross-platform. As proof, witness this screenshot of k9s running natively in Windows Terminal:
jattach on GitHub: https://github.com/apangin/jattach
Kubernetes documentation for kubectl exec: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec