Troubleshooting OpenJDK applications on Kubernetes at the command line with jattach

There are a lot of things to love about Kubernetes. It's flexible, scalable, and has a robust ecosystem. However, battle-scarred engineers may find themselves missing one thing in particular: the command line. In the old days of bare-metal servers, troubleshooting a misbehaving application was an ssh login away. However, Kubernetes - with its abstractions of nodes, pods and containers - makes that approach somewhat less straightforward. Furthermore, our applications now run in spartan, containerized environments, often lacking many familiar libraries and tools we took for granted.

To facilitate efficient deploys, Kubernetes encourages using the lightest Linux distribution possible, and recent licensing changes have caused many organizations to adopt OpenJDK as their Java runtime. Hence, Alpine Linux with OpenJDK has become a standard base image for many Java applications. In bare-metal environments running Oracle Java, we relied on jcmd and jstack for troubleshooting. However, it turns out that jcmd doesn't work so great on Alpine Linux. So, engineers supporting these production environments have a few hurdles to overcome.

jattach to rescue

Fortunately, there is a solution: jattach is an open source project that replicates the functionality of common Java SDK tools. These tools, such as jcmd and jmap implement the well documented Hotspot Attach API. So, it is able to reproduce from scratch the functionality of jcmd and friends - awesome!

For jattach to be available, we need to add it to our docker image. Here is an example Dockerfile snippet that installs jattach using the Alpine Linux package manager, apk

RUN apk add --no-cache openjdk8-jre="$JAVA_ALPINE_VERSION" jattach="$JATTACH_VERSION"

...where $JAVA_ALPINE_VERSION and $JATTACH_VERSION are the versions of each you wish to install, defined earlier in your Dockerfile.

Using jattach

Now, our deployed application will have jattach available for troubleshooting. After connecting to the container with kubectl exec, we can grab a thread dump from the Java application and see what it's up to.

First, we connect to the pod, running /bin/sh to obtain an interactive shell:

$ kubectl exec --stdin --tty alpha-app-65bb6db4f9-bw9sb -- /bin/sh

Note that in this example, we assume our Kubernetes pod hosts a single container (see the kubectl exec documentation for more about this topic). In such deployments, our Java process will have a pid of 1. We can confirm this with the familiar ps command:

/nv $ ps
PID   USER     TIME  COMMAND    
1 novusapp 38:46 java -XX:InitialRAMPercentage=80.0 -XX:MaxRAMPercentage=80.0     
21629 novusapp  0:00 sh    
21648 novusapp  0:00 ps

We further confirm that jattach is available, per our Dockerfile:

/nv $ jattach 1
jattach 1.5 built on May 12 2019
Copyright 2018 Andrei Pangin 

Usage: jattach   [args ...]

Next, we inspect the available jcmd commands provided by OpenJDK...

/nv $ jattach 1 jcmd help

Connected to remote JVM
Response code = 0

The following commands are available:
VM.native_memory
ManagementAgent.stop
ManagementAgent.start_local
ManagementAgent.start
VM.classloader_stats
GC.rotate_log
Thread.print
GC.class_stats
GC.class_histogram
GC.heap_dump
GC.finalizer_info
GC.heap_info
GC.run_finalization
GC.run
VM.uptime
VM.dynlibs
VM.flags
VM.system_properties
VM.command_line
VM.version
help 

For more information about a specific command use 'help '

/nv $

...and using the familiar jcmd Thread.print command, we can peek inside the Java application, piping output to less for paging goodness:

/nv $ jattach 1 jcmd Thread.print | less

Connected to remote JVM
Response code = 0
2021-06-10 20:04:01
Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):

"akka:event-driven:dispatcher:2b4d5f60-ca1f-11eb-8486-6eefc1f2ccf0-13746" #23035 prio=5 os_prio=0 tid=0x0000558b620be800 nid=0x5a2d in Object.wait() [0x00007f007ca98000]
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
  at java.lang.Object.wait(Native Method) 
  at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:268) 
  - locked <0x00000005ca6c46f0> (a org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl) 
  at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:389)

‍

Dumping JVM threads to a file

Running jattach from an interactive shell is familiar and comforting, but perhaps we need to quickly capture the threads from a JVM for later analysis. Again, we can accomplish this with a single kubectl exec command, this time invoking jattach directly and redirecting output to a file:

$ kubectl exec --stdin --tty alpha-app-65bb6db4f9-bw9sb -- /usr/bin/jattach 1 jcmd Thread.print > threads.txt

The file threads.txt will contain the desired Java thread dump.

Triggering bulk JVM garbage collection

In an earlier DevOps era, we might have used Chef and knife to force JVM garbage collection across all applications in a production environment. With Kubernetes, can accomplish the same result using kubectl, jattach and xargs.

First, we can obtain a list of pods by label (or other attribute specific to our application):

$ kubectl get pods -l='app.kubernetes.io/instance='=alpha-app -o name
pod/alpha-app-65bb6db4f9-bw9sb
pod/alpha-app-65bb6db4f9-sqvg7

...and then using xargs we execute jcmd GC.run across these pods, specifying the -P flag to expedite the operation with parallel execution:

$ kubectl get pods -l='app.kubernetes.io/instance='=alpha-app -o name | xargs -P 8 -I{} kubectl exec {} -- jattach 1 jcmd GC.run
Connected to remote JVM
Response code = Connected to remote JVM
Response code = 0

Connected to remote JVM
Response code = Connected to remote JVM
Response code = 0

$ 

‍

Bonus content: k9s

While kubectl is certainly versatile, recalling all its permuations can be somewhat burdensome. Happily, Fernand Galiana (@kitesurfer) has developed an extraordinarily useful tool for working with Kubernetes clusters: k9s ‍

https://k9scli.io/.

After using k9s, it's rather difficult (if not impossible!) to go back to using only kubectl. With k9s, we can quickly browse, filter and sort all our cluster resources, including pods, cronjobs, events and logs. Even better, k9s can obtain a container shell with a stroke of the "s" key - extremely convenient and a huge upgrade from the days of juggling multiple ssh and screen sessions.

Like kubectl, k9s is cross-platform. As proof, witness this screenshot of k9s running natively in Windows Terminal:

‍

Links

jattach on GitHub: https://github.com/apangin/jattach

Kubernetes documentation for kubectl exec: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec

k9s: https://k9scli.io/