Supervising JVM with Dynamic Attach and Golang

This post is a continuation of a series focused on monitoring & supervising untrusted JVM process that runs inside docker container provided by the third-party provider. In the first article I described how to share process namespace between containers, this article shows how to get a better understanding of the JVM process that we do not control. Tempted to know what “better understanding of JVM” means? Read on! Presented techniques are especially useful when we do not have direct access to the docker image internals (binary, configuration, etc.) we run. That’s a very rare and not so secure scenario – no doubt about that, however, such situations still happen.

(Dynamic) Attach API

Dynamic attach is quite poorly documented mechanism that allows to “attach” into already running JVM process by executing a few simple, language-agnostic steps. The only required tools are Unix Sockets and ability to send signals (SIGQUIT in this case) to the target process. Java’s Attach API uses that technique to communicate with external JVM. This article shows how to get a better understanding of the JVM process by attaching to it using a program written in Golang with Linux and HotSpot-based JVM. The Linux and HotSpot parts are very important here – implementation of JVM Dynamic Attach differs between operating systems, not to mention that it’s a very internal tool available only in certain JVM implementations. I will link to the source code of Amazon Corretto 8 distribution and explain internals so we could know not only how it works but also how it is implemented.

This file contains all supported commands and its handlers, for now, available commands are:

  • agentProperties
  • datadump
  • dumpheap
  • load (JVMTI agent – there will be an article about that soon!)
  • properties
  • threaddump
  • inspectheap
  • setflag
  • printflag
  • jcmd

Attaching to the JVM from the caller perspective

To attach to live JVM, one needs to fulfill a few simple requirements:

  • Be the same user who started the main JVM process
  • Create .attach_pid$pid (eg. .attach_pid12) in the working directory of the process or tmp directory. (eg. /proc/$pid/root/tmp)
  • Send SIGQUIT to the main (target) process.
  • Connect to the socket created by the main process (eg. /tmp/.java_pid12) and send proper commands.

Step by step explanation:
First step is a kind of security gate – we don’t want to give access to the attach mechanism to every user. This is why in the second step JVM checks effective permissions of the user that created “attach file” – “simple check to avoid starting the attach mechanism when a bogus user creates the file” as stated in source code is enough to explain its destiny. The third step is the real trigger for our main process – target JVM intercepts our SIGQUIT and initializes attach mechanism. In step four we have socket ready to receive our commands. That’s a bit simplified description of what happens under the hood, however, whole attach mechanism isn’t that difficult – I recommend going through links I pasted and check what’s going on there.

Implementation of Dynamic Attach API in HotSpot JVM

While I described most of the flow details in the above section, there are still some interesting things going on under the hood. First such thing is that the whole attach mechanism can be disabled by adding -XX:+DisableAttachMechanism to the start parameters of JVM. From the other hand, Dynamic Attach (read: ready to use Unix socket) can be obtained in a few ways:

  • Lazily by executing a few steps(file+SIGQUIT) presented in the previous paragraph.
  • Eagerly during JVM start once -XX:+StartAttachListener option was passed.
  • Again eagerly during JVM start once -XX:+ReduceSignalUsage option was passed.

The protocol used to communicate between peer and target process is described in just a few sentences – quite small but still powerful enough. Caller sends command to the socket, target JVM replies to the same socket allowing caller to read its response. To my best knowledge, there is only one version of that API yet please remember that we still have to send the protocol version (1) in each request.

Demo

Theory is quite interesting but let’s do something cool and real. My real use case was that I wanted to see all JVM properties and periodically fetch histogram (all classes, number of instances + their size in memory). I implemented it some time ago for my customer, it allows to gather lots of information about JVM itself without it even knowing we mess around. Since in mentioned use case we couldn’t modify the original docker image(and thus modify JVM config), that was the only option to get basic information of JVM process.

I have created a repository with a demo code that does the same things I explained above, it could be found here. My code is heavily inspired by apagin’s jattach – the only differences are:

  • error handling – my code is just a demo, c’mon
  • language – I used Golang instead of C as my production-ready supervisor was created in Go.
  • minor fixes required to run it in Docker (eg. use /proc/$pid/root/tmp instead of just /tmp – beware! these are two different paths when running two containers that share PID namespace!)

For the demo purposes I prepared following docker-compose file definition:

version: '3.7'
services:
  third-party-service:
    image: "tomcat"
  supervisor:
    pid: "service:third-party-service"
    build:
      context: ./

third-party-service here is the docker image with JVM process (just plain Tomcat for demo purposes) that we do not control whereas supervisor is a container that contains binary that is the client of Dynamic Attach: it waits 5 seconds, triggers Dynamic Attach of target JVM, fetches histogram from third-party-service (this process runs in different docker container!) and prints it to the standard output. These steps execute in a loop meaning that every five seconds we will see new data being displayed. Even though we don’t do many things in this demo, it still looks very appealing to me that such capabilities are available. Combining different commands listed above might give you powerful tool that can deeply integrate with any JVM deployments.

Closing words

Do you imagine doing something like this for Python or Golang? One could argue that JVM is old, bloated with legacy stuff but not-so-obvious business features like this reminds me why I like this platform. It worth pointing out that I talked about JVM in general so this feature isn’t tied to Java only. Kotlin, Scala and all other JVM-based languages can benefit as well.

I can’t wait to publish my next articles about Java Agents and JVMTI!

Leave a Reply