Monitoring heap space on local JMVs in Java 5

In the last couple of days I have been helping a colleague with a problem. The problem is that they have a couple  of java processes running on a server. These processes go out of memory every no and then and it is important that they can get a notice when this is about to happen.

First I had a look at some monitoring applications.

• jConsole - Had no email functions. It looks like you could install a MBean with mail funtions but I think that has to be done from inside each process and I'm not allowed to change the code.
• Nagios - Is a big infrastructure monitoring utility which have a JMX pluging, but it seemed to overkill.
• Munin - Appears to have some notification ability, but seems hard to set up. 

The solution was to write my own little notification and monitoring program. Often not the best solution but I also saw some value in learning more about JMX monitoring.

I think everybody can agree, it better to fix the memory problem then to just restart the servers when they are running out of memory, but no time for that right now.

Enough of problem, now to the solution =)

To get the memory used by the processes, I needed to summarize the memory used y each process.

As I were going to monitor all the JVMs running om the machine I have to somehow connect to the JMX of every process.

I first stumbled on to the Attach API in java 6 which have a lot of functionality for listing and working with the VMs on the machine. Problem is I was going to build the program for java 5. So I soon dropped this.

Second I tried to find out how jConsole lists the VMs in its startup dialog. Finally I found the class  sun.jvmstat.monitor.MonitoredHost used by the jvmstat program. As Oracle states it's not recommended to use sun.* classes because they are not guaranteed to work in future releases and all platforms.   http://www.oracle.com/technetwork/java/faq-sun-packages-142232.html. But because this is a simple utility meant for use on one server I decided to use them anyway.

So I could use the method activeVms to get a list of all process ID of the active vms.

MonitoredHost.getMonitoredHost("localhost").activeVms();

But how to connect to the JMX of a process only using a PID. Google gave me very many unhelpful suggestions about the Attach API. But finally I found another helpful class in the sun.* classes. sun.management.ConnectorAddressLink

Using the method importFrom I got the JMX address for the JVM process and the hard part was over.

ConnectorAddressLink.importFrom(pid);

Then it was just the simple task of getting the max and used heapspace from the Memory MX Bean.

jmxConnector = JMXConnectorFactory.connect(serviceUrl, null);
  MBeanServerConnection con = jmxConnector.getMBeanServerConnection();
  MemoryMXBean memBean = ManagementFactory.newPlatformMXBeanProxy(con, ManagementFactory.MEMORY_MXBEAN_NAME, MemoryMXBean.class);
memBean.getHeapMemoryUsage().getUsed();
memBean.getHeapMemoryUsage().getMax();

So to the whole process for calculating used heap space on the machine would look something like this

long usedHeapSpace = 0;
for (Object OPid : MonitoredHost.getMonitoredHost("localhost").activeVms()) {
 int dpid = Integer.parseInt(OPid.toString());
 String address = ConnectorAddressLink.importFrom(dpid);
 JMXServiceURL serviceUrl = new JMXServiceURL(address);
 JMXConnector jmxConnector = JMXConnectorFactory.connect(serviceUrl, null);
 MBeanServerConnection con = jmxConnector.getMBeanServerConnection();
 MemoryMXBean memBean = ManagementFactory.newPlatformMXBeanProxy(con
   , ManagementFactory.MEMORY_MXBEAN_NAME
   , MemoryMXBean.class);

 usedHeapSpace += memBean.getHeapMemoryUsage().getUsed();
 jmxConnector.close();
}

Note that all java processes must use the -Dcom.sun.management.jmxremote for the JMX to be enabled.

If there are processes without this flag the resulting calculation will be incorrect.

So to summarize, bad lessons learned, there is a lot of fun stuff in sun.* classes =)