Tomcat PermGen – a fix?

While decent folk were asleep, I was profiling Tomcat. (YourKit rocks, funky UI notwithstanding.)

I found that our PermGen woes of old were indeed caused by classes not unloading. I was inspired by this how-to and yes, we do leak an instance of WebappClassLoader which has a reference to every class the app ever uses. Why?, I hear you ask.

There's a java.util.TimerThread named "MySQL Statement Cancellation Timer" that does not terminate when the app is unloaded and so it hangs on to the class loader, which in turn hangs on to every class under the sun. For a rigorous explanation, read the MySQL bug report.

According to that bug report, the MySQL developers capitulated and as of not that long ago, this is the status:

[18 Dec 2009 23:24] Mark Matthews

Fixed for 5.1.11. Unfortunately no great fix exists that lets us keep the cancellation timer shared amongst connection instances, so instead it's lazily created if need be per-instance, and torn down when the connection is closed.

So in theory, we can get rid of this problem by upgrading our MySQL Connector (JDBC driver) from 5.1.10 to 5.1.11.

If I have time over the weekend, I'll be testing the new JDBC driver with GH and monitoring it closely with YourKit to see if the new driver solves the problem.

Labels: mysql, tomcat

0 comments

Remote monitoring of Tomcat

Got memory leaks? Dogged by OutOfMemoryErrors?

JDK 1.5 and later come with jconsole.

First you have to tell Tomcat how to publish it's inner workings for jconsole. This process differs by platform and also depends on how you installed Tomcat.

The typical MacOS X install consists of downloading the zip and expanding it somewhere. The "somewhere" is your CATALINA_HOME. If that's how you installed Tomcat, go to $CATALINA_HOME/bin and create a file called setenv.sh. Into it put this line:

export CATALINA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=8086 -Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"

On RHEL, Tomcat is typically installed via rpm – I use yum. In that case, you need to add the CATALINA_OPTS to /etc/sysconfig/tomcat6 Leave out the "export" since this file is not a shell script.

Now reboot Tomcat and you're good to go.

Connecting is trivial. Start jconsole. On MacOS X, simply type jconsole at the command line and it opens. For other other platforms, let me know and I will add it.

When jconsole comes up, it will have discovered any local tomcats so simply connect. For a remote tomcat, connect using port 8086 as specified above, and leave user name and password empty.

That's it!

Labels: macos, rhel, tomcat

1 comments

Reverse lookup will bite you
Atrocious performance in a cluster or other cross-host forwarding/proxying scenario? Maybe your app (or the container) is doing a reverse lookup, timing out on a failed DNS call, and generally tying up the whole app.

We had this situation with Tomcat and Nginx on two separate machines, communicating on a private network. Going directly to Tomcat via the public NIC gave great performance. But going through Nginx yielded horrible performance (20s or so for a page transition in our app.)

Going through an ssh tunnel to the private NIC was equally terrible. I tried tunneling through 3 separate machines on the same private network, and they all exhibited the same slowdown.

The solution is deliciously simple. Just add an entry for the proxy host to the Tomcat host's /etc/hosts file. (Actually, in our case it's a Win 2008 Server, so it's /windows/system32/drivers/etc/hosts). That way, Tomcat (or the app it's running) can do its reverse lookup by hitting the hosts file, and not by waiting for DNS to try and fail.

Labels: tomcat

0 comments

Unreliable Tomcat
Tomcat has had a memory leak for at least a decade that I am aware of. If you deploy by copying a war file to the webapps directory, sooner or later Tomcat runs out of memory. It's hard to kill tomcat when it's wedged in this way, since the standard way of shutting down is to send a request (via the shutdown script). If Tomcat is too ill to listen to requests, you can't shut it down. Usually the solution is to find its pid and kill it from the command line. How to automate this? That is, I want to reboot tomcat automatically, even when it's wedged. It sure would be nice if there was a pid file... Well, there can be a pid file. Set CATALINA_PID to the path to a pid file you want tomcat to use, and the catalina.sh script (called by startup.sh) will do the right thing. Then, in order to get shutdown.sh to make use of the pid file, simply stipulate -force.

Labels: tomcat

3 comments