Tomcat PermGen – a fix?

While decent folk were asleep, I was profiling Tomcat. (YourKit rocks, funky UI notwithstanding.)

I found that our PermGen woes of old were indeed caused by classes not unloading. I was inspired by this how-to and yes, we do leak an instance of WebappClassLoader which has a reference to every class the app ever uses. Why?, I hear you ask.

There's a java.util.TimerThread named "MySQL Statement Cancellation Timer" that does not terminate when the app is unloaded and so it hangs on to the class loader, which in turn hangs on to every class under the sun. For a rigorous explanation, read the MySQL bug report.

According to that bug report, the MySQL developers capitulated and as of not that long ago, this is the status:

[18 Dec 2009 23:24] Mark Matthews

Fixed for 5.1.11. Unfortunately no great fix exists that lets us keep the cancellation timer shared amongst connection instances, so instead it's lazily created if need be per-instance, and torn down when the connection is closed.

So in theory, we can get rid of this problem by upgrading our MySQL Connector (JDBC driver) from 5.1.10 to 5.1.11.

If I have time over the weekend, I'll be testing the new JDBC driver with GH and monitoring it closely with YourKit to see if the new driver solves the problem.

Labels: mysql, tomcat

0 comments

Remote monitoring of Tomcat

Got memory leaks? Dogged by OutOfMemoryErrors?

JDK 1.5 and later come with jconsole.

First you have to tell Tomcat how to publish it's inner workings for jconsole. This process differs by platform and also depends on how you installed Tomcat.

The typical MacOS X install consists of downloading the zip and expanding it somewhere. The "soomewhere" is your CATALINA_HOME. If that's how you installed Tomcat, go to $CATALINA_HOME/bin and create a file called setenv.sh. Into it put this line:

export CATALINA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=8086 -Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"

On RHEL, Tomcat is typically installed via rpm – I use yum. In that case, you need to add the CATALINA_OPTS to /etc/sysconfig/tomcat6 Leave out the "export" since this file is not a shell script.

Now reboot Tomcat and you're good to go.

Connecting is trivial. Start jconsole. On MacOS X, simply type jconsole at the command line and it opens. For other other platforms, let me know and I will add it.

When jconsole comes up, it will have discovered any local tomcats so simply connect. For a remote tomcat, connect using port 8086 as specified above, and leave user name and password empty.

That's it!

Labels: macos, rhel, tomcat

1 comments

Ah, geek humor
I enjoyed this a lot:

0 comments

Automating your MacPorts upgrades

Porticus is a very nice tool for maintaining your MacPorts. I find it easier than the command line port utility. But mostly, I use it for routinely updating (uh, sorry — “upgrading”) my ports. What if that could be automated?

Here's how.

Create /etc/daily.local with this content

#!/bin/sh
#
# /etc/daily.local
#
# I got this from http://hacks.oreilly.com/pub/h/336

# General parameters
PATH=/bin:/usr/bin:/sbin:/usr/sbin
. /etc/hostconfig

# Update MacPorts, if present
if [ -f /opt/local/bin/port ]; then
    /opt/local/bin/port selfupdate >> macports.log 2>&1 && /opt/local/bin/port upgrade installed >> macports.log 2>&1
fi

This adds the MacPorts update to your daily maintenance regimen. Typically, the Mac's periodic scripts are run at around 3:00 AM and they take care of all kids of things like updating your locate database. The /etc/daily.local file is executed if present.

Naturally, if you would rather do your MacPorts update weekly or monthly, then use /etc/weekly.local or /etc/monthly.local instead.

Create /etc/newsyslog.d/macports.conf with this content

# logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
/var/log/macports.log                   640  7     *    @T00  J

This additional config file for newsyslog says to rotate the macports.log file created by our daily.local. For more info, use man newsyslog

And so on…

Obviously, you can use this trick for other things you'd like to run periodically. For example, you can update Fink be putting this into your daily.local:

# Update Fink, if present
if [ -f /sw/bin/fink ]; then
    /sw/bin/fink selfupdate >> fink.log 2>&1 && /sw/bin/fink update-all && /sw/bin/fink cleanup >> fink.log 2>&1
fi

Labels: macos

0 comments

Reverse lookup will bite you
Atrocious performance in a cluster or other cross-host forwarding/proxying scenario? Maybe your app (or the container) is doing a reverse lookup, timing out on a failed DNS call, and generally tying up the whole app.

We had this situation with Tomcat and Nginx on two separate machines, communicating on a private network. Going directly to Tomcat via the public NIC gave great performance. But going through Nginx yielded horrible performance (20s or so for a page transition in our app.)

Going through an ssh tunnel to the private NIC was equally terrible. I tried tunneling through 3 separate machines on the same private network, and they all exhibited the same slowdown.

The solution is deliciously simple. Just add an entry for the proxy host to the Tomcat host's /etc/hosts file. (Actually, in our case it's a Win 2008 Server, so it's /windows/system32/drivers/etc/hosts). That way, Tomcat (or the app it's running) can do its reverse lookup by hitting the hosts file, and not by waiting for DNS to try and fail.

Labels: tomcat

0 comments

Selenium won't launch Firefox under Snow Leopard

The latest FireFox (3.5.*) includes a version of sqlite that drives Selenium nuts. The basic problem is that Selenium is allowing FireFox's (whacko) version of sqlite to take precedence over the OS-supplied one.

The error message you'll see in the Selenium server log files looks like this:

dyld: Library not loaded: /usr/lib/libsqlite3.dylib
Referenced from: /System/Library/Frameworks/Security.framework/
Versions/A/Security
Reason: Incompatible library version: Security requires version
9.0.0 or later, but libsqlite3.dylib provides version 1.0.0

The fix is to disavow Selenium of that silly notion, which entails setting the DYLD_LIBRARY_PATH correctly. This is all explicated in the Selenium bug report.

The problem has already been fixed in the Selenium trunk, so the solution is to build your own Selenium. It's quite trivial, so here goes:

First, grab the patch, then execute these commands:

svn co http://svn.openqa.org/svn/selenium-rc/trunk .
patch -p0 < snow-leopard.patch
mvn install
cd $WHEREEVER_YOU_KEEP_SELENIUM
cp selenium-server.jar selenium-server.jar.bak
cp selenium-server/target/selenium-server-1.0.2-SNAPSHOT-standalone.jar selenium-server.jar

That takes care of the prob.

Labels: firefox, macos, selenium

0 comments

@#&*#! you; I quit.

After 8 years doing IT support for a local private elementary school, I have finally given up. I feel like whining and complaining about just how I screwed up and making a lot of excuses about why my mission was a failure, but I'd rather distill the underlying causes into something that I can learn from.

So what went wrong? In a word, pain. Well, actually, insufficient pain.

From the outset, I wanted to introduce the staff to better ways of doing things. I wanted them to use modern techniques and tools to make their lives easier. But the problem was they didn't have a good sense of how bad things were. They had no idea how clean and efficient the outside world is in comparison to the ways they were used to. They didn't know they had a problem, much less understand how my fancy techniques and tools would be better by some metric. Sure, they were in some sort of vague discomfort, but they were not in sufficient pain to drive them to call the IT "doctor".

As time went by, I became part of the landscape. I made sure they had a completely wired facility and got some sweet bargains on equipment. I had some success with implementing solutions to problems they didn't know they had by simply going ahead with projects.

But while the staff went about their business, I constantly leapt to address problems big and small, shielding them from the very fact that things were broken or in need of improvement. I was efficient and effective, taking care of things before the staff were even aware that anything needed taking care of. Like an misguided parent, I protected them from every day to day scrape and bump, so they never knew the realities that lay beyond my safety net. I did not allow them to experience pain.

So what will I do differently next time? There's the rub: I don't yet have an answer to that.

Labels: it

1 comments