Blogging Roller

In addition to offering a week of high-tech training for $95, TechEngage has also arranged a <a href= "http://www.techengage.org/shootout.aspx">Dot-Net vs. Java shootout on Wednesday, March 12, 6:30 PM, at the N.C. State University College of Management in Raleigh, N.C. Representatives from Microsoft and Borland will face off against representatives from IBM, JBoss, and Sun.

Matt Raible has successfully tested Russell Beattie's <a href= "http://www.russellbeattie.com/notebook/20030225.html#231612">world-famous <a href= "http://www.manywhere.com/Moblogger.html">ManyWhere Moblogger with Roller.

If you read the Java blogs, you've probably seen Bill Venners' excellent How to interview a Programmer article by now. If not, check it out. My favorite quote is below:

Scott Meyers: I hate anything that asks me to design on the spot. That's asking to demonstrate a skill rarely required on the job in a high-stress environment, where it is difficult for a candidate to accurately prove their abilities. I think it's fundamentally an unfair thing to request of a candidate.

This one is a close second:

Bruce Eckel: I ask candidates to create an object model of a chicken.

I mean, how can you argue with Scott Meyers and how can you object to a chicken?

Speaking of arguing with Scott Meyers... I worked at one of those phone company research labs back in the 90's and we had enough money to bring in Scott Meyers to teach us all about C++. Scott was explaining how you can overload operators and you can even overload the equals sign, when one student raised his hand. The student explained that there were some situations in telecommunications systems where A was equal to B, but B was not equal to A. Scott immediately objected, of course, but the student went off into some jargon-filled explanation of his particular problem domain. Scott let the student finish and then said, "if you were to overload the equals operator so that A was equal to B, but B was not equal to A, then I would want to kill you." That was the end of the argument.

To prepare for the move to <a href= "http://hibernate.bluemars.net">Hibernate, I've been refactoring the current implementation of the Roller business tier. There is a lot of code in the Castor implementation of the <a href= "http://www.rollerweblogger.org/javadoc/org/roller/model/package-summary.html">persistence manager interfaces that is generic enough to be used in the Hibernate impementation. So, I'm moving that code up into abstract base classes that may be used by both implementations.

Looks like another great Eclipse-oriented talk at the Research Triangle Park Websphere Users Group (RTP-WUG) meeting Tuesday. Sridhar Iyengar of IBM and Randy Miller of TogetherSoft will speak about Model Driven Architecture and the Eclipse Modelling Framework. Full details on the RTP-WUG web site.

As FreeRoller users and readers know, FreeRoller is having problems keeping up with the load of 500+ weblogs and the 101-list traffic. Sometimes the main page takes several minutes to load and often the system just collapses under the stress. According to Anthony Eden, who runs FreeRoller, the system is powered by one little eMachines box. I could use that as an excuse, but, regardless of the hardware, the poor performance of FreeRoller reflects poorly on the Roller software.

I considered using a profiler to figure out the root cause of the performance problem, but a profiler is difficult to setup for a Servlet app, there's a learning curve, and there are only so many Roller hours in my weekend. Plus, I had a gut feeling that the Roller main page, index.jsp, was causing the slow-downs.

Looking at the code behind index.jsp, I found that it uses several queries to build a list of weblogs, hit counts, and the last update time for each weblog. The queries are not complex, but because they are executed using an Object-Relational (O/R) persistence framework (Castor) each query results in the creation of one object per row returned. FreeRoller has over 500 weblogs, so each refresh of the index.jsp page results in the creation of thousands of objects. Even with page caching in place to reduce the frequency of refreshes, this has the potential to bog down the whole system.

I rewrote the code behind index.jsp to use only one query and I also added a limit to that query so that a Roller administrator can limit the number of weblogs displayed on the main page. Next, I did some load testing with JMeter. I set up a JMeter test plan to use 3 threads to hit index.jsp, a 20-second ramp up, and a 300 ms delay between requests. I did this testing on a 1.5Ghz Athlon, 768MB RAM box running Windows XP, Java 1.4.1, and Tomcat 4.1.18. Here are the results:

                   Avg.  Dev.  Throughput  Note

1) Old index.jsp 1051ms  8629  120/min     Heavy CPU usage
2) New index.jsp  248ms  1326  248/min     Limit 500 weblogs
3) New index.jsp   93ms   186  247/min     Limit 30 weblogs
4) TC index.jsp    90ms    72  337/min     Tomcat example page

The old index.jsp would peg the CPU at 100% for minutes at a time during cache refreshes. I'm not sure why, but I suspect that multiple requests were simultaneously refreshing the cache (I'm not sure how OSCache hanldles this). During test #1, Roller was so slow as to be almost unusable. Test #2 and #3 show that the new index.jsp is a great improvement. With a limit of 500 weblogs, the CPU usage is very heavy during a cache refresh but doesn't get to 100%. With a limit of 30 weblogs, the cache refresh does not have a noticable effect on the CPU. Test #4 tests the Tomcat examples page, it is included only as a baseline for comparison.

I made the above changes in the Roller 0.9.6 code branch so that they can me applied to FreeRolller right away. I'm still not happy about Roller performance and memory usage. I'm sure there is plenty of room for improvement. My next experiment is implementing the Roller business tier with the Hibernate O/R framework. I'm curious to see how it compares to Castor.

Lance is experimenting with Prevayler by using it and JXPath to implement the Roller business tier. Cool stuff. I'm gearing up for a Hibernate implementation; more about that later.

Lots of interesting features, but the one I find myself wanting most is an un-installer.

I found this very interesting read on the history of AWT, Swing, and SWT fom an undisclosed source via Roller user <a href= "http://blog.xesoft.com/page/jon.lipsky/20030221#a_good_read_about_history">Jon Lipsky's blog. Here is a tasty excerpt:

Alan Williamson's mysterious "source close to IBM": At IBM we hated Swing from day one. Big, buggy, and looks [like] crap. Initially our tools such as VisualAge for Java were all written in Smalltalk ( which used native widgets ) so when we started to migrate these to a Java codebase we need a widget set. All of the IBM developers are the same crowd who used to work with Smalltalk, and we reluctantly under management orders built our WebSphere Studio tools using Swing. It was a terrible, buggy, monster. In our initial previews when it was demo'd against Microsoft Visual Studio products all our users hated it just because of how it looked, never mind what it let you do. Most shoppers don't like to get in car that looks and smells terrible, even if it does have a nice engine.

UPDATE: <a href= "http://blog.xesoft.com/page/jon.lipsky/20030221#a_good_read_no_truth">Jon Lipsky was contacted by somebody at Sun who claims there are many and major inaccuracies in the above story.

(via Lance) This quote from Free Software Foundation lawyer Eben Moglen seems to clarify the issue of using an LGPL jar in a non-GPL application (such as Roller). As long as the jar is LGPL, rather than GPL, you can include it in your application and then license your application however you choose. Here is the key quote:

Eben Moglen: If the author of the other code had chosen to release his JAR under the Lesser GPL, your contribution to the combined work could be released under any license of your choosing, but by releasing under GPL he or she chose to invoke the principle of "share and share alike."

Some conversations at the last RTP bloggers lunch left some doubt in my mind about LGPL, but the above quote clears it up nicely.

Roller uses the Castor persistence framework. FreeRoller can be very slow.

Are the two things related? I have no real empirical evidence (yet) to prove a link, but I did find an interesting link of a different sort in my referer logs this morning to a blog entry by (Roller user) Matthew Porter. Back in December 2002, Matthew chose Hibernate instead of Castor for the Java Lobby Community Port (JLCP) project. Here's why:

Persistent Framework Choice for JLCP: Castor was not chosen for two primary reasons. The first is the lack of development of Castor in the past year. In addition, one the tests we performed at DMI, Castor was significantly slower than other PFs- to the point where it was intolerable. The first reason and recent tests led me to believe that the situation regarding speed had not changed.

Also, an interesting link was posted in a comment on my Long Transactions post yesterday. This is a pretty interesting article:

O/R Mapping with Castor JDO in the Real World: Castor holds up to its promises in simple testing and trial runs. However, it has proven to fall short in some practical issues with our application of about twenty-five data classes and as many tables. Most of our problems come from the need to hold onto objects across transactions and perform complex updates.

I'm tempted to rewrite the Roller backend using Hibernate just for the hell of it, but I really should to do some profiling of Roller to see where the problem lies, don't you think? I guess I could do a 30-day eval of OptimizeIT or JProbe, but I would be happy to hear your recommendations for free and/or open source profiling tools. Got any?

From the Roller-dev list:

Lance: Castor is *supposed* to be caching objects for us so that we don't need to make repeated calls to the database. I suspect the way we are using Castor may not be optimal, though I really don't know enough to suggest improvements.

Webapps seem to cause problems for O/R frameworks. In an O/R framework, things seem to work properly only inside of a O/R "transaction" or "session." In a webapp, you get an object from the database in one session, close that session, allow the user to modify the object in an HTML form, then start another session to update the object in the database. Castor calls this a "long transaction." I wonder how these long transactions effect Castor's ability to cache efficiently.

I've known about this Google buys Pyra thing since Saturday night, but it seems like everybody I know is forwarding me articles about it from MSNBC, CNN Money, and other mainstream news sources. These people are blogmuggles, they don't read my weblog and don't have their own weblogs, but they can clearly see that Google buying Pyra makes this The Year That Punk Broke for blogging. Damn, that was 12 years ago. Anyhow... I'm gonna go with that analogy. I'm hoping that Rollerville is going to be the next Seattle and Roller is going to be the next Superchunk, but I'm not holding my breath.

I upgraded the Roller main branch (and this site) to Struts 1.1b3, Castor 0.9.4.3, Xerces 2.3, and Ant 1.5.1. I'll pop in Struts 1.1 RC1 when it arrives. I think we are pretty close to a Roller 0.9.7 release. Lance has finished replacing all of the old Roller "macros" with VelociMacros (don't worry, we'll keep the old macros around for a while) and we only have a couple of open issues left to go. I do want to do a little work on the comments feature before we release. Lance did a great job on comments, but I want to polish it up a little before it goes out.

Looks like we might be in for <a href= "http://www.nbc17.com/news/1978469/detail.html">another terrible and deadly ice storm here in North Carolina. We are expecting below freezing temperatures and heavy rain from 3PM to 9PM today.

As you can see, I've returned to mismanaging my time (i.e. blogging). I've finished the 2nd drafts of my Wrox Professional JSP chapters. The reviewers seemed to like my 1st drafts, especially the Performance and Debugging chapter, but that did not stop them from making hundreds of comments and thus burning two perfectly good weekends.

Speaking of Wrox, I picked up JBoss 3.0 Deployment and Administration Handbook by Meeraj Kunnumpurath the other day. I like the new small and concise handbook format. You don't need a wheel-barrow to move it around the office like most Wrox books. The book appears to be perfect for somebody, like me, who already knows J2EE and just needs to understand how to configure and deploy to JBoss.

<a href= "http://www.evhead.com/archives/2003_02_01_archive_default.asp#104537225413624191">Holy crap is right. <a href= "http://weblog.siliconvalley.com/column/dangillmor/archives/000802.shtml#000802">Dan Gillmore speculates that Yahoo, Microsoft, and AOL will also be buying or building their own weblogging systems. Congratulations to Evan Williams and Pyra team. Five people supporting 1.1 million webloggers? Amazing.

That's what <a href= "http://joi.ito.com/archives/2003/02/15/my_position_on_warblogging.html#004088">Joi Ito says and I have a hard time disagreeing, but I sure as hell don't know what to do. I listened to the U.N. deliberations in full yesterday and found myself agreeing with just about everything that was said by the French, the Brits, the Russians, and the U.S. That is how messed up I am.

I've been reading warbloggers, peacebloggers, <a href= "http://pages.prodigy.net/thomasn528/blog/2003_02_09_newsarcv.html#89055906">peacebloggers turned warbloggers, the liberal media, the Moonie press, and even <a href= "http://www.boosman.com/blog/2003_02_01_blogarchive.html#90313256">pseudo-bloggers ;-) My mind is a muddled mess. I'm glad I'm not the one making the decisions, but I sure do wish we had somebody other than Shrub, Jr. with the final word.

All this B.S. about France being ungrateful is just plain silly. As Russell points out, we'd be driving on the left and drinking warm beer if not for France.

<img src="http://rollerweblogger.org/resources/roller/oilwar-sm.jpg" alt="collage of Simulations Publications, Inc. OIL WAR game">

The above image is not a political commentary, it is a collage that I created of a wargame that I played as a kid, Oil War: a "game of hypothetical warfare in the Middle Eastern Oil states" by Simulations Publications, Inc.