Blogging Roller

Congratulations and good luck to fellow RTPBlogger Joe Gregorio who recently launched his new company BitWorking, Inc. to do custom systems development.

Big round of applause for Anthony Eden and Min Idzelis who have just joined the Roller project. Hip, hip, hoooooray! By the way, Min lives right in the Triangle. I'll have to convince him to come out to the next RTP Bloggers lunch.

Charles Miller - Version Numbers and You: Too many Open Source projects treat Version 1.0 as some kind of Holy Grail that can only be reached when the project is perfect. I find that highly annoying, because it makes it really, really difficult to tell a sketchy alpha from production code that is just still in pre-1.0 because the author wants it to do everything.

If we add multiple categories per post, I'd be happy calling the next release Roller 1.0 rather than Roller 0.9.8.

Roller has been honored by the BileBlog. Most of Hani's criticisms are valid, but none new or especially insightful, at least not to me. I'm a little surprised he did not attack the sucky UI as it is a prime target. Also, it is interesting that he zoomed in on the multiple connections per request problem, I think he had an inside tip ;-) That is what we are working on this week. I sincerely hope he is correct and that fixing the multiple connection per request (enhancement #9 from my earlier post) solves the FreeRoller current performance problems.

As you may or may not remember, I started developing Roller, in part, to teach myself about database driven web applications in the world of Java and J2EE. So, the history of Roller performance is also the history of my school of hard knocks education in Java webapp performance. I'm not sure if anybody can learn anything of value from this, but I feel like reviewing the story, and this is my blog, so that's what I'm gonna do. If you don't like it, you know where the back button is. Here is the story so far.

In the beginning, I implemented Roller using EJB. Every Roller business object was an Entity Bean and every Roller business tier interface was a Session Bean. I learned a lot about EJBs and XDoclet and everything was just fine, but Roller, or Homeport as I used to call it, took 20 seconds to render a weblog page. I studied the various EJB patterns and realized that returning a collection of Entity Beans from the business tier was not a good idea. I learned about the Data Access Object and Value Object patterns. I discovered that XDoclet could generate Value Objects (which Sun now calls Data Transfer Objects) for me, and I changed the business tier interfaces to return the light-weight data objects rather than Entity Beans. Much better! That was performance enhancement #1.

Once I had the business tier interfaces returning data objects, I found that it was easy to convert Roller over to use Castor for persistence. I was tired of waiting for EJB containers to start, tired of waiting for EJBs to deploy, and generally disillusioned with EJB. So, I created Castor mappings to map the XDoclet generated Value Objects to database tables and switched over to Castor. Again, much better. That was performance enhancement #2.

When I tried to get Roller to display my bookmarks page, which has a hundred or more bookmarks, I still found that Roller was way too slow. Rendering a page was taking 10 seconds or so. I realized that I was not taking advantage of database connection pooling. Configuring database connection pooling was performance enhancement #3.

I'm not sure about the next chapter, but I believe that it happened this way. I was forced to switch from Resin to Tomcat and Tomcat was slow (back then it was slow, now it screams ;-). I found that OSCache could allow me to hide, almost completely, this problem. OSCache was enhancement #4.

I'm a little fuzzy about this one too, but here is what I remember. One day I read that some guy named Anthony Eden had decided to set up Roller and to offer free accounts on a site called FreeRoller. I figured he would close FreeRoller off after reaching 10 or 20 users. He didn't. He kept going and going. Now I'm not sure if he is ever going to stop. Anyhow, Roller started running out of memory, so we adjusted to session timeout to a lower value and I moved the Roller business tier implementation from being a session thing to being an application thing. Those changes, call them enhancement #5, fixed the out-of-memory problems.

FreeRoller did pretty well for a while, especially considering the el-cheapo little eMachines box that Anthony started with. Eventually, it got bogged down again. I was starting to think that our persistence framework was at fault and I had heard that Hibernate was much faster than Castor, so I switched to Hibernate. Hibernate seemed to be about twice as fast. I'll call that enhancement #6.

Around this time, JavaLobby offered to host FreeRoller. FreeRoller moved from Anthony's little eMachines box to a big honkin' JavaLobby server. Everybody danced a wild fandango of glee and revelled in the new found speed and snappiness of Roller. FreeRoller != Roller, but for the sake of this story, we'll call that enhancement #7. The fast machine worked great for a while, but eventually it too started to slow down. At first, users did not notice the problem, but Rick Ross and Matthew Schmitt complained that Roller's load was way too high. After some investigation, we figured out that the RSS Servlet was doing more work than it should. So, enhancement #8 was to make sure that RSS feeds are cached and that the IF-MODIFIED-SINCE header was being handled and handled efficiently. That fixe brought the Roller load down to reasonable levels.

That brings us to today. The number of FreeRoller users and readers has grown and FreeRoller is again creaking and straining under the pressure. I hadn't noticed this because I was so busy with other things, but Anthony noticed, and so did Lance. Lance did some research and theorized that the problem is that Roller uses too many database connection to process each request. Each Roller weblog page is rendered by a user-edited Velocity script with a dozen or more embedded macros. Each of these macros makes one or more calls to the Roller business tier and each Roller business tier method opens a connection, performs some database operation, and closes the connection. Roller uses database connection pooling, but even so, grabbing and releasing a connection dozens of time for each request can't be a good thing.

Long story long, Lance modified Roller to use one connection per request. Now we are testing Roller and if all goes well, I'll backport the changes to the Roller 0.9.7 branch so that they can easily be applied to FreeRoller this weekend. I'm keeping my fingers crossed and hoping that enhancement #9 will hold us over for a couple of months while we figure out enhancement #10. I've learned a lot from Roller and I suspect that I have a lot more to learn.

Java World: JSP [document use] XML-compliant include and forward actions, as well as custom tags, so making the entire document XML-compliant leads to increased code consistency.

Jason Carreira: Scaling doesn't have anything to do with whether your objects are remote. How often do you really have one web request cross JVM boundaries? The whole Servlet-tier and Business Object tier topography has been pretty well discredited. It doesn't scale to have to make remote calls.

I've been investigating web application testing tools, both for unit testing of web components such as Struts Actions and for functional testing. I read about JUnit extensions Cactus and HttpUnit in Hightower and Lesieki's book Java Tools for Extreme Programming. I learned about StrutsTestCase and JUnitDoclet from Matt Raible's gold-mine of example Ant code: the AppFuse project. That was enough to get me started.

I put together a simple test app with Cactus and StrutsTestCase and was pleased to find how easy it is to invoke Cactus from an Ant script and have Cactus start Jetty, run my tests within the Jetty, and shutdown Jetty when the tests complete. Cactus is going to make it really easy for us to unit test our Servlets, Filters, JSP pages, and Struts Actions. Cactus uses HttpUnit and HttpUnit makes it easy to make sense out of the responses that come back from the web layer, see the HttpUnit cookbook page for an example if this. I wish I had done this a long time ago, how did I live without this stuff!

So, I'm all set for unit testing. Now I need to figure out what to do for functional testing. HttpUnit, MaxQ, and Canoo look promising. I found the bewilderingly long list of functional testing tools at opensourcetesting.org. What open source or commercial tools have you used for functional testing of web applications?

I finally submitted the last of my chapter edits this week for the Apress JSP book. Now I have my evenings back and can get back to the kids, to clearing a giant Netflix queue, cleaning out the garage, and to working on Roller.

I've been too busy do anything other than watch the emails fly by, but Min has been busy. He added full weblog search to Roller via Lucene. Take a look at Matt's write-up for details.

Good stuff from Mr. Lowem: Minor Roller anoyances and More Roller Annoyances. Interested in improving Roller? Join us on the Roller development mailing list. There are lots of little bugs to fix and minor improvements to be made, great for those getting started with server-side Java, but there are probably some deeper architectural issues for those interested in performance and scalability.

Slashdot picked up Andy's LGPL is viral for Java story and provoked a response from the Free Software Foundation (FSF):

FSF Executive Director, Brad Kuhn LGPL's S. 6 allows you to make new works that link with the LGPL'ed code, and license them any way you see fit. Only the LGPL'ed code itself must remain Free. Such 'client code' can even be proprietary; it need not be LGPL'ed.

I'm so confused. Is that the final word? He didn't mention Java specifically. He is not a lawyer. I don't think this is going to be the final word for Apache. I'm starting to think that FSF wants this to be a confusing issue.

Matt's taking the day off to celebrate his birthday. Have a good one Matt.

(C) 2003 Charles S. Johnson

Andy Oliver: I clarified the LGPL in Java issue with the FSF and posted to poi-dev. According to the FSF, the LGPL is indeed viral towards Java code. Thus, in essence GPL and LGPL are the same for Java. You could still find that the copyright holder of a particular software package held a different view and is willing to state that explicitly, but it looks like the Apache Software Foundation and Sam were right and I was wrong.

That really sucks. Not the part about Andy being wrong, that happens all the time. I'm talking about the now confirmed viral nature of the LGPL when applied to Java. I believe that Roller uses a couple of LGPL components, JSPWiki for example. I guess this means that either those components should be removed or Roller should switch to a dual GPL and APL license.

This year we are vacationing in Emerald Isle, NC, about a three hour drive from home. My parents rented a big house about a block from the ocean and invited us to come along. With three kids, having both parents and grand-parents around is really, really convenient. There is always somebody available to take the big kids to the beach, somebody to stay home when the littlest guy needs to nap, and somebody to babysit so Andi and I can drive over to Beaufort for dinner one night. Dad, the expert photographer and physical chemist extraordinaire, brought along his big honking digital SLR camera and his assortment of telephoto lenses, so I don't even have to worry about taking pictures.

This is all great and the only thing really causing any stress is the ProJSP book, but I don't think I'll have to miss a day at the beach to wrap up my changes to my two chapters. My Dad and I have our laptops setup in the kitchen and the only thing that really interferes with my works is the constant derision and calls of "hey nerd-boys" and "get a life" from our wives. This really doesn't bother me at all. I always arrange to bring a laptop along on vacation. I'm not like Raible. My idea of a vacation is tinkering with all things digital, which, coincidentally, is also my idea of work. I'm the last of the great indoorsmen.

I do go outside occaisionally. I'm writing this sitting in a rocking chair on the porch overlooking the ocean. We watched a little motorized para-glider fly by just before sunset. Now we are watching some impressive amateur fireworks going up accross the street and trying to keep the boys from stepping on the tiny tree frogs that appear after dark. Damn, I can't see the keyboard anymore. Time to go back inside.

Via Corante: Mena Trott on AOL's blog offering: "This isn't just some message board with a blogging label slapped on -- the AOL Journals team is taking the time and effort to get this right and that's highly commendable."

MozillaZine: It has been learned through public and private sources that AOL has cut or will cut the remaining team working on Mozilla in a mass firing and are dismantling what was left of Netscape (they've even pulled the logos off the buildings). Some will remain working on Mozilla during the transition, and will move to other jobs within AOL.

I'm not being entirely fair. AOL gave Mozilla a nice little send off and a nice wad of cash to move Mozilla forward.

I was unable to use the early Tomcat 5.0 alphas when I was working on Professional JSP 2.0 and I've heard plenty of complaints about Tomcat 5's extreme instability, but the latest alpha seems to work just fine. I downloaded it, loaded my Data Access for Web Applications example in, and it just worked! I was also happy to find that the Sysdeo Tomcat plugin has been updated to support the Tomcat 5.X line.

I noticed the link to Typepad, Six Apart's new Movable Type-based weblogging service, on Joi Ito's blog. The feature list and screenshots look awesome. The integrated photo gallery looks pretty appealing. The wizard-style weblog template builder looks really nice too. Not only does it support multiple weblog authors, each with different priviledges, it also allows you to invite guest authors. I'm judging from a couple of screenshots and a marketing blurb, I know, but Typepad looks powerful and really easy to use.

It's been more than a month since I refreshed the code running behind this site. I'm now running the latest Roller 0.9.8-dev from CVS. This means I'm on Struts 1.1 final (thanks to Lance) and Hibernate 2.0.1 (thanks to the Hibernate guys). I'm hoping that I might actually get some time to work on Roller in the next couple of weeks. I need to. The natives are getting restless, but I'm going to the beach and you can't stop me.