Blogging Roller

A number of people have asked how Freeroller suddenly became so fast and so stable. The answer is Roller 0.9.8. Freeroller had been running Roller 0.9.7, which had a history of performance and stability problems from the start.

The first problem we experienced with Roller 0.9.7 wasn't noticed by the users, but our gracious hosts at The Javalobby noticed an unusually high load on their server coming from the Roller and MYSQL processes. Later, as the number of bloggers blogging and readers visiting the site grew, users started to notice unpredictable behavior. Spurious 404 errors, sluggish performance, and frequent downtime started to irritate everybody.

Roller 0.9.8 seems to have resolved all of these problems. Let's take a look at the changes made since Roller the initial 0.9.7 release starting with the changes that were applied to Freeroller before the Roller 0.9.8 upgrade:

• Caching weblog update times. Because Roller uses page caching, most content is served out of the page cache and for most requests there is almost no need to hit the database. For most requests we only hit the database to bump up hit counters and to determine last-update-time for RSS newsfeeds. Still, Roller was killing MYSQL and the load from Roller and MYSQL was dragging the Javalobby server down. Since RSS newsfeeds account for more than 60% of all traffic, I decided to cache those last-update-times. Javalobby noticed an improvement in load, but load was still a little high.

• The Open session in view pattern. With new users signing up every day and a number of more high-profile bloggers starting to draw in the hits, Freeroller really started to drag and users started to notice. The Roller development team started to wonder "what have we done wrong?" We eventually concluded that Roller uses too many persistence sessions, which roughly correspond to database connections, per request. Lance implemented the open session in view pattern in the Castor and the Hibernate persistence implementations in the Roller main branch. I backported the changes to the Roller 0.9.7 branch and deployed them to Freeroller only to find a minimal performance increase.

• OSCache 2.0-beta upgrade. Around that time, the OSCache team relased a beta of OSCache 2.0 and Hani told me that the old OSCache was rubbish and had a terrible memory leak, so I switched Roller and Freeroller over to the new OSCache, configured it for memory caching plus unlimited disk cache, and noticed a pretty big leap in performance and decrease in system load. Still, the stability was poor and spurious 404s continued.

Those changes were important changes, but they did not fix the performance and stability problems. Now, let's move on to the changes that were added by the Roller 0.9.8 upgrade:

• Numerous small fixes. While I was fussing with Freeroller and Roller 0.9.7, Lance and Min were looking for further performance improvements in the Roller 0.9.8 branch. Min fixed the Roller calendar control to keep it from hammering the database as it use to. They both worked to change the persistence layer interfaces to return lists instead of arrays, since both Castor and Hibernate return lists, list to array conversions are just silly in this case.

• Database indices. At some point during Roller 0.9.8 development somebody (Jim Smart?) contributed a patch for the Roller database creation script and added indices where indices were missing. I didn't think much of it at the time, but this was a pretty significant change.

• Hibernate/JCS caching. Freeroller was getting worse and worse, even causing Freeroller founder Anthony Eden to leave and, believe it or not, even causing the Hibernate team to feel the heat. Some were suggestingthat perhaps Hibernate did not scale. Gavin King stopped by the Roller dev-list and suggested that we try using proxies and JCS caching. I didn't get far with proxies, but it was easy to turn on JCS caching. Unfortunately, JCS caching by itself didn't seem to help much.

So, what change made the big difference in performance? It can't have been only update-time caching, open session in view, and OSCache 2.0 changes because they were all in place before the Roller 0.9.8 upgrade. It can't have been Hiberate/JCS caching because stress testing showed that JCS actually slowed down the system a bit. All of these fixes played a part in the performance improvements, but because of the greatly reduced load on MYSQL I think the database indices made the biggest difference.