I had a revelation: Cometd and Orbited are not at odds. Cometd is not competing with Orbited — each provides a unique set of features, and they solve problems in different domains. Moreover, a hybrid of the two technologies could result in a system more flexible than either alone.

I want to back up for a moment and reference the SQLAlchemy site. This is the very first time I saw a software system praised as being able to “scale down.” What the heck is that? We’re all in a rat race to scale up and it never occurred to me that there could be benefit to moving in the opposite direction. But then I thought about my past introduction to Python. The year was 2002, and I wanted a “web framework”, and someone so politely pointed me towards Zope. I was intrigued as it was the first whole-solution system I’d ever used for building web pages. But within six months I stopped drinking the kool-aid and despaired. It was so hard to make simple apps with Zope. There were endless repositories of great Zope packages and features that eventually just got in the way of development. I had to work around them to make something simple.

So here’s my pitch for Orbited. It scales down. It allows you to use simple APIs to create as lightweight a solution as you like. For all the small-scale, in-house, or prototype applications, you just want something easy and fast to develop. That’s Orbited.

Of course, SQLAlchemy also claims to “scale up”. Turns out that the same is true about orbited. It is designed around libevent for maximum speed on a single node. More importantly though, its simplicity allows you to scale laterally such that 5 orbited nodes support exactly 5 times as many users as 1 orbited node.

Something is missing though, and it smacks of the word Enterprise. I personally detest the beastly word, but there are lessons to be learned from it. An Enterprise solution has three requirements, generally. 1) Clear Scaling Path, 2) All-in-One solution, and 3) Java, as far as I can tell anyway. Orbited has a clear scaling path, check. Orbited also has a Java client library in the works (check… basically). It is missing the second requirement.

Enter Cometd and the Bayeux Protocol. The great thing about Bayeux and Cometd is that they handle every conceivable aspect of HTTP Push that you could ever hope for. There is event confirmation, room for authentication, pubsub, pluggable transports, complete dojo integration, and much more. I think Orbited does pretty well itself in the areas of authentication and pluggable transports, but the real issue is that we are missing the publish/subscribe support. This pattern of development is an overkill for many HTTP Push apps, but it is mighty helpful for writing anything that resembles a set of chat rooms, as I suspect many comet apps will.

Which brings me back to my revelation: Cometd and Orbited are not competitors. Orbited handles the transport layer of HTTP Push. Cometd can be built on top of Orbited without much difficulty. It’s a bit of work to implement the Bayeux protocol, but it’s clear how to go about it. The real issue I have with Cometd is that its current architecture is inherently hard to distribute across server nodes. Please note: this is not a theoretical shortcoming of Cometd or Bayeux; rather, the technology is scalabilty-bound by current implementation details. I am ready to jump on the Cometd bandwagon, and I bring Orbited to the table. From Orbited to Cometd: Let us handle the difficulties of scaling, both vertically (libevent) and laterally (share nothing), and we’ll let you take care of the protocol, Bayeux, and the browser-side development, dojo toolkit.

The last piece of the puzzle is as easy to find as your favorite IRC daemon. Or perhaps, for the truly hip, Jabber conferencing. Any distributed, channel-based system that scales laterally can effectively handle the subscription functionality necessary to implement the Bayeux protocol. I’ve started work on a Jabber-Orbited interface, and Mario Balibrera has already made substantial progress on a IRC-Orbited interface. The jury is still out on names for these two interfaces, so send me an email if you have any ideas.

I’ve identified all the pieces, so at this point I give this fledgling union of cutting edge technologies a name: TailSpin. That’s certainly how I feel when I try to wrap my head around the scope of operations necessary to power a modern web application. And it takes elements of both Orbited and Cometd.

Here’s some visualization of the proposed system. You can see how it compares to other Comet-style stacks at our Stack Comparison page.

In summary, I don’t think Cometd is a bad idea. I think that packing so many layers of functionality into a single project is a bad idea. From Cometd we should take Bayeux, but leave the rest of the stack to technologies better suited to solve lower-level problems. The result of decoupling these layers is that experts can tackle each field directly without the headache of interaction bugs. Furthermore, end-developers can use whichever parts of the stack they feel they need. If they want low-level but easy-to-use HTTP Push, then they can just build on Orbited directly. If they are creating limited-user-base applications, then they might be better served by something like a current Cometd implementation—a single process solution for Bayeux. If they want publish subscribe and don’t need the browser at all, then they can run an IRC server or run Jabber.

This entry was posted on Wednesday, July 18th, 2007 at 1:50 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.