So we just released 0.1.1, which means you guys all get to play with the toys we’ve been working on. You’ve seen the first Orbited chat demonstration. Now here’s the first Orbited game (brought to you by the new iframe transport)!

I know you better than you think. You had your 10th birthday party at a library because you didn’t get internet access until middle school. When you got there, the librarian wouldn’t even let you play because it was peak library season. And now, you relive the shame five days a week because your boss won’t let you install zMUD on the company computer.

But you’re reading this in a web browser, right? Well, dust off your vorpal sword and join the dozens already exploring this monument to the 90’s.

This was surprisingly easy to implement, and you will soon see that I have very little ’splainin to do. Here’s how it works:

You have

• http.py
• game.py, where all the objects in the game live
• a few short files with very human-readable information about the world
• app.py

app.py is a very simple animal. Besides a world clock and a couple functions for handling basic game mechanics, app.py doesn’t really have anything but connect() and msg(), which listen to http.py, which in turn takes orders from the browser. But isn’t it time the browser listened to me, for a change?

Downstream messages are sent out by the objects in game.py and delivered by Orbited to the iframe, which signals an event that is processed in the game’s javascript.

But the developer doesn’t need to know anything about orbit, or, really, the iframe. She only needs to know four things:

1. <iframe id=”events” style=”display:none;”></iframe> goes in websitename.html
2. the browser connects to and updates the server with XMLHTTPRequests
3. the server generates Orbit events
4. the application magically updates itself

Go forth brave adventurer, full speed ahead, you’re welcome.

UPDATE 6-6-2007

0.1.3 auto-wraps events, so the server code is further simplified! And the game will be running here for a while.

Announcing Orbited 0.1.1

• New Transport type: iframe
• Rewrote Proxy
• keepalive was messing it
• Old method: read headers, move connection to proxy mode. Problematic due to keep alive
• New method: read all data being proxied
• Result: Keepalive works but More Cpu
• New Demo: Cherrychat

So lately I’ve been hearing a lot of hype about Twisted. Twisted is event based. Twisted is fast. Twisted does every single thing in the universe.

I recently started working on a secret Orbited project with a generous helping of Twisted, which practically wrote the bulk of it for me.

This time yesterday, I sat here with a (rather skeletal) backend that seemed to do what it was told. This was encouraging, but I knew that it was all pointless without a web interface. I mean, web integration is the whole point of Orbited.

The problem at this point was a little strange: there were multiple Twisted HTTP libraries, they were too comprehensive, too featured, too big, and I was too tired to figure it out. Maybe I was burned out on the Twisted API, perhaps I was even a little crazy at this point, but I didn’t want anything to do with IResource and leaf nodes, and I decided that it would be easier to just write my own HTTP server.

This, of course, isn’t to say Twisted abandoned me in my hour of need. I mean, really, most of http.py IS Twisted (including Twisted’s famous Protocol and ServerFactory). But it’s exactly what I needed, something simple: the status line, the headers, the form, and a way to write your response. All it does is call your App().dispatch(request) function and provide headers and a response object.

More on the secret Orbited project coming soon…

Before and during the process of developing the concepts and architecture behind Orbited, I spent some time chatting with the folks in #cometd on irc.perl.org. I was a bit confused by the project for a while — I knew what problem domain they were attempting to solve, but I wasn’t sure how exactly they solved it. I was specifically confused on a number of points:

1. Why does Cometd have a different version of the server for every language?
2. How is the web-app supposed to send events to the browser?
3. How do I communicate with the Cometd server from Javascript?
4. How do I to run multiple Cometd nodes in a sensible way?

I’m going to give my best bet at answering each of these questions.

1. Why does Cometd have a different version of the server for every language?

   Ultimately there will be one or two Cometd implementations to rule them all. And probably they’ll be written in Java. In the meantime, smart hackers are using whatever language they know best to experiment. There are multiple implementations because no one really knows what will work in the end.

2. How is the web-app supposed to send events to the browser?

   The web application is supposed to pretend to be a browser that speaks Bayeux. So if the application wants to receive events, it will be using an iframe stream or long polling. If it wants to subscribe, unsubscribe, or publish, it will make json encoded Bayeux requests to the cometd server.

3. How do I communicate with the Cometd server from JavaScript?

   The obvious answer is to use Bayeux. But it’s really hard to implement and you aren’t expected to do so. The guys at Dojo Toolkit have already implemented Bayeux, and presumably there are more to follow. So the short answer is, use Dojo Toolkit.

4. How do I to run multiple Cometd nodes in a sensible way?

   I sort of have an answer, but I don’t fully understand it. I talked to some guys in #cometd on this issue, and the first response was “Good question… I don’t really know.” I did some research on the web and noticed this thread on a Cometd-user google group. The long and the short of it is “No one knows.” But on the other hand, Greg Wilkins told me that this is an easy problem to solve. You simply stick a Java Message Queue (JMS) behind your Cometd nodes and you’re done. I’m maybe a quarter willing to buy this, but in the face of this post by Greg, I have some doubts. I understand that Terracotta is completely unrelated to JMS, but it indicates that attempts at scaling aren’t going well. So the short answer is, design your app so that clusters of channels and users are isolated from each other such that each Cometd server supports a specific set of users/groups, and only those users and groups.

I hope my answers help someone who is interested in Cometd. Now I’m going to justify this post’s title. I have a number of problems with Cometd.

1. No easy way for my web app to send an event to a specific user.
2. You are locked into Dojo
3. There is no possible method to pre-process events, short of rewriting the Cometd server
4. There currently exists no security.
5. Lacks App-based configurability (or any sort of configuration)
6. Cometd is Inherently difficult to scale.

I’ll even substantiate my claims.

1. No easy way for my web app to send an event to a specific user.

   Any server whose purpose is comet-like communication has two points of interaction for the end developer: The web application and the browser. These are the most important parts because they dictate how a server can or cannot be used. Furthermore, the end-developer only ever sees these two parts of the server. If they are complicated, it doesn’t matter how beautiful the internal implementation is. APIs that are complicated or hard to implement are generally not well received and take longer to be adopted.

   So let’s say that I want to use Cometd with a currently unsupported language for my web application… Lisp. I have to implement a Bayeux client in Lisp. Now, I’ve spent half a year thinking about the problem space of Comet, read the Bayeux spec through multiple times, conversed with the authors of Bayeux, and even closely analyzed the source code of the twisted Bayeux implementation, as well as the Dojo client implementation. I am still lost. I will not be implementing Bayeux in Lisp, or any other language for that matter. Sure, eventually someone will after Cometd is popular, but what do I do in the meantime? And what if I don’t like the implementation that exists for a given language? Nothing, I’m stuck. It’s not as god awful as being stuck in some proprietary system, but it is still terrible in its own right.

2. Locked into Dojo

   The same logic applies for choice of Javascript toolkit. If you want to use Cometd, then you have to use Dojo Toolkit. Personally, I have little problem with that. But someone has a problem with it, otherwise everyone would already be using Dojo. And because it’s the Dojo guys who are pushing Bayeux, we aren’t likely to see them write an implementation for a competing framework.

3. There is no possible method to pre-process events, short of rewriting the Cometd server

   Lets take a moment to examine the flow of events through a Cometd server. Here is my understanding of the Cometd stack:

   

   Lets say that user A sends a chat msg to channel 1, which contains user A, B, C, and the webapp. The event is sent from User A’s browser to Cometd. Cometd takes that message and delivers it to user B, C and the webapp. But lets also say that user C is in Chile where they don’t speak English. We want our application to translate the message on the fly from English to Spanish. Unfortunately, the Web App isn’t the hub, its just a client like all the browsers. It can receive notification of the message, but it can’t alter the message before it gets to the other users.

4. There currently exists no security.

   There is no bar on who can join what channel, or even on who can publish to a given channel. In fact, check out the chat Cometd demo. Pay particular attention to line 51 and 52. This is the leave function, for leaving a chat room. Line 51 shows the browser unsubscribing from the chat channel, and line 52 shows the browser publishing a leave message to the channel. You don’t even have to be in a channel to publish to it. Is this by design, and if so what’s the advantage? I see this is a gaping security hole.

5. Lacks App-based configurability (or any sort of configuration)

   Okay, let’s entertain the thought for a moment that Cometd is really intended to allow users to publish to channels they aren’t subscribed to. But now we want to make certain rooms private. How do we do this? Well, right now you can’t. But assume that it became a configuration option. How would the web app tell Cometd which rooms to make private or public? There are a zillion other settings that need to be created and modified on the fly by the web app, but that doesn’t exist in Cometd. There are no hooks for the app. Your only bet is to reprogram the Cometd daemon itself, which is not very realistic. How do you kick a user out of a channel? How do you moderate a channel? How do you set a channel size limit, and change it as the app runs? I could keep asking questions, but the answer to all of them is: You cannot.

6. Cometd is Inherently difficult to scale.

   All of the previous problems are small beans though compared to my real problem: Cometd is inherently hard to scale. I mean, more so than Comet-style communication is inherently hard to scale. The reason is the publish subscribe mechanism that is the heart of Cometd. Every node in a Cometd cluster needs to contain information about every group. It needs to additionally know about every user and their membership in every group, or alternatively, relay every event to every server in the cluster. You’ll notice i used the word ‘every’ an awful lot just now, and that’s a problem. Every * every = n^2 of something, bandwidth or cpu, it doesn’t really matter. It’s difficult to scale an O(n^2) system. The cost of the second user is going to be a thousand times less than the cost of the thousandth user.

I honestly don’t understand why Cometd feels the need to tackle that problem. I’m not saying publish/subscribe isn’t ever necessary in a web app, rather that it’s a layer that doesn’t belong in a Comet server. Cometd shouldn’t be in the pubsub business… that’s a tough business. As if Comet wasn’t tough enough already.

In the end I think that Cometd is making amazing strides in advancing state of the art technology on the web, but it doesn’t mean we should be blind to alternatives.

I’ve had a few versions of python orbited clients floating around, so I decided to package them together and make an egg. I plan on making tutorials, so this is a good step towards those ends. You can use setuptools easy_install to get the pyorbited package, or download it directly from [cheeseshop].

Announcing Pyorbited 0.1.0

Pyorbited is a collection of python orbited clients. They can be used in conjunction with Orbited (www.brbx.com/orbited) to enable real-time communication in your web apps.

The current implementations are:

• Pyevent-orbited
• Twisted-orbited
• Python-orbited

There are all sorts of fun prototypes you can build using orbited and your favorite web framework. Unfortunately, most of these applications will never scale past a few hundred users on a single node. The reason is that your favorite web framework is probably pretty bad at network IO. It almost certainly dispatches each request to its own thread. And to add insult to injury, these applications probably won’t be very interesting.

Can your web framework create an IRC connection for every user logged into the system? Can it do it fast? Can it do it for more than 24 users? Well, guess what: Twisted can do all of that. For hundreds, if not thousands of users. It is event based, highly scalable, and has almost more protocol implementations than existing protocols.

Both Orbited and Twisted are event based. Both are targeted at low-latency and high scalability applications. Orbited is about receiving real-time notifications; Twisted is about connecting to real-time sources. If you want to connect X to orbited, where X is anything on the internet, then Twisted can do it and do it well.

To this end, I’ve written a twisted-orbited client (Update: This is now part of the pyorbited project) and done some rudimentary tests. I sense the beginning of a great relationship.

I’ve been putting out a lot of propaganda about how scalable Orbited is. I think its time I explain why Orbited is scalable, and what the cost of that scalability is.

Let me first direct your attention to memcached, a brilliant distributed caching system from the livejournal guys. If you haven’t heard about it, go read and come back. The short description is that it acts like a distributed hash table. Each piece of data hashes to a specific server, so you always know where to find data, and memcached nodes don’t ever share information.

This is known as share nothing architecture. The purpose behind it is the cut down on intra-node dependance, and ultimately to allow scaling an application laterally. The classic version of this is web application sessions. If you store session data on each server node then as users access other nodes they need to contact the “home” node of that user to ask for session information. While this works, you end up with increasing overhead of each additional server. So adding the 10th node might not actually get you any performance increase because of the overhead in exchanging session state information with the other 9 nodes.

As far as Comet style communication goes, the main problem is maintaining state information about channels in the publish/subscribe paradigm. Cometd uses this pattern. Other examples include IRC and Jabber. The basic idea is that users can subscribe to channels, and publish to channels. Upon publishing an event to a channel, every member of that channel receives the message.

This is seductively trivial to implement on a single server. But what happens when you add the second server? You have to figure out some way to partition your user base. Partitioning by channel wouldn’t make much sense. Let’s say you had users subscribed to channel A connect to server A, and users subscribed to channel B connect to server B. What if all users need to connect to both channels? Then all users connect to both servers, and you’re back to square one.

Okay, then let’s partition based on users. Half the users will connect to server A, and the other half to server B. This works a bit better, but creates an additional problem. The crux of that obstacle is this: If you want to dispatch an event to channel A, to which server do you send that event? The answer, inevitably, is both servers. We’ve solved the public-side of our scaling problem nicely. I mean, the bandwidth coming from the public to us scales linearly. But our back-end suffers the consequences.

Long before we reach a respectably sized server cluster, our intranet will be completely saturated by our webapp dispatching events. On top of that, our webapp will have some serious cpu usage issues what with all the tcp connections and event dispatching to each of the Comet nodes. We could abstract a bit — use a central dispatching server, or put logic on each node in our Comet cluster such that each node will replicate an event to the other nodes. But no matter how you slice it you’ll end up with an unscalable solution, as event data needs to be re-exchanged an increasing amount with each additional node.

Disclaimer: I’m simplifying the explanation of scaling publish/subscribe to some extent. There are all sorts of graph theory experts who know great ways to solve some of these problems. So I’m not saying that we should throw out publish/subscribe. Rather, I’m saying that it’s damn hard to scale. I doubt we’ll see a truly scalable and open source comet server that provides a good publish/subscribe mechanism.

It’s time to see how Orbited fits in to this mess. You can think of Orbited the same way you would think of Memcached. Except, instead of distributing pieces of data across nodes, we’re distributing Comet connections. Each connection has a unique id, and based on that id the browser is assigned to a particular node. Each web app node uses the same hashing algorithm to dispatch events, so they always know where every user is without communicating with other nodes.

Orbited does not handle publish/subscribe whatsoever. There is no way for us to do that and still scale linearly. But as a result, every time we send an event from our application to a user, it takes a single path through our intranet to their browser.

So now the bad news. Most applications benefit from publish/subscribe. Even a simple stock ticker or chat application. And Orbited doesn’t provide that support. Our philosophy is that we aren’t experts at scaling publish/subscribe so we’ll let someone else do it. For example Jabber, or IRC. So if you want to use publish/subscribe, then you need to add an additional layer to your stack. This is going to be frustrating for newcomers who want to create a prototype and leave worrying about all the scaling rigamarole until later, or never. As a result, many applications, even my most basic demo app (UPDATE: this demo app has become cherrychat) have to roll their own subscribe based system.

I don’t want to end with the bad news, so I’m going to point out an academic paper from 2004 titled “Building Content-Based Publish/Subscribe Systems with Distributed Hash Tables” which discusses the future of scaling publish/subscribe on the internet. The authors propose that we use distributed hash tables as the basis of scalable publish/subscribe systems. So, rest assured that by adopting Orbited you are on the right track. It may not be a full stack solution, but it’s fast and scalable and sometimes those concerns trump all else.