Please take a look at A Fast Introduction to Hookbox, an article/tutorial that I just published on Comet Daily. I hope that you’ll find the technology to be much more approachable than anything else out there.

Feedback is very welcome.

Well, how about just a little more navel-gazing. It will pay off, I promise.

Have a look at the “About Orbited” page in the orbited.org wiki. There are eleven committers listed on the page and another seven contributors. Not represented there are the scores of people who have written blog posts describing how they’ve used orbited or laying out some steps for getting started. It would be extraordinarily difficult to count the number of visits that #orbited has received through the livehelp portal. Assuming that one of twenty livehelp contacts generated a constructive criticism or honest opinion and that everyone else I’ve mentioned made non-trivial contributions to the project, I think it’s fair to say that orbited’s evolution has been influenced by hundreds.

This makes orbited, in my opinion, a great illustration of the value in open development. orbited started as a solution to one person’s problem and so it changed a lot as that one person faced different challenges and had new ideas. orbited adoption grew in time and we were fortunate that new users would often take the time to discuss the problems they were having or the ways that they thought orbited could be improved. All of these influences have had a stabilizing effect on the project and work has shifted from figuring out what orbited should be to making it a better instance of that thing.

Early development was chaotic and the chaos suited the work. Now that we’ve entered a phase of refinement we need to determine the way to best match our organization to the work we’re doing. I would like to follow the example of the project that provided the foundation upon which orbited is built. This is the Twisted Project.

The people of Twisted Matrix Labs know how to do open development. Twisted is an impressive project, not only because of the subprojects it has produced, but also because of the way the developers work to produce them. I’ll leave it to you to read more about their development process or browse the project’s extensive documentation. For now, let’s focus on three aspects of their process: ticket discipline, code review and continuous integration.

Continuous Integration is something that I’ve heard about for a while, and had a hunch was for real, but hadn’t experienced until recently. It’s as simple as, well, ABB. “A” Always, “B” Be, “B” Building. Automate your “build” process and do it. A lot. Either on a preset schedule, or anytime there is a change to one of the pieces of your project that you control. Build goes in quotes, by the way, since here it represents whatever it is that you do to turn your raw materials (source) into something that people can use. The benefit for you is that when something goes wrong, you’ll know about it immediately. Everyone will, in fact, since you will make the results of your builds public. When you’re proud of what you’ve made, this is strong motivation for keeping it working.

The DivMod Ultimate Quality Development Process outlines a way of using a ticketing system in combination with code review to maintain a project’s level of quality. There’s a rule: every change to the source must be traceable to a ticket. That ticket must contain certain pieces of information and it will be used as the basis for a code review. So the change needs to match the request in the ticket.

There are two benefits (that I’m interested in) to using tickets and carrying out code reviews. Code reviews put more eyes (and brains) on the code, so more people involved with the project know how it works and, more importantly, why it works. Recording change requests and code review comments in tickets has a similar benefit; tickets become an established source of information about the system. Where code reviews have the effect of familiarizing project members with the state of the project, the ticket system makes the development process more readily available to those not part of the project.

Project transparency is something that orbited has lacked in the past, and something we need to do better. The orbited code base is small but dense and less than packed with comments. The bar has always been high for new contributors, but we can do something about that. Let’s work out a sustainable development system built on tickets, testing and code review. Let’s make it our goal to keep the state of the project available at all times and be ready to point people toward it.

Michael Carter started designing and prototyping orbited in August of 2006. At the time, orbited was to be a comet server that did not provide an publish/subscribe functionality. When I arrived in the Summer of 2007, this was the orbited I met. orbited allowed you to push messages between a browser and a client written in the language of your choice. Each connection was assigned an identifier and the clients could only send a message to one connection at a time. Associating groups of connections to create ‘channels’ was intentionally left to the server.

orbited 0.1 though 0.3 also shipped with an http proxy. Intended for development, the proxy worked around same-domain restrictions by making it look, to the browser, as though the orbited daemon and a web application are on the same domain. The proxy was dropped since it was a maintenance headache and the choice was make to simply support cross-port and cross-domain deployments.

orbited 0.5 was the first release implementing the concepts we find in the current orbited. orbited has become a TCP socket proxy. Language-specific server-side client support was dropped in favor of the socket support that already exists in most language’s environments. The key insight here is that orbited should do one thing and do it well. The thing orbited does well is provide a tcp socket to browser javascript.

orbited 0.6 introduced the “quickstart” setting that runs morbidq alongside the orbited daemon. A common concern expressed by those new to #orbited is that they would need to run two separate programs to get the publish/subscribe behaviour they wanted. The quickstart made orbited and morbidq appear as one.

orbited 0.7 saw tighter integration with the morbidq STOMP broker and the inclusion of some new protocols (via the js.io project) as demonstrations and examples. The nature of orbited remained the same, but the distribution gained several “extras” intended to make it easier for new users to get started.

This is where we are today. orbited 0.7.10 line is the latest stable orbited release. The orbited 0.8 branch is focused on factoring out the core comet server that orbited uses and making it available to servers in other problem domains. Development of the orbited 0.8 branch has been slow and not always steady since extracting the comet server from the orbited daemon also meant extracting the corresponding javascript code from the bundled Orbited.js javascript file. This work is complete, and the js.io project now holds several standard protocol implementations that can be used in conjunction with orbited’s socket proxy to allow the browser to communicate with existing systems.

So while orbited has taken a number of forms since it was first released, it has stabilized as a tcp socket proxy. It is important to note that while orbited ships a number of protocol implementations as demonstrations, they are only examples. orbited does now, and will continue to, provide the expected tcp socket interface to browser javascript.

Michael is on the WebSocket, Comet, HTML 5, and web standards panel that SV Web Builder has organized for October 29th. The panel also includes Alex Russell, Dion Almaer, Ted Goddard, John Fallows, and Kevin Nilson. It’s all going down at the Google headquarters in Mountain View. The event is free if you register online.

A couple of new features include:

• Embedded Stomp broker via MorbidQ. Now you put stomp:// urls in the [listen] section of the configuration
• Reworked startup api for easier use with outside projects like WillowChat.
• Improvements to the IRC client (via js.io)
• Increased stability for Connection handling when the page is reloaded navigated away from then returned to
• Various small bug fixes
• See the commit logs and timeline

Obviously it makes sense to migrate to separate processes for a high-concurrency production deployment, but this method is perfect for developing applications and deploying small to medium instances.

The tutorial uses a different technique than the STOMP method I described. Dark Porter takes advantage of the fact that Django and Orbited are both written in Python, and puts everything in one process, including a custom threaded TCP server.

Dropped Connections

A web application can’t prevent most dropped connections. It can only handle them in the best manner possible. Dropped connections are a problem because unless there’s some notification mechanism, the browser won’t automatically try to reestablish the connection. Furthermore, the user will think that the application is operating in a connected state, and won’t realize that they are missing messages.

This hurts the functionality of the application, and also decreases the user’s confidence in the application, which has negative implications for how much they use it and whether they will use it for important tasks. Preventing disturbances in the user experience and increasing user trust are both very important goals. Orbited prevents lost messages by keeping a buffer of recently sent messages, but we still want to resume normal operations as possible.

Fixed Interval Reconnects

A common way to handle dropped connections is to have the browser reconnect every 30 seconds, regardless of whether the connection has dropped. Prior to 0.6.1 this is what Orbited did. This method caps the period of browser unawareness at 30 seconds + the browser timeout interval. ~30 seconds is a long time for the browser to be ignorant of the connection state. The interval can be decreased, but reconnects should be avoided for 2 reasons:

• The browser can’t receive messages while it’s reconnecting. The connection is usually fine, so most reconnects are actually making things worse for a short period of time.
• Each reconnect requires a full HTTP message to be sent and processed. This isn’t that bad for the client, but really adds up for the server, which is almost always the bottleneck for web applications. And some people are on dial-up or mobile connections.

Fixed interval reconnects aren’t horrible, and they’re used by live deployments. But there’s a much better way.

The Heartbeat Solution

Orbited 0.6.1 adds a feature called heartbeats, which provide better browser awareness at a lower overhead cost than fixed interval reconnects. A “heartbeat” is a tiny amount of data (~1 byte) sent from the server to the browser without separate HTTP headers (it’s sent in the context of the Comet connection HTTP headers), meaning the cost both of bandwidth and server processing is negligible.

If we send a heartbeat every 5 seconds, the maximum time the browser is unaware of the connection state drops from 30 seconds + the browser timeout interval (with fixed interval reconnects) to the heartbeat timeout, which defaults to 15 seconds (the timeout is longer than the interval to account for latency). And the browser only has to reconnect if the connection drops, instead of having to reconnect every 30 seconds no matter what. The timeout and the heartbeat interval can both be lowered if necessary.

Timeouts

Heartbeats also prevent browser and proxy timeouts. Browsers and proxies often close open connections after around 60 seconds of inactivity. The fixed interval reconnect avoids the connection being closed unceremoniously, but since heartbeats count as activity, they prevent the timeouts from even happening. The fewer reconnects the better.

Usage with Transports

Orbited provides two Comet transports: streaming and long polling. Streaming provides better performance and is the default/recommended option. Both transports work with heartbeats. IE and Opera can only read heartbeats when they’re used with a streaming connection, so a long polling connection can’t use them for dropped connection handling, but they’re still handy for avoiding timeouts. Streaming with heartbeats works with most firewalls, forward proxies, reverse proxies, and routers, but if a particular deployment of intermediaries does manage to thwart streaming, the server can fall back to long-polling.

What You Need to Do

Probably nothing! Orbited 0.6.1 and onward use streaming with heartbeats by default. If you want you can lower the heartbeat interval and timeout for even better browser awareness of dropped connections.

If you’re using a version of Orbited lower than 0.6.x, this is a great time to make the change, because 0.6.0 marked the change from alpha to beta, with the corresponding stabilization in API and core functionality. Remember to stop by the mailing list, IRC channel (#orbited on freenode or use the LiveDemo), and Trac system to participate in the development process and/or get help with Orbited.