My last post was a short walk through the history of the orbited project. We examined orbited’s evolution over the past three years to help us better understand its current state. A better understanding of what orbited is now should lead to a better plan for the future. I’ll post about this plan very soon, but first I’d like to talk about how we’ll get there.

Well, how about just a little more navel-gazing. It will pay off, I promise.

Have a look at the “About Orbited” page in the orbited.org wiki. There are eleven committers listed on the page and another seven contributors. Not represented there are the scores of people who have written blog posts describing how they’ve used orbited or laying out some steps for getting started. It would be extraordinarily difficult to count the number of visits that #orbited has received through the livehelp portal. Assuming that one of twenty livehelp contacts generated a constructive criticism or honest opinion and that everyone else I’ve mentioned made non-trivial contributions to the project, I think it’s fair to say that orbited’s evolution has been influenced by hundreds.

This makes orbited, in my opinion, a great illustration of the value in open development. orbited started as a solution to one person’s problem and so it changed a lot as that one person faced different challenges and had new ideas. orbited adoption grew in time and we were fortunate that new users would often take the time to discuss the problems they were having or the ways that they thought orbited could be improved. All of these influences have had a stabilizing effect on the project and work has shifted from figuring out what orbited should be to making it a better instance of that thing.

Early development was chaotic and the chaos suited the work. Now that we’ve entered a phase of refinement we need to determine the way to best match our organization to the work we’re doing. I would like to follow the example of the project that provided the foundation upon which orbited is built. This is the Twisted Project.

The people of Twisted Matrix Labs know how to do open development. Twisted is an impressive project, not only because of the subprojects it has produced, but also because of the way the developers work to produce them. I’ll leave it to you to read more about their development process or browse the project’s extensive documentation. For now, let’s focus on three aspects of their process: ticket discipline, code review and continuous integration.

Continuous Integration is something that I’ve heard about for a while, and had a hunch was for real, but hadn’t experienced until recently. It’s as simple as, well, ABB. “A” Always, “B” Be, “B” Building. Automate your “build” process and do it. A lot. Either on a preset schedule, or anytime there is a change to one of the pieces of your project that you control. Build goes in quotes, by the way, since here it represents whatever it is that you do to turn your raw materials (source) into something that people can use. The benefit for you is that when something goes wrong, you’ll know about it immediately. Everyone will, in fact, since you will make the results of your builds public. When you’re proud of what you’ve made, this is strong motivation for keeping it working.

The DivMod Ultimate Quality Development Process outlines a way of using a ticketing system in combination with code review to maintain a project’s level of quality. There’s a rule: every change to the source must be traceable to a ticket. That ticket must contain certain pieces of information and it will be used as the basis for a code review. So the change needs to match the request in the ticket.

There are two benefits (that I’m interested in) to using tickets and carrying out code reviews. Code reviews put more eyes (and brains) on the code, so more people involved with the project know how it works and, more importantly, why it works. Recording change requests and code review comments in tickets has a similar benefit; tickets become an established source of information about the system. Where code reviews have the effect of familiarizing project members with the state of the project, the ticket system makes the development process more readily available to those not part of the project.

Project transparency is something that orbited has lacked in the past, and something we need to do better. The orbited code base is small but dense and less than packed with comments. The bar has always been high for new contributors, but we can do something about that. Let’s work out a sustainable development system built on tickets, testing and code review. Let’s make it our goal to keep the state of the project available at all times and be ready to point people toward it.

The first post to the current incarnation of the orbited blog is dated just over three years ago. It almost certainly lived in Michael Carter’s imagination for a while before that, but let’s say that orbited was born on the Ides of March in 2007. In the three busy years that have passed, orbited has taken several shapes. Since we’re getting ready to plan for the next stage of orbited’s career, it will be instructive to look back at what orbited has been before we decide what it should become.

Michael Carter started designing and prototyping orbited in August of 2006. At the time, orbited was to be a comet server that did not provide an publish/subscribe functionality. When I arrived in the Summer of 2007, this was the orbited I met. orbited allowed you to push messages between a browser and a client written in the language of your choice. Each connection was assigned an identifier and the clients could only send a message to one connection at a time. Associating groups of connections to create ‘channels’ was intentionally left to the server.

orbited 0.1 though 0.3 also shipped with an http proxy. Intended for development, the proxy worked around same-domain restrictions by making it look, to the browser, as though the orbited daemon and a web application are on the same domain. The proxy was dropped since it was a maintenance headache and the choice was make to simply support cross-port and cross-domain deployments.

orbited 0.5 was the first release implementing the concepts we find in the current orbited. orbited has become a TCP socket proxy. Language-specific server-side client support was dropped in favor of the socket support that already exists in most language’s environments. The key insight here is that orbited should do one thing and do it well. The thing orbited does well is provide a tcp socket to browser javascript.

orbited 0.6 introduced the “quickstart” setting that runs morbidq alongside the orbited daemon. A common concern expressed by those new to #orbited is that they would need to run two separate programs to get the publish/subscribe behaviour they wanted. The quickstart made orbited and morbidq appear as one.

orbited 0.7 saw tighter integration with the morbidq STOMP broker and the inclusion of some new protocols (via the js.io project) as demonstrations and examples. The nature of orbited remained the same, but the distribution gained several “extras” intended to make it easier for new users to get started.

This is where we are today. orbited 0.7.10 line is the latest stable orbited release. The orbited 0.8 branch is focused on factoring out the core comet server that orbited uses and making it available to servers in other problem domains. Development of the orbited 0.8 branch has been slow and not always steady since extracting the comet server from the orbited daemon also meant extracting the corresponding javascript code from the bundled Orbited.js javascript file. This work is complete, and the js.io project now holds several standard protocol implementations that can be used in conjunction with orbited’s socket proxy to allow the browser to communicate with existing systems.

So while orbited has taken a number of forms since it was first released, it has stabilized as a tcp socket proxy. It is important to note that while orbited ships a number of protocol implementations as demonstrations, they are only examples. orbited does now, and will continue to, provide the expected tcp socket interface to browser javascript.

The Silicon Valley CodeCamp is next weekend (Nov 8 & 9). On Saturday Michael Carter is presenting on “WillowChat: Out-of-the-Box Comet Webchat,”, which is powered by Orbited and js.io.

The Orbited Project always strives to push the boundaries of bi-directional web technology, and then roll our advances back into standards. The Orbited.TCPSocket is a prime example — initially it was our internal standard, the separation between protocol and transport. The js.io project is targeted against the TCPSocket API instead of being built directly into Orbited because we always hoped to provide the widest range of integration. For this reason, I’m excited to announce that Lightstreamer will be joining Orbited and Sproket.Socket in supporting the js.io.TCPSocket API. This means that you will be able to swap Orbited out for Lightstreamer in any applications that depend on js.io, including applications developed against the js.io.WebSocket implementation for future-proof compliance with the HTML5 standard.

Michael just wrote an article on WebSocket for the Silicon Valley Web Builder blog. He talks about his experiences explaining WebSocket to developers, and how we can “recover our ancestor’s secrets” of good architecture.

Michael is on the WebSocket, Comet, HTML 5, and web standards panel that SV Web Builder has organized for October 29th. The panel also includes Alex Russell, Dion Almaer, Ted Goddard, John Fallows, and Kevin Nilson. It’s all going down at the Google headquarters in Mountain View. The event is free if you register online.

Today marked the release of Orbited 0.7.0, along with the first 0.7.0 tutorial over at CometDaily. Thanks to everyone who helped with the release.

A couple of new features include:

• Embedded Stomp broker via MorbidQ. Now you put stomp:// urls in the [listen] section of the configuration
• Reworked startup api for easier use with outside projects like WillowChat.
• Improvements to the IRC client (via js.io)
• Increased stability for Connection handling when the page is reloaded navigated away from then returned to
• Various small bug fixes
• See the commit logs and timeline

Dark Porter posted a tutorial on using Orbited with Django. Michael Carter linked to it on Comet Daily, and said:

Obviously it makes sense to migrate to separate processes for a high-concurrency production deployment, but this method is perfect for developing applications and deploying small to medium instances.

The tutorial uses a different technique than the STOMP method I described. Dark Porter takes advantage of the fact that Django and Orbited are both written in Python, and puts everything in one process, including a custom threaded TCP server.

The 0.6.1 release of Orbited adds a feature called heartbeats that improves dropped connection handling and also decreases reconnects due to timeouts. Dropped connection handling is important because it affects the user experience of a web application. Heartbeats are better than the previous solution of reconnecting at a fixed interval because they do a better job at a lower network cost.

Dropped Connections

A web application can’t prevent most dropped connections. It can only handle them in the best manner possible. Dropped connections are a problem because unless there’s some notification mechanism, the browser won’t automatically try to reestablish the connection. Furthermore, the user will think that the application is operating in a connected state, and won’t realize that they are missing messages.

(more…)

The simplest Orbited architecture is to have Orbited proxying messages between the TCP server and the browser using socket connections. Many people, however, choose to use web application frameworks like Django or Ruby on Rails because they helpfully abstract away certain cumbersome tasks. You can easily connect Orbited and frameworks using message queue brokers, and the 0.7 release will feature an integrated basic broker.

(more…)

Michael and I have recently formed a nonprofit corporation to manage Orbited and related projects such as js.io. One of the main reasons for the nonprofit is so that contributors can sign a Contributor License Agreement (”CLA”) with an appropriate legal entity. The CLAs will protect users of the open source software projects managed by the nonprofit, and are a good reminder to think about intellectual property.

(more…)