Phil Leggetter - Real-Time Web Software and Technology Evangelist » comet

Delivery as a Service and Data as a Service in 2011

Phil Leggetter — Sat, 08 Jan 2011 13:32:30 +0000

Delivery as a Service and Data as a Service (both DaaS) are definitely going to be big in 2011. Things really kicked off in 2010 with the emergence of companies focusing on real-time data delivery and mobile. We also started to see really exciting services focused on delivering real-time client push notifications to web browsers. I would put Kwwika in both of these categories.

As 2011 continues we might to see some of these companies either partner up or start to tread on each others toes as they look at delivering data to alternative platforms and clients. Ultimately we are going to see services offering real-time communication of any type to any web enabled device using the web as the network.

The following lists aren’t 100% comprehensive and as time goes by I’ll probably create some pages on my site dedicated to this information.

Here’s the list of services I’m aware of and the type of service they offer and technology they use.

BeaconPush - [Web browser]
Google Channel API (Python/Java) – Part of Google App Engine – [Web browser]
DataSift – [Server to server, WebSockets, HTTP Streaming]
Kwwika – [Web browser, mobile, server to server, Desktop]
Hookbox – [Web browser]
Notifo – [Mobile]
Notify.io – [Desktop, IM, Email]
Notify.me – [Mobile, Desktop, IM, Email, XMPP]
PubNub – [Web browser, mobile, server to server, Desktop]
Pusher – [Web browser]
Superfeedr – [Server to server, PubSubHubbub, WebSockets, HTTP Streaming]
Twilio – [Mobile]
Urban Airship – [Moblie]

A quick not of the key technologies behind these types of service:

Comet (Note: this version by Jacobolus (Jacob Rus) is by far the best description. The current wikipedia description, IMHO, is inaccurate and incomplete and was ruined by Damiens.rf. Damiens.rf even put Greg Wilkins, who contributes to Jetty and Cometd, off contributing to Wikipedia all together.)
Growl
HTTP Streaming
PubSubHubbub
XMPP
Webhooks
WebSockets

Note: I originally wrote this as a comment over on Anil Dash’s post on Delivery as a Service but I thought it was good content so it deserved a post of its own.

Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub

Phil Leggetter — Fri, 07 Jan 2011 01:30:32 +0000

There are a number of ways of delivering data in real-time but until recently it has looked like PubSubHubbub, with the backing of Google, was going to be the preferred method. However, the past couple of weeks have seen a couple of interesting developments which could indicate that the developer community may actually prefer HTTP Streaming.

The emergence of the real-time web has seen an increase in the visibility of technologies that facilitate the delivery of data in real-time. Twitter was most probably the catalyst for this due to the many high profile cases where Twitter has been able to deliver the news before any other traditional news medium; the Hudson river plane crash is probably the best example of this. Some of the real-time technologies include PubSubHubbub, RSSCloud, Comet, XMPP, MQTT, Adobe LiveCycle, Google Wave Protocol, WebHooks, WebSockets and HTTP Streaming to name but a few.

We’ve also seen an increase in the number of real-time services over the past year who have used these technologies. Services such as Beacon, DataSift, Google Buzz, Kwwika (disclosure: author is a founder), notify.me, PubNub, Pusher, Superfeedr and of course Twitter. You can also find a number of other real-time APIs in our directory.

HTTP Streaming has been generally associated with Ajax in the past. In fact the Wikipedia entry for HTTP Streaming (under the Push Technology page and listed as HTTP server push) talks only about “sending data from a web server to a web browser.” This is out of date and HTTP Streaming is now much more than this. HTTP Streaming takes advantage of the fact that the Internet infrastructure has been built with HTTP in mind (as does PubSubHubbub). HTTP is fully supported so as well as using this protocol to distribute your static content such as HTML, images, CSS and JavaScript why not use it to distribute real-time data as well. The part of the Wikipedia definition for HTTP Streaming that is correct is:

Generally the web server does not terminate a connection after response data has been served to a client. The web server leaves the connection open such that if an event is received, it can immediately be sent to one or multiple clients.

A client in this context doesn’t have to be a web browser. It can be another web server, a desktop app, a mobile phone app, an embedded program running on a piece of hardware, a web application; basically any web enabled device capable of making a persistent HTTP connection.

This might be why services such as Superfeedr, who consistently champions PubSubHubbub, have introduced support for HTTP Streaming and why new services like DataSift has provided support from almost day one.

So, why are services starting to offer HTTP Streaming? The first thing you may think is that a persistent HTTP connection might be a faster way of receiving data than PubSubHubbub and it’s intermittent HTTP Push requests. Surprisingly this isn’t supposed to be the case since “HTTP 1.1 reuses TCP connections by default” as I recently found out.

One thing that PubSubHubbub does require is that the push notifications have to be made to a web server. This means that PubSubHubbub is highly unlikely to be used for real-time client data delivery because client applications don’t tend to run their own web server. Therefore HTTP Streaming is a more accessible real-time data delivery mechanism since any technology that can make a web request, and hold a persistent HTTP connection, can receive real-time push notifications. This means that by offering a HTTP Streaming API a service can be consumed by anything from a hardware embedded system to a mobile application as long as they are connected to the Internet.

The other thing that PubSubHubbub does is define the message format. This can be seen as a positive and a negative but since we are seeing JSON continuing to win over XML as the preferred data format it looks like PubSubHubbub will have to evolve away from XML to keep up as this question on Quora suggests.

This is an exciting trend which will most probably continue and will lead to us seeing truly real-time applications on any web-enabled device. It certainly doesn’t signal the end of the road for PubSubHubbub, which has its roots firmly in RSS (and XML), along with so much of the Internet. However, HTTP Streaming could become defacto standard for client push applications.

Photo via Blake Patterson

Originally posted on Programmable Web

XMPP PubSub or PubSubHubbub for real-time server push?

Phil Leggetter — Fri, 17 Sep 2010 08:32:00 +0000

In case you hadn’t noticed, when I’m not getting hit by flying trout, I like to think of myself as a bit of a real-time web guy. However, there’s one thing I’d like clarification on, and I’m confident somebody will be able to shed some light on this:

What are the differences between XMPP PubSub and PubSubHubbub and in what situations should each be used?

I do think I know some details about this but I’d like what I think I know to be confirmed or indeed corrected.

My knowledge of the real-time web, and specifically real-time client push, comes from working for Caplin Systems who pioneered building Comet servers around 10 years ago. From working with Caplin Systems and with Comet servers I’ve gained an understanding of how connections are made using different technologies and maintained in different scenarios. I also believe that I have a good understanding of what the best connection method is to deliver real-time data whether the delivery be from server to client using real-time client push or server to server using PubSubHubub or XMPP PubSub.

Persistent Connections

Unless the frequency of data updates is very low, and if you want to truly deliver data in real-time, the best type of connection is a persistent one.

As well as a persistent connection being able to deliver data faster it also means that your data subscriber does not have to handle a new HTTP connection for each piece of data it receives. This can actually be a really big thing.

As part of building Kwwika I wrote a demo which integrated the real-time server push capabilities of Superfeedr with the real-time client push capabilities of Kwwika to build a real-time news reader (RTNR) (blog article). In order to receive updates from Superfeedr I had to implement a publisher-subscriber outside of the Kwwika infrastructure. I decided to write a PubSubHubbub implementation in ASP.NET so that I could use a few other .NET Kwwika components to easily integrate with Kwwika.

So, each time Superfeedr has an update for the RTNR it establishes a HTTP connection to the RTNR PubSubhubbub server and sends the update which is then parsed and pushed into Kwwika to be distributed to subscribing web clients. If the RTNR is subscribing to something that is updating really frequently (such as “google” using Superfeedr track) then that can mean the RTNR server has to handle a lot of HTTP requests. Since I am running the demo on a small Amazon Windows instance running IIS it doesn’t take all that long, under a heavy load, for the server to stop responding.

There must be a better way!

Real-time client push

By way of a comparison let’s first take a look at real-time client push.

Before WebSockets provided the simplest way (but as yet not the most reliable since the WebSocket standard is not set in stone and isn’t supported cross-browser) to achieve real-time push to a web browser the best connection method was to maintain a streaming and persistent HTTP connection from the publisher to the client subscriber (using an IFRAME, XMLHttpRequest or XDomainRequest). The persistent connection means that the connection has already been established so as soon as the publisher has any new data for the subscriber it can instantly be pushed to it. This means that the data can be delivered to the web browser as close to real-time as possible.

Real-time Server Push

PubSubHubbub

PubSubHubbub turns things on it’s head in comparison to real-time client push (it actually uses HTTP as it was designed and it’s HTTP streaming that uses things in reverse. See Reverse Ajax.). In this case the publisher still pushes the new data to the subscriber but it does it using a HTTP request. The problem with this is that for each push a new HTTP connection needs to be established and the data then needs to be transferred. Establishing a connection takes time and resources so clearly a single persistent connection is a better solution. This is where I think XMPP PubSub is a better solution.

XMPP PubSub

If you need your data to be delivered in truly real-time (or as close to real-time as web technology will allow) then you have to use a persistent connection between the publisher and the subscriber. It’s my understanding that since XMPP was developed as an instant messaging protocol that it uses a persistent connection.

Conclusion

Here are things as I see them:

If you want your data in real-time you should use a persistent connection between the publisher and subscriber.
If you are making a server to server subscription to data that doesn’t update all that often and instant real-time doesn’t matter then PubSubHubbub is fine.
If you are making a server to server subscription to data that updates very frequently then you need to use a persistent connection and XMPP PubSub is a must.

One thing to also consider is if you are subscribing to multiple sources which update frequently then maintaining a persistent connection to each of those sources, assuming they even allow you to, takes a considerable amount of effort and potentially resource. This is where a service like Superfeedr comes in to its own. They manage the subscriptions and connections to the sources for you which means you only need to maintain one persistent connection to Superfeedr. They do all the heavy lifting so you don’t have to.

Permalink
| Leave a comment »

Using real-time web software and technology to distribute events

Phil Leggetter — Sat, 27 Feb 2010 11:08:36 +0000

I’ve just come across William Vambenepe’s blog and an article called “Waiting for events (in Cloud APIs)“ where he discusses how an event system is missing from cloud vendor APIs.

With my interest in Comet I straight away thought of this as a solution to the notification requirement and it’s interesting that the article goes on to talk about subscription management and then moves on to thinking about the delivery mechanism which are all key aspects of a good comet implementation:

How do you deliver notifications? Do you keep HTTP connections open through tricks similar to how self-updating web pages work (e.g. COMET, long polling and soon WebSockets)? Or do you just provide a listener endpoint to which the notifier tries to connect (which, in the case of public cloud deployments, means you need to have a publicly-addressable listener, but hopefully not on the same Cloud infrastructure). Do you use XMPP? AMQP? Email? Can I have you hold my events and let me come pull them?

However, when considering a notification system such as this you need to consider how frequently events would actually occur. How often would a new virtual machine in the cloud be spun up? How frequently do virtual machines unexpectedly crash? In relation to events in other scenarios such as stock rate updates or events within the Premier League between 15:00 and 17:00 on a Saturday, probably not all that frequently.

You also need to think about what would be consuming the event. Would it be an application with a user interface that prompts the user to take action or would it be an application with built in logic to handle a virtual machine failure event? If the application is running on a server then there is no reason why that server couldn’t be informed of the event via a single call to a web service, or maybe using something such as PUbSubHubbub, rather than holding open a connection to the cloud API – William describes this as a “listener endpoint”. If the application consuming the event is a web application, maybe the application is the cloud vendors VM dashboard, then Comet would be an ideal choice for instant notification.

Frequency of events and the event consuming application are two key things to consider when thinking about the best way to distribute and consume events using real-time web technologies. As an event-based API developer you may need to consider distributing your events in a number of ways to give application developers the choice.

Are there any notification systems out there that can be used by an API developer to easily distribute events in either way?

Why Comet is key to the Truly Real-Time Web

Phil Leggetter — Sat, 14 Nov 2009 21:01:57 +0000

A lot of websites use HTTP Polling to update data on on their website or application without the need for the user refreshing the website. This is clearly better than a purely static page, but during the time between each poll the data on the page has the potential of being out of date (stale) so to make a decision based on the information on a potentially stale page can be a risky one.

The normal example given to show the benefits of using Comet to achieve full HTTP streaming and the importance of non-stale data is of a trader making a trade on an out of date price. Another example is given where a person places a bet using odds that are on screen but do not match up with the odds in the betting system.

I just read a post on Comet Daily about a company called Frozen Mountain who have just released a component that adds Comet Support for IIS. Along with this they’ve released a video which doesn’t cover the standard example.

It’s possible that in this example that a polling solution might actually be just as suitable but this is a good demonstration of the benefits of Comet. In the example Comet provides almost instant feedback on the number of available tickets to potential buyers. This has two benefits. The first is that the user (Bob in the video) has less of a chance of missing out on his tickets. The second is more of a commercial benefit in that the user gets instant feedback on the popularity of the event and availability of the tickets and is more likely to make an instant purchase.

Until a solution such as Comet is used the data that a user sees, in a event booking, trading, betting or even search system, cannot be classed as part of the truly real-time web.

Caplin Systems Blog Post – What is the real-time web?

Phil Leggetter — Tue, 28 Apr 2009 15:27:40 +0000

About a week ago I wrote a blog article at work entitled “What is the real-time web?“. I’m seeing even more tweets and posts about the real-time web and I’d love to hear if other people agree with my opinion on what the real-time web is.

View the What is the real-time web? article here.

What is the “real-time web”?

Phil Leggetter — Mon, 20 Apr 2009 11:00:54 +0000

Are we really seeing the real-time web?

The phrase “real-time web” has been streaming its way around the Internet for a while now. It’s presently being used to describe information being available in search results as soon as it has been published by its author. Examples of this are Twitter or FriendFeed search.

As far as I’m aware it was Robert Scoble who made this phrase mainstream. Here’s an example from the start of February, but by this time Robert had been using the term for a while and a quick Google finds the term used as early as June 2008:

Let’s do a search for anyone who has written about the Canon 5D MK II but lets constrain that to posts that have at least one like and at least four comments. Here’s the search. Note that the post I wrote just one minute ago is already in the results page. This is the real-time web. – Robert Scoble (http://scobleizer.com/2009/02/09/is-the-real-time-web-a-threat-to-google-search/)

Whilst this is an example of something being almost instantly available from search does it really qualify as being “real-time”?

The first thing to do, as others have done, is understand what the term “real-time” means. Rather than going into this I recommend you have a read of the wikipedia definition of real-time and watch a Kaazing presentation on What is the Real-Time Web. There have been fewer attempts at defining “real-time web”. Even Wikipedia presently (19/04/2009) lacks a definition.

For me for something to be real-time web it should meet the following criteria:

Get up to the second information on a chosen subject.
Make an initial request for information to be delivered across the Internet. Subsequent updates to that information, or information stream, should be delivered as soon as possible without the need to make a additional requests.
Initial information and subsequent updates should be delivered in an acceptable amount of time.

The real-time web experience

If I go to my twitter homepage I don’t want to have to refresh the page, by clicking the refresh button, or hitting the F5 key, in order to see new messages from the people that I’m following. Twitter doesn’t really want users doing this either as it puts load on their server – but let’s leave this for another time.

Twitter

This lack of real-time updates on the Twitter site has lead to a surge in Twitter client applications such as TweetDeck and Thwirl. These clients simulate real-time by polling the Twitter API at intervals. I’d like to highlight the word “simulate” again and clarify what polling is. What the Twitter client applications are actually doing is making requests to Twitter at periodic intervals to see if the user has had any messages and if they have it displays them, if there are no new messages then it’s a wasted request.

TweetDeck

Twhirl

A much better model would be for Twitter to notify the client that new messages are available by passing the additional messages straight to the Twitter client.

Real-time web delivery time

What determines the amount of time that is acceptable between a message being published and it being received by the consumer (user) depends on the context of its use. For something such as instant messaging or a Twitter conversation the time between publication and consumption can be reasonably large, maybe 10 seconds. However, if the publisher is a trading system sending a price and the consumer is a trader 10 seconds is far too long. By the time the price reaches the trader it may be out of date.

Real-time technology

XMPP and SUP have been touted as the technology to deliver the real-time web but they appear to be server side solutions or require clients with knowledge of the specific technologies. For a web solution people start to turn to Comet server solutions. Caplin Systems – the real-time web company – have had the technology available to deliver real-time data for many years now so I wanted to understand why, only now, has the idea of a “real-time web” really taken off.

Caplin Systems

Caplin Systems have a high performance scalable real-time comet server called Liberator that can push updates to subscribing clients as soon as it receives information. The clients are:

Java clients: standalone application or applet
.NET clients such a windows desktop application or web page embedded SilverLight widget
JavaScript clients: any website such as http://twitter.com or http://friendfeed.com

Twitter and FriendFeed don’t use Liberator but they could easily do so. All that needs to be done to integrate Real-time Rich Internet Application (RTRIA ?) functionality to an existing web site is include a JavaScript library called SL4B (StreamLink for Browsers). The RTRIA functionality can then be added to an existing site in the same way that RIA functionality can be introduced to a website using other JavaScript libraries such as JQuery. However, in this case the RIA functionality that can be added is a true real-time web experience of data being pushed to the client as soon as it becomes available, without any need for the user to check to see if any new information is available.

Conclusion

My opinion is that the instant availability of information via search straight after it’s published doesn’t really justify the term “real-time web”. To be truely real-time the information needs to be pushed to the user. The true real-time web experience and the technology to achieve this is already here but at the moment few people are aware of it and therefore it’s not in mainstream use. The ideas behind Comet, the Liberator server and Liberator clients provide technology and infrastructure to push data to clients without the need for polling or a user driven information request. Scalability and side by side integration with existing systems are already possible which means there’s no reason why existing clients such as websites can’t start taking advantage of it.

Some relevant links: