Since PubSubHubbub has been around for a while it’s relatively easy for a developer to start using a hub such as Guzzle Ayup (Ayup for short). There are a good number of PubSubHubbub publisher and subscriber clients that can be used. There is also a reasonable amount of documentation on the subject including the Google PubSubHubbub home page and Ayup’s documentation.

Ayup was created by a web agency based in Paris called Lemonchik following the requirement to provide notifications to a real-time theme-based news aggregator called Guzzle and for future web and iOS applications that will require real-time notifications. They were so happy with the final implementation that they decided to build a front-end console and offer it as a public service.

Marca Tatem of Lemonchik (and Guzzle Ayup) explains a bit more about the internals of the the service:

The internals of Ayup are fun to look at. It’s a ruby application with a Sinatra web-service, and everything is happening in high-performance, RAM only, Resque queues (backed with Redis). Ayup structure is in itself completely scalable, adding a server with hundreds of new workers is a matter of minutes.

Ayup offer a simple pay for what you use pricing policy and allow you to set a notification limit which can be handy in keeping costs down when subscribed feeds update more than expected. They also offer something called Virtual hubs which Marca explains as:

Virtual Hubs are PubSubHubbub hubs content publishers can create to push free notifications to subscribers. For example, let’s say that you want people to be able to receive instant push notifications each time you publish a story, you simply create a virtual hub ([YourHubName].ayup.us) people can subscribe to (with a subscribe request) and notifications will be sent to their http callback for free.

Marca continues to explain why Virtual Hubs are a good way of encouraging simple and free access to your content:

The difference between a virtual hub and the actual ayup’s hub is that with a virtual hub, you can only send a subscribe request for a topic that belongs to the virtual hub’s owner. In other words, if you create a virtual hub, I can’t send a subscribe request for MacRumor’s RSS feed. If I want to subscribe to many different feeds without limitations, then I’ll have to create an Ayup account and pay for sent notifications.

Ayup’s focus for the near future is to keep things simple by delivering a good experience for developers and good quality of service such as fast and constant notifications and clean ATOM.

At the time of writing there are only two hosted PubSubHubbub hub services (Hub implementations and hosted services). The first well-known service Superfeedr and now there is the welcome addition of a second in Guzzle Ayup

Originally written by me and posted on Programmable Web

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. XMPP PubSub or PubSubHubbub for real-time server push?
3. Basic Authentication against the Superfeedr HTTP PubSubHubbub API using a .NET HttpWebRequest

A big thanks to @aboynejames for suggesting I do the talk and Bruce Scharlau for inviting me along. I’d also like to thank everybody who attended and made me feel very welcome and certainly seemed interested.

Feedback

If you where there then please leave a comment letting me know what you thought or just to let me know who you are. If you’d rather email you can get me using phil@leggetter.co.uk.

Video

Here’s a video of the first 10 minutes of the talk. Unfortunately the video recorder ran out of batteries.

Presentation and Notes

Unfortunately I can’t find a way of showing my slide notes in the above embeded Google Presentation so here they are in pretty rough form.

What is Real-Time?

Respond or react within a predictable time after an event has occurred.

There may also be a maximum time constraint.

Because after that maximum time the information loses value and context.

Maybe we should really call it:
Right-Time

When does Real-Time matter?

When does NOW matter?

When do you think being informed of something the instant it occurs really matters?

• Critical systems.
• Live events where things lose context.
• Systems requiring synchronisation:
  The 2Screen experience e.g. TellyLinks.com
• Interactive/collaborative things e.g. chat, Google Wave, Google Docs (docs, spreadsheets, drawing)
• Sensor data e.g. Car ABS breaks
• Trading: You’ve got to be sure you are seeing the correct trade price in order to be able to make a decision on whether to trade or not.
• Betting: You want to make sure you bet at the right odds and you don’t want clients to be able to bed when the odds have changed.

Evolution of the Web

From Pull to Push

Although I’m going to use the Web Browser as the example client the client could of course be anything. Another web server any desktop application, a mobile application, an embedded app on any device.

I’m also going to focus on HTTP but the mechanisms could be applied to standard TCP socket connections.

HTTP Pull (Request, Response)

Note: client. This can be a web browser but doesn’t have to be.

1) Each request would result in a page reload.
2) Event occurs on server there’s no way of the server telling the browser.
3) e.g. chat pages where you would need to click the “Refresh” button to see what other users are saying

HTTP Polling

A user hitting F5 or clicking the refresh button is one manual example.

1) REFRESH-META
2) Then frames
3) XMLHttpRequest -> Outlook web access (ActiveX object)

Check if the information has changed by polling.

The problem with this is you can easily waste requests and it can be resource intensive for both the client and the server.

A large amount of services still take this approach although the web, and the technologies, have evolved to allow for real-time push.

For quite some time I’ve said if a service uses HTTP Polling when push technology is available and a much better option that it’s not real-time

HTTP Push

In this diagram you could replace the Client with Server. The initialization could be classed as a subscription or webhook.

Various techniques have been developed to hold a connection open to a web server from a browser.

Difficulties :

* Proxies/Firewall
* Connection reselience
* Browsers
* Messaging API

Related posts:

1. Delivery as a Service and Data as a Service in 2011
2. XMPP PubSub or PubSubHubbub for real-time server push?
3. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub

Here’s my comment in full slightly edit to better suit a blog post.

As you state the Twitter API is not a real-time protocol. I’m assuming that you will be using one of Twitters streaming APIs – either the filter API method or maybe the users streams API.

Although not a real-time protocol these are examples of using a HTTP Streaming API to receive instant notifications. HTTP Streaming seems to be becoming the API technology of choice when the speed of notifications really matters and I think we are going to see a lot more APIs offer this. I wrote an article recently on Programmable Web that covers this topic a bit further and discusses HTTP Streaming verses PubSubHubbub.

So, I would suggest you update the real-time protocols section and state HTTP Streaming being another method for real-time data delivery (still not strictly a protocol), use the Twitter API as an example of an API using HTTP Streaming to deliver data in real-time and also update the docs links to point to dev.twitter.com.

Hope this information is useful.

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. Caplin Systems Blog Post – What is the real-time web?
3. Comment on The Rise of Multi-user, Multi-device Tablet Applications by Kate Ho

The emergence of the real-time web has seen an increase in the visibility of technologies that facilitate the delivery of data in real-time. Twitter was most probably the catalyst for this due to the many high profile cases where Twitter has been able to deliver the news before any other traditional news medium; the Hudson river plane crash is probably the best example of this. Some of the real-time technologies include PubSubHubbub, RSSCloud, Comet, XMPP, MQTT, Adobe LiveCycle, Google Wave Protocol, WebHooks, WebSockets and HTTP Streaming to name but a few.

We’ve also seen an increase in the number of real-time services over the past year who have used these technologies. Services such as Beacon, DataSift, Google Buzz, Kwwika (disclosure: author is a founder), notify.me, PubNub, Pusher, Superfeedr and of course Twitter. You can also find a number of other real-time APIs in our directory.

HTTP Streaming has been generally associated with Ajax in the past. In fact the Wikipedia entry for HTTP Streaming (under the Push Technology page and listed as HTTP server push) talks only about “sending data from a web server to a web browser.” This is out of date and HTTP Streaming is now much more than this. HTTP Streaming takes advantage of the fact that the Internet infrastructure has been built with HTTP in mind (as does PubSubHubbub). HTTP is fully supported so as well as using this protocol to distribute your static content such as HTML, images, CSS and JavaScript why not use it to distribute real-time data as well. The part of the Wikipedia definition for HTTP Streaming that is correct is:

Generally the web server does not terminate a connection after response data has been served to a client. The web server leaves the connection open such that if an event is received, it can immediately be sent to one or multiple clients.

A client in this context doesn’t have to be a web browser. It can be another web server, a desktop app, a mobile phone app, an embedded program running on a piece of hardware, a web application; basically any web enabled device capable of making a persistent HTTP connection.

This might be why services such as Superfeedr, who consistently champions PubSubHubbub, have introduced support for HTTP Streaming and why new services like DataSift has provided support from almost day one.

So, why are services starting to offer HTTP Streaming? The first thing you may think is that a persistent HTTP connection might be a faster way of receiving data than PubSubHubbub and it’s intermittent HTTP Push requests. Surprisingly this isn’t supposed to be the case since “HTTP 1.1 reuses TCP connections by default” as I recently found out.

One thing that PubSubHubbub does require is that the push notifications have to be made to a web server. This means that PubSubHubbub is highly unlikely to be used for real-time client data delivery because client applications don’t tend to run their own web server. Therefore HTTP Streaming is a more accessible real-time data delivery mechanism since any technology that can make a web request, and hold a persistent HTTP connection, can receive real-time push notifications. This means that by offering a HTTP Streaming API a service can be consumed by anything from a hardware embedded system to a mobile application as long as they are connected to the Internet.

The other thing that PubSubHubbub does is define the message format. This can be seen as a positive and a negative but since we are seeing JSON continuing to win over XML as the preferred data format it looks like PubSubHubbub will have to evolve away from XML to keep up as this question on Quora suggests.

This is an exciting trend which will most probably continue and will lead to us seeing truly real-time applications on any web-enabled device. It certainly doesn’t signal the end of the road for PubSubHubbub, which has its roots firmly in RSS (and XML), along with so much of the Internet. However, HTTP Streaming could become defacto standard for client push applications.

Photo via Blake Patterson

Originally posted on Programmable Web

Related posts:

1. Basic Authentication against the Superfeedr HTTP PubSubHubbub API using a .NET HttpWebRequest
2. Comment on Real-Time Delivery Explained post by feedmyinbox
3. Delivery as a Service and Data as a Service in 2011

What are the differences between XMPP PubSub and PubSubHubbub and in what situations should each be used?

I do think I know some details about this but I’d like what I think I know to be confirmed or indeed corrected.

My knowledge of the real-time web, and specifically real-time client push, comes from working for Caplin Systems who pioneered building Comet servers around 10 years ago. From working with Caplin Systems and with Comet servers I’ve gained an understanding of how connections are made using different technologies and maintained in different scenarios. I also believe that I have a good understanding of what the best connection method is to deliver real-time data whether the delivery be from server to client using real-time client push or server to server using PubSubHubub or XMPP PubSub.

Persistent Connections

Unless the frequency of data updates is very low, and if you want to truly deliver data in real-time, the best type of connection is a persistent one.

As well as a persistent connection being able to deliver data faster it also means that your data subscriber does not have to handle a new HTTP connection for each piece of data it receives. This can actually be a really big thing.

As part of building Kwwika I wrote a demo which integrated the real-time server push capabilities of Superfeedr with the real-time client push capabilities of Kwwika to build a real-time news reader (RTNR) (blog article). In order to receive updates from Superfeedr I had to implement a publisher-subscriber outside of the Kwwika infrastructure. I decided to write a PubSubHubbub implementation in ASP.NET so that I could use a few other .NET Kwwika components to easily integrate with Kwwika.

So, each time Superfeedr has an update for the RTNR it establishes a HTTP connection to the RTNR PubSubhubbub server and sends the update which is then parsed and pushed into Kwwika to be distributed to subscribing web clients. If the RTNR is subscribing to something that is updating really frequently (such as “google” using Superfeedr track) then that can mean the RTNR server has to handle a lot of HTTP requests. Since I am running the demo on a small Amazon Windows instance running IIS it doesn’t take all that long, under a heavy load, for the server to stop responding.

There must be a better way!

Real-time client push

By way of a comparison let’s first take a look at real-time client push.

Before WebSockets provided the simplest way (but as yet not the most reliable since the WebSocket standard is not set in stone and isn’t supported cross-browser) to achieve real-time push to a web browser the best connection method was to maintain a streaming and persistent HTTP connection from the publisher to the client subscriber (using an IFRAME, XMLHttpRequest or XDomainRequest). The persistent connection means that the connection has already been established so as soon as the publisher has any new data for the subscriber it can instantly be pushed to it. This means that the data can be delivered to the web browser as close to real-time as possible.

Real-time Server Push

PubSubHubbub

PubSubHubbub turns things on it’s head in comparison to real-time client push (it actually uses HTTP as it was designed and it’s HTTP streaming that uses things in reverse. See Reverse Ajax.). In this case the publisher still pushes the new data to the subscriber but it does it using a HTTP request. The problem with this is that for each push a new HTTP connection needs to be established and the data then needs to be transferred. Establishing a connection takes time and resources so clearly a single persistent connection is a better solution. This is where I think XMPP PubSub is a better solution.

XMPP PubSub

If you need your data to be delivered in truly real-time (or as close to real-time as web technology will allow) then you have to use a persistent connection between the publisher and the subscriber. It’s my understanding that since XMPP was developed as an instant messaging protocol that it uses a persistent connection.

Conclusion

Here are things as I see them:

• If you want your data in real-time you should use a persistent connection between the publisher and subscriber.
• If you are making a server to server subscription to data that doesn’t update all that often and instant real-time doesn’t matter then PubSubHubbub is fine.
• If you are making a server to server subscription to data that updates very frequently then you need to use a persistent connection and XMPP PubSub is a must.

One thing to also consider is if you are subscribing to multiple sources which update frequently then maintaining a persistent connection to each of those sources, assuming they even allow you to, takes a considerable amount of effort and potentially resource. This is where a service like Superfeedr comes in to its own. They manage the subscriptions and connections to the sources for you which means you only need to maintain one persistent connection to Superfeedr. They do all the heavy lifting so you don’t have to.

Permalink
| Leave a comment  »

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. The benefits of Real-Time Push-Once
3. Chrome/Safari (webkit) + real-time push = always loading indicator

So, this doesn’t work:

string username = "username";
string password = "password";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("url-here");
request.Credentials = new NetworkCredential(username, password);

To work around this you need to force the authentication to be sent. I found the details of how to do this in a blog post by Ian Dykes here:

http://devproj20.blogspot.com/2008/02/assigning-basic-authorization-http.html

But for fullness here’s the code:

string username = "username";
string password = "password";
byte[] authBytes = Encoding.UTF8.GetBytes(username + ":" + password.ToCharArray());
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(authBytes);

Permalink

| Leave a comment  »

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. How to make a cross domain web request with SilverLight 2
3. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request

With my interest in Comet I straight away thought of this as a solution to the notification requirement and it’s interesting that the article goes on to talk about subscription management and then moves on to thinking about the delivery mechanism which are all key aspects of a good comet implementation:

How do you deliver notifications? Do you keep HTTP connections open through tricks similar to how self-updating web pages work (e.g. COMET, long polling and soon WebSockets)? Or do you just provide a listener endpoint to which the notifier tries to connect (which, in the case of public cloud deployments, means you need to have a publicly-addressable listener, but hopefully not on the same Cloud infrastructure). Do you use XMPP? AMQP? Email? Can I have you hold my events and let me come pull them?

However, when considering a notification system such as this you need to consider how frequently events would actually occur. How often would a new virtual machine in the cloud be spun up? How frequently do virtual machines unexpectedly crash? In relation to events in other scenarios such as stock rate updates or events within the Premier League between 15:00 and 17:00 on a Saturday, probably not all that frequently.

You also need to think about what would be consuming the event. Would it be an application with a user interface that prompts the user to take action or would it be an application with built in logic to handle a virtual machine failure event? If the application is running on a server then there is no reason why that server couldn’t be informed of the event via a single call to a web service, or maybe using something such as PUbSubHubbub, rather than holding open a connection to the cloud API – William describes this as a “listener endpoint”. If the application consuming the event is a web application, maybe the application is the cloud vendors VM dashboard, then Comet would be an ideal choice for instant notification.

Frequency of events and the event consuming application are two key things to consider when thinking about the best way to distribute and consume events using real-time web technologies. As an event-based API developer you may need to consider distributing your events in a number of ways to give application developers the choice.

Are there any notification systems out there that can be used by an API developer to easily distribute events in either way?

Related posts:

1. Government invests £200m in technology centres but no mention of Software or Information Technology
2. Marriage, Work/life integration, real-time web technology evangelism & Kwwika: A retrospective of my 2010
3. Which Rich Internet Application Technology will dominate?

I mentioned RTRIAs for the first time back in April 2009 in a post called “What is the real-time web” on the Caplin Systems Platformability blog, and since then there has been a real-time web explosion. The “real-time web” is now the buzz phrase around the Internet that everybody is jumping on. Back in April I set up a Google Alert for the term “real-time web”. Back then I got maybe one Google Alert a day for this term, probably less. Now, I get at least two dense emails a day from Google with people using the term for all sorts of things; real-time web stats, real-time analytics, real-time search, rssCloud, pubsubhubbub, the list goes on. This is not the real-time web! Whilst some of the things on this list will help  the web become truly real-time (rssCloud and pubsubhubbub may even form the back bone), none of these things give the user a truly real-time web experience.

My opinion about what the real-time web is has not changed from that first blog post. For me, it’s not just about data being available as soon as it’s been published. In this situation the user still has to actively make a request to see if any data is available. Instead it’s about making a single request for data (a subscription) and that data being pushed to your RIA to be consumed by the application user in real-time. Quite a few applications have implemented a polling solution (more on this later) but the experience presented to the user is not truly real-time.

"Real-Time?"

In my opinion the only way to guarantee being able to deliver truly real-time data (TRTD) to a web application is by maintaining a streaming connection to the source of the data using a continuous streaming HTTP connection. Most web applications use a technique called polling or long polling to retrieve their data which means that although the data is initially “fresh”, within seconds it could be out of date, or more information may be available. I’ll discuss server technologies further in a later post.

At the moment there are a number of technologies that allow you to develop RTRIAs. Ajax, Silverlight and Flash are the three mainstream solutions. The core barriers to RTRIAs are the scalability of server technology and access to truly real-time data.

Related posts:

1. Which Rich Internet Application Technology will dominate?
2. Comment on The Rise of Multi-user, Multi-device Tablet Applications by Kate Ho
3. Why client APIs are an important part of any real-time service