In this tutorial I’ll show how to:

• subscribe to a presence channel
• authenticate a subscription to a channel
• provide Pusher with additional information about a user
• display presence information on a product page for the “Who’s shopping?” widget

If you are desperate to see the demo in action you can see the Real-Time Web Store demo here.

Pusher Presence

To achieve the “Who’s shopping?” functionality I’m going to be using a feature in Pusher called presence. Presence provides you with additional information about a channel you are subscribed to so that you know:

• who is subscribed to that channel
• when new users subscribe
• when existing users unsubscribe (by either actually unsubscribing or navigating away from the page).

We are going to have a presence channel per product so that we know who is viewing each product.

User Info & Authentication

Subscribing to a presence channel

You subscribe to a presence channel in the same way that you do to any other channel but the name of the channel must have a presence- prefix. Presence channels are normal channels with two additions; authentication and presence information. With this in mind we are just going to update our application to use a presence channel.

The JavaScript that makes the subscription in our Razor view looks like this:

var productId = "@Model.ProductId";
var pusher = new Pusher("APP_KEY");
var channel = pusher.subscribe("presence-" + productId);

We also need to update the code in our StoreController to publish our stock events on the new presence channel:

var stockEvent = new StockUpdatedEvent(model, socketId);
ObjectPusherRequest request = new ObjectPusherRequest("presence-" + stockEvent.ProductId, "stockUpdated", stockEvent);
_provider.Trigger(request);

Note: If you are continuing where we left off in our last blog post there are a final couple of updates that are required to change the app to use the latest version of the Pusher JavaScript API. We recently released version 1.9 which introduced new connection state functionality and also a new connection object. So, update your Pusher script tag as follows:

<script src="http://js.pusherapp.com/1.9/pusher.js"></script>

And you’ll also need to update any pieces of code that access the socket_id via the Pusher instance. It should now be accessed via the new connection object as follows:

var socketId = pusher.connection.socket_id;

Getting User information

If Pusher is to send events about users subscribing to and unsubscribing from presence channels it needs information about the users. It gets this information from your application when the subscription request to the channel is made (pusher.subscribe('presence-channel')). Since we can’t really trust the web browser/client (it’s so easy to hack JavaScript running in a web browser) the Pusher library requests this information from your web server by making an AJAX call. By default this call goes to /pusher/auth and passes two parameters; channel_name, which is the name of the channel being subscribed to, and socket_id, which is a unique identifier for the current user’s connection to Pusher.

/pusher/auth/?channel_name=presence-pusher-tshirt&socket_id=<unique_socket_id>

When our application responds to this request we must provide an authentication signature to confirm that the user can subscribe to the channel and, importantly for our “Who’s shopping?” widget, information about the current user. The way we’ll handle this within our ASP.NET MVC application is by creating a PusherController with an Auth(string socket_id, string channel_name) action, and by using the authentication functionality within the PusherRESTDotNet library. This library is also available as a NuGet package.

Note: If you got the NuGet package as part of the last tutorial you’ll need to update it since the authentication functionality has just been added. You should also check that the .NET 3.5 runtime version of Newtonsoft.Json is added.

Handling the authentication request

As mentioned above, the Pusher JavaScript library will make a request to /pusher/auth when making the authentication request. Our new PusherController with Auth action does the following:

1. Fetches our Pusher credentials from the Web.config file.
2. Creates a new PusherProvider using the Pusher credentials
3. Creates a unique user_id for the presence channel
4. Creates an authentication string and returns that string as the Content of a ContentResult' with theContentTypeset toapplication/json` in response to the AJAX request.

For the moment this code doesn’t do any user authentication or provide any additional information about the current user.

using System;
using System.Configuration;
using System.Web.Mvc;
using PusherRESTDotNet;
using PusherRESTDotNet.Authentication;

namespace RealTimeWebStore.Controllers
{
    public class PusherController : Controller
    {
        public ActionResult Auth(string channel_name, string socket_id)
        {
            var applicationId = ConfigurationManager.AppSettings["application_id"];
            var applicationKey = ConfigurationManager.AppSettings["application_key"];
            var applicationSecret = ConfigurationManager.AppSettings["application_secret"];

            var channelData = new PresenceChannelData()
            {
                user_id = Guid.NewGuid().ToString()
            };

            var provider = new PusherProvider(applicationId, applicationKey, applicationSecret);
            string authJson = provider.Authenticate(channel_name, socket_id, channelData);

            return new ContentResult { Content = authJson, ContentType = "application/json" };
        }
    }
}

If we use one of the many web browser development tools available to us to inspect the authentication call within the browser we’ll see the JSON response coming back.

You’ll see the response contains a channel_data property which itself has a user_id with a unique Guid value and a user_info property with a null value. Pusher uses this user_id value to uniquely identify the user subscription to the presence channel. So it’s very important to make sure that each user has a unique ID.

Adding authentication

We’ve mentioned authentication a few times but as yet we haven’t authenticated the user. If the user has already logged (our app doesn’t have this functionality, but most do) in we can use the existing User.Identity or else we can just assign a guest identity to the user. Once we have a unique ID for the user we’ll also add some additional user_info to the channelData. The value of user_info can be anything you like from a simple string to a complex object. This gives you the ability to push as much additional information through Pusher and to the web page as you like. In our case we’ll just send through a timestamp which identifies how long the user has been on the site.

public ActionResult Auth(string channel_name, string socket_id)
{
    var channelData = new PresenceChannelData();
    if (User.Identity.IsAuthenticated)
    {
        channelData.user_id = User.Identity.Name;
    }
    else
    {
        channelData.user_id = GetUniqueUserId();
    }
    channelData.user_info = GetUserInfo();

    var provider = new PusherProvider(applicationId, applicationKey, applicationSecret);
    string authJson = provider.Authenticate(channel_name, socket_id, channelData);

    return new ContentResult { Content = authJson, ContentType = "application/json" };
}

Note: In our case we don’t really need to authorise a user but in other situations where the user needs to be logged in we can return a 401 HttpStatusCodeResult.

Who’s Shopping?

Now that we’ve got a PusherController that gives Pusher information about the user, we can start showing information about the user on the product page. You can get information about the users subscribed to presence channels by binding to the pusher:subscription_succeeded event on the presence channel object. The callback method for this event receives a members parameter which contains all the information about users subscribed to the channel.

First we’ll create some HTML within our web page where we are going to show “Who’s shopping?”. Then we’ll add the users to the HTML when pusher notifies us of them.

HTML

<div class="whos-shopping">
    <h3>Who's shopping?</h3>
    <ul></ul>
</div>

JavaScript

var pusher = new Pusher("006c79b1fe1700c6c10d");
var channel = pusher.subscribe("presence-" + productId);
channel.bind("pusher:subscription_succeeded", function(members) {

    members.each(function(member) {
        addMember(member);
    });

});

function addMember(member) {
    var enteredSite = new Date(member.info.timestamp);
    var now = new Date();
    var timeOnSite = (now - enteredSite);
    var li = $("<li data-user-id='" + member.id + "'>" +
                    member.id + " here for " +
                    toReadableTime(timeOnSite) +
               "</li>");
    $(".whos-shopping ul").append(li);
};

Note: The members object comes with a handy each method to make iterating the members collection really easy.

Of course new users can navigate to the page and existing users can leave it so the Pusher JavaScript library also exposes pusher:member_added and pusher:member_removed events on the presence channel object. When these events fire we should add or remove the user as required.

channel.bind("pusher:member_added", function(member) {
    addMember(member);
});
channel.bind("pusher:member_removed", function(member) {
    removeMember(member);
});

function addMember(member) {
    /* as before */
};

function removeMember(member) {
    $(".whos-shopping ul li[data-user-id='" + member.id + "']").remove();
};

With this in place we now have a fully functioning “Who’s shopping?” widget that shows the current user who else is viewing the same product as they are.

As mentioned in the opening paragraph, the theory here is that if shoppers can see that others users are viewing the same product it might give them that little push they need to take the plunge and make that purchase “while stocks last”.

There are a few refinements and enhancements that could be made to this widget such as filtering out the current user from the “Who’s shopping?” list or possibly showing them which one they are. You could also use the notification system from last time to notify the shopper when another shopper joins or leaves the product page. And, of course, you could add some user chat functionality to get the users discussing the product and really engaging. You could also have a staff member user who could answer any questions that the shoppers may have.

Just as last time all the code from this post is available in the real-time web store github repo. You can also see the Real-Time Web Store application up and running on AppHarbor. I’ve tried to link to relevant parts of the Pusher documentation throughout the post but if there anything that isn’t clear, if there’s anything that I’ve not provided enough detail on and it all just seems too ‘magical’, then please leave a comment or send an email to me (phil@pusher.com).

Here are some links to the key things covered in this post:

• Presence channels
• Presence channel events
• Authenticating Users
• Pusher REST .NET Library | NuGet package
• ASP.NET real-time web store in github
• Real-Time Web Store application up and running
• ASP.NET MVC 3

Addendum: What about WebForms?

The post above shows how to user the Pusher REST .NET library within an ASP.NET MVC application but it can just as easily be used within an ASP.NET WebForms app. The way I achieved this was by adding a new Generic HTTP Handler to our web app which will handle the authentication AJAX call.

In the code below the ProcessRequest method does the following things:

1. Fetches our Pusher credentials from the Web.config file.
2. Gets the values of the channel_name and socket_id parameters from the context.Request
3. Creates a new PusherProvider using the Pusher credentials
4. Creates a unique user_id for the presence channel
5. Creates an authentication string and returns that string as the response body of the AJAX request.

For the moment this code doesn’t do any user authentication or provide any additional information about the current user.

using System.Configuration;
using System.Web;
using PusherRESTDotNet;
using PusherRESTDotNet.Authentication;
using System;

namespace RealTimeWebStore
{
    public class AuthHandler : IHttpHandler
    {
        public void ProcessRequest(HttpContext context)
        {
            var applicationId = ConfigurationManager.AppSettings["pusher-application-id"];
            var applicationKey = ConfigurationManager.AppSettings["pusher-application-key"];
            var applicationSecret = ConfigurationManager.AppSettings["pusher-application-secret"];

            var socketID = context.Request["socket_id"].ToString();
            var channelName = context.Request["channel_name"].ToString();
            var channelData = new PresenceChannelData()
            {
                user_id = Guid.NewGuid().ToString()
            };

            var provider = new PusherProvider(applicationId, applicationKey, applicationSecret);
            string authJson = provider.Authenticate(channelName, socketId,  channelData);

            context.Response.Write(authJson);
        }

        public bool IsReusable
        {
            get
            {
                return false;
            }
        }
    }
}

Finally we need to configure our handler in the application Web.config file. We want the ProcessRequest method of our handler to be invoked for any call to /pusher/auth. To do this we just add a handler to the httpHandlers element and specify our handler, RealTimeWebStore.AuthHandler as the handler:

<system.web>
  <!-- other config -->
  <httpHandlers>
    <add verb="*"
         path="/pusher/auth/"
         type="RealTimeWebStore.AuthHandler" />
  </httpHandlers>
</system.web>

Related posts:

1. The easiest way to add real-time functionality to an ASP.NET e-commerce application
2. Kwwika Powered Real-Time Opta Sports Cricket Widget
3. Recent article in .net magazine: WebSockets – Code a real-time survey

One of the problems with existing e-commerce solutions can be trying to buy something which, whilst you’ve been online, has sold out. This can be a big problem for high demand products like gig tickets or sporting events such as the Olympics. A really good way of avoiding this is to show the customers just how fast tickets are selling. The added benefit of this, from a business point of view, is that it can give users that little push they needed to make that purchase. In this tutorial I’ll show how to:

• establish a connection to Pusher in the view
• subscribe to a product channel
• bind events from this channel to our stock indicator in the view and update the stock level
• send stock level update from our server when someone buys something

Getting started

To demonstrate adding real-time functionality to an ASP.NET website I’m going to start with a simple one page e-commerce site which has one product: our cool Pusher t-shirt (Sorry, they’re not really for sale. You’ve got to earn them!). On the product page there is a counter which tells the customer how many t-shirts we have in stock.

To start off with our e-commerce website is a simple ASP.NET MVC3 app with:

• A StoreController with two actions. One to display the default view and one to handle the “Buy” button being clicked and the stock level being decreased.
• A single Razor view displaying the single product
• Two models; one for the product called ProductModel and one for images called ProductImage
• A ProductRepository where we store our products

Connect to Pusher

As I’ve already said, adding real-time functionality is really simple – that’s the point! Firstly we have to include the Pusher JavaScript client library. We’ll add this to the view for the moment although we might want to make the include part of the site layout in the future.

Note: If your e-commerce page was being served up over HTTPS you would include our library over HTTPS too to avoid any browser warnings.

Since we are using ASP.NET MVC we’ve already got jQuery included from the _Layout.cs.html so we can wait until the page has loaded and connect to Pusher. The script include and the code that connects to Pusher looks as follows:

<script src="http://js.pusherapp.com/1.8/pusher.min.js"></script>
<script>
    $(function() {
        var pusher = new Pusher("006c79b1fe1700c6c10d");
    });
</script>

Subscribe to the product channel

Now that we’ve connected to Pusher we can push events from our StoreController to all connected browsers if any information changes about the product. The main piece of information that customers might want to know about is if the stock level changes. These t-shirts could sell out fast!

We need to choose a channel name to subscribe and publish to and the obvious choice is to use the unique product id for this. We’ll also prefix it with product- just so it’s really clear that the channels is for a product. We subscribe to this channel on the client as follows getting the product id from the Model that has been bound to the page:

var channel = pusher.subscribe("product-@Model.ProductId");

Bind to stock update events

Once we’ve got a channel object we can bind to events on it. We want to be informed when the stock level changes so let’s bind to an event called stockUpdated. We’ll also expect a JSON object to be passed which represents the ProductModel and will have the same properties on it. That way we can simply update the stock level in the page to match the value on the server by accessing the ProductModel.StockLevel value. We can also update the stock status:

channel.bind("stockUpdated", function(product) {
    $(".product .stock .level").html(product.StockLevel);
    $(".product .stock .status").html(product.StockStatus);
});

Trigger stock update events

As you can see, this is really easy to do. It’s just as easy to trigger the event on the server too. To do this we’re going to be using the C# REST API library within our StoreController. You can get the PusherRESTDotNet library from github or via NugGet.

Once we’ve added our PusherRESTDotNet reference we can add a few lines of code to our StoreController to instantly push any changes in stock level to all connected web browsers viewing our t-shirt page.

The first thing to do is to create a PusherProvider and pass in our Pusher details. In this example we are storing these details in the ASP.NET MVC3 Web.config.

Web.config

<configuration>
  <appSettings>
    <add key="application_id" value="APP_ID" />
    <add key="application_key" value="APP_KEY" />
    <add key="application_secret" value="APP_SECRET" />

    <!-- more settings -->
  </appSettings>

  <!-- more config -->
</configuration>

StoreController.cs

public class StoreController : Controller
{
  private IPusherProvider _provider;

  public StoreController()
  {
      string applicationKey = ConfigurationManager.AppSettings["application_key"];
      string applicaitonSecret = ConfigurationManager.AppSettings["application_secret"];
      string applicationId = ConfigurationManager.AppSettings["application_id"];
      _provider = new PusherProvider(applicationId, applicationKey, applicaitonSecret);
  }

  // more code here...
}

Once we’ve created our instance of the PusherProvider we can create a PusherObjectRequest to push the ProductModel, with an updated StockLevel following the purchase, to the clients. The serialisation of the model object is handled for us by the PusherRESTDotNet library. The full StoreController.Index HTTP POST handling action looks like this:

[HttpPost]
public ActionResult Index()
{
  bool bought = MvcApplication.ProductRepository.Buy(MvcApplication.BLUE_TSHIRT_ID);
  var model = MvcApplication.ProductRepository.GetProductById(MvcApplication.BLUE_TSHIRT_ID);

  if (bought)
  {
      ViewBag.Info = model.Title + " successfully bought";

      ObjectPusherRequest request = new ObjectPusherRequest("product-" + model.ProductId, "stockUpdated", model);
      _provider.Trigger(request);
  }
  else
  {
      ViewBag.Error = "There was a problem buying " + model.Title;
  }

  return View("Index", model);
}

Now, if you run your application and open two or more windows, as soon as a user in one window clicks the “Buy” button you’ll see that the stock level value instantly updates in all other windows.

Next time – displaying real-time “Who’s shopping” info with Pusher presence

We now have the stock level indicator changing in real-time. This tells the users that these t-shirts are selling out fast and gives them an added incentive to make that impulse buy. This is a simple yet effective change but there’s more we can do. What if we also show the user how many other users are viewing that product at the same time so that they know there is competition for the last few t-shirts? We can do that easily using Pusher’s presence functionality and I’ll cover that in my next Real-Time ASP.NET blog post which will also go into how to add Pusher user authentication to your ASP.NET web application.

In the meantime you can download the source of the Real-Time Web Store from github. The solution contains a project with the basic web store without any real-time Pusher functionality as well as a solution project. Here are some additional things you you might also want to try out:

• Changing the form submission when the user clicks “Buy” so that the POST to the Buy action it is made using a jQuery ajax call
• Making the change in stock levels more visually interesting. Try adding an effect to the number change or providing a growl-like notification
• Changing the CSS – it’s not my strong point

Related posts:

1. Adding a real-time "Who’s shopping?" widget to an ASP.NET Web App
2. Recent article in .net magazine: WebSockets – Code a real-time survey
3. Which Rich Internet Application Technology will dominate?

.net magazine issue 227 (August)

The article covers building a real-time survey using Ruby on Rails and Pusher. I also wrote about how Pusher can be used to progressively enhance an application in the way that has been touted as a “best practice” approach when adding JavaScript to a web application. I won’t got into any more detail, if you are interested you can pick up a copy I bought one.

This is a really big achievement for me and for Pusher, so I’d like to thank the guys at Pusher for taking me on and also for giving me the opportunity to write the article. It showed a lot of confidence in my ability from day one and that means a lot. Maybe it was a test?

Mugshot

I’ll keep this short as I don’t want to make it too much of a “look at me” post. I just want to post something as a “record of achievement” of something I’m proud of and as a thanks to the guys as Pusher.

Related posts:

1. Goodbye Kwwika. Hello Pusher!
2. What came before WebSockets?
3. Q&A: Are WebSockets ready for commercial use?

A very important question, that I recently heard from Roan Laverly of FreeAgent: “Is your product solving a hard enough problem?” Well, what problems was Collecta solving?

The first problem Collecta was solving was search, and the problem with building a search product in general is that the market is massively dominated by Google. Even Microsoft’s Bing and Yahoo! can’t really make a significant dent in the search market, so how can anybody compete? Collecta tried to compete by using real-time search results as their unique selling point – Google now has a real-time search option, and if a topic is trending real-time search results can form part of the general search results too.

Another problem that Collecta was solving was making real-time data accessible. So many companies that have real-time data are now opening up their own APIs and increasing the ease at which anybody can access that data. This means individuals and companies can now got direct to source and build their own search offering, or just extract just the data they need themselves. I’m quite sure that Collecta has some very clever technology that delivered their best guess (algorithmically computed) search results in real-time. But again, it doesn’t sound like that was enough.

Creating real-time technology components capable of consuming, processing and distributing large amounts of data was another problem that Collecta had to answer. Initially building the technology capable of doing this was difficult but now, a few years later, real-time technology components are much more widely accessible. So whilst the difficulty level is still high the problem isn’t as difficult as it once was. This in itself has meant that data companies can build their own APIs.

Then there’s the problem of creating a compelling application and products. A website showing real-time search results in a intuitive and informative way. Widgets showing search results in an easy to digest and embeddable format. The post on ReadWriteWeb provides a comment by one of their own writers, Jared Smith, on Collecta’s applications:

Collecta’s emphasis on a search experience that went beyond Twitter into photos and videos made it a great tool to truly watch a story unfold in real time. Their embeddable widget, which I used regularly on ReadWriteWeb’s event sites, was far more powerful than what Twitter provided and is still unmatched in my mind.

So, even though they were building a useful products it wasn’t enough. Did Jared pay for the widget, and if not would he have been willing to? Would enough users have been willing to pay?

The good news out of all of this is that it sounds like the Collecta technology is going to be open sourced. A quote on the ReadWriteWeb post states:

Collecta now says it will open source its software and is working with a variety of organizations to do so – including United Nations crisis relief projects.

Is the real-time search problem hard enough? We should keep our eyes on Topsy and Social Mention to see how they get on. What we can be sure of though is that with Collecta open sourcing their technology the real-time search technology question will now be easier than ever to answer.

Originally written by me and posted on Programmable Web

Related posts:

1. Problem solving lessons relearnt
2. How I approach problem solving in code?
3. Is It Finally the End for Real-time Search Engines?

My new role as Developer Evangelist at Pusher is the type of role that I’ve been trying to build for myself for a long time and it’s fantastic that Max and Damien at Pusher have given me this opportunity. The days that I’ve enjoyed most over the past year and a bit have been the days when I’ve been working with others, interacting online, attending events, writing blog posts to gauge opinion and share ideas, and learning from others. This is exactly what I’ll be doing in my new job. Awesome!

Me with co-founder of Pusher, Damien Tanner

What is Pusher?

Pusher is a hosted HTML5 WebSockets service that makes it possible to quickly, easily and securely add real-time functionality to any web, mobile or desktop application (anything connected to the Internet). This sort of thing is ideally suited to collaboration and communication apps, multiplayer games, real-time dashboards, social media integration, real-time messaging and a whole range of applications that have yet to be discovered.

Please drop me an email or send me a tweet if you have any questions at all.

Related posts:

1. Recent article in .net magazine: WebSockets – Code a real-time survey
2. Kwwika Silverlight API – Chat Example
3. Want to try out the Kwwika API but don’t want to register?

Since PubSubHubbub has been around for a while it’s relatively easy for a developer to start using a hub such as Guzzle Ayup (Ayup for short). There are a good number of PubSubHubbub publisher and subscriber clients that can be used. There is also a reasonable amount of documentation on the subject including the Google PubSubHubbub home page and Ayup’s documentation.

Ayup was created by a web agency based in Paris called Lemonchik following the requirement to provide notifications to a real-time theme-based news aggregator called Guzzle and for future web and iOS applications that will require real-time notifications. They were so happy with the final implementation that they decided to build a front-end console and offer it as a public service.

Marca Tatem of Lemonchik (and Guzzle Ayup) explains a bit more about the internals of the the service:

The internals of Ayup are fun to look at. It’s a ruby application with a Sinatra web-service, and everything is happening in high-performance, RAM only, Resque queues (backed with Redis). Ayup structure is in itself completely scalable, adding a server with hundreds of new workers is a matter of minutes.

Ayup offer a simple pay for what you use pricing policy and allow you to set a notification limit which can be handy in keeping costs down when subscribed feeds update more than expected. They also offer something called Virtual hubs which Marca explains as:

Virtual Hubs are PubSubHubbub hubs content publishers can create to push free notifications to subscribers. For example, let’s say that you want people to be able to receive instant push notifications each time you publish a story, you simply create a virtual hub ([YourHubName].ayup.us) people can subscribe to (with a subscribe request) and notifications will be sent to their http callback for free.

Marca continues to explain why Virtual Hubs are a good way of encouraging simple and free access to your content:

The difference between a virtual hub and the actual ayup’s hub is that with a virtual hub, you can only send a subscribe request for a topic that belongs to the virtual hub’s owner. In other words, if you create a virtual hub, I can’t send a subscribe request for MacRumor’s RSS feed. If I want to subscribe to many different feeds without limitations, then I’ll have to create an Ayup account and pay for sent notifications.

Ayup’s focus for the near future is to keep things simple by delivering a good experience for developers and good quality of service such as fast and constant notifications and clean ATOM.

At the time of writing there are only two hosted PubSubHubbub hub services (Hub implementations and hosted services). The first well-known service Superfeedr and now there is the welcome addition of a second in Guzzle Ayup

Originally written by me and posted on Programmable Web

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. XMPP PubSub or PubSubHubbub for real-time server push?
3. Basic Authentication against the Superfeedr HTTP PubSubHubbub API using a .NET HttpWebRequest

DataSift launched its Alpha service at TechCrunch Disrupt in September and describe it as a “real time social media filtering engine.” The initial buzz around DataSift was generated when Twitter agreed to give them access to the Twitter firehose, but they now have access to a much wider range of data including the Google Buzz, MySpace, SixApart, WordPress, Facebook and Digg. These sources of data within DataSift, sometimes called input services, are defined as Targets in the DataSift knowledge base.

Users can use Targets to create their own streams from within the My Streams section of the DataSift dashboard using a language called FSDL (Filtered Stream Definition Language). The Web editor used to define your streams is pretty simple but it does provide some basic syntax highlighting as well as validating your syntax whenever you save. FSDL also provides access to augmentation Targets through services such as Salience, TweetMeme, Peer Index, Klout and InfoChimps that allow streams to be augmented with third party data.

Once a stream has been defined you can choose to build your feed. At present this takes up to 60 minutes and the dashboard will show you the progress of the build as well as a host of other features including a data preview, a live example of the data, a graph showing matched stream items (iterations per minute) and the history of the feed definition.

Update: Nick Halstead (CEO of DataSift) has provided a clarification about the stream preview, the live stream feature and also when the stream can be used via the API:

The stream ‘preview’ does not need to be built for anything to work, you can define + attach to API immediately (or just hit ‘live’ tab to see live results) – the preview was to allow stream owners to demonstrate what the stream would offer.

You can see Nick’s full comment in the comments section below.

DataSift is encouraging its users to build feeds that are discoverable and accessible to other users, although it does offer a private feed option, by providing a number of options on the stream page including tagging, an area to encourage you to tell others about it on common social networks and a comments area to encourage users to interact and visits to the stream page. Recently added, most commented and top rated streams are also featured on the home section of the DataSift dashboard.

Once the stream has been built it can also be used in the definition of another user stream, and it in another stream and so on. DataSift really have exposed a lot of functionality and capabilities within their user dashboard and the documentation that they provide is quite thorough and really helps a user get to grips with creating streams reasonably quickly.

All these rich features would be a waste unless there was a way of accessing the data and using it with an application. Unsurprisingly, the DataSift API delivers by providing three endpoints, including access to filtered data in a paged manner, HTTP Streaming and an RSS endpoint. It also recently introduced a WebSockets Streaming API.

Although DataSift is still in Alpha it is offering what seems an affordable and very impressive service which should excite any developer interested in real-time technologies and data. The company’s access to a wide range of data sources, engaging and intuitive user dashboard and range of API endpoints should mean that most developers will have their technology needs met.

A good starting point to learn about DataSift is an interview by Robert Scoble with Nick Halstead, the CEO of DataSift (embedded above). The video is a little old but provides a good overview and an example of creating a stream. If you’ve any comments or questions about DataSift please leave a comment here. After that you should head over to http://datasift.net and register for the DataSift Alpha program.

This post was originally written by me for ProgrammableWeb

Related posts:

1. Who Curates the Real-Time Web?
2. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
3. Why client APIs are an important part of any real-time service

The requirement for a “distributed stream computing platform” came about for Yahoo! in order to be able to process thousands of search queries per second, from potentially millions of users per day,  to facilitate the generation of highly personalized adverts for web search. A new framework was required because Yahoo! felt that MapReduce, which is commonly used to process large datasets in batch jobs, was “hard to apply to stream computational tasks”.

Yahoo! describe the S4 framework using a number of terms that have become common place in the world of cloud computing:

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.

Exactly what Yahoo! S4 is, and what it is capable of, has been discussed in a number of other places. The most commonly used term by comparable frameworks is Complex Event Processing with applications including filtering, correlation and pattern matching. These discussions will no doubt continue but ultimately a framework is something that can be put to multiple uses which is why Yahoo! chose to call it “general-purpose”.

Yahoo! have created a couple of examples to demonstrate some of the basic capabilities and clarify what S4 can do. One of the examples recieves data from the Twitter real-time Garden Hose stream, counts the number of times a hashtag is mentioned and keeps an ordered list of the most commonly mentioned hashtags. Each step of the process is performed in what Yahoo! are calling Processing Elements and it’s these elements that enforce the separation of each logical step of the process (e.g. recieve update, extract hashtags, count hashtags, order hastag count list)  and allow the execution of the process to take place on a distributed system.

One potential thing holding S4 adoption back is that as yet it’s not offered as a service. As well as writing their own Processing Elements developers will have to host their own distributed stream computing platform. If S4 proves to be a useful and popular framework then we may start to see hosted distributed stream computing platform services in the same way that we’ve already seen MapReduce being offered as a service by Amazon.

Yahoo! S4 is yet another powerful real-time component now available to the Programmable Web. It opens up a number of possibilities for developers to start building exciting data-centric applications, mashups or hosted services which could integrate with other components such as real-time APIs, real-time client push services and DaaS services.

Originally posted on Programmable Web

Related posts:

1. Client Push Services Open Up Real-Time to Everyone
2. Browse, Build and Share Real-time Streams with DataSift
3. Who Curates the Real-Time Web?

The conversation continued:

The main thing here for me is Nick’s point that “you just need a client-side library“. This is an assumption that a lot of services make. Services tend to offer one or a number of server APIs; maybe a REST one and now luckily we are seeing a HTTP Streaming and WebSocket support. That service may then release some example code that can be used to call the web service. I see a number of problems with this mainly when it comes to HTTP Streaming and WebSockets.

• DataSift WebSocket Streaming Docs
• DataSift HTTP Streaming Docs

WebSockets aren’t as yet supported in enough browsers. Therefore it’s not viable to purely rely on WebSocket support in any production application. So, you can’t build a web application unless there is a fallback alternative. HTTP Streaming in a web browser could be a fallback (XmlHttpReqeust, XDomainRequest or Flash connections) if the server provides the necessary Access-Control-Allow-Origin response header, the browser is new enough, or if the user has Flash installed the server serves up a crossdomain.xml file and the developer is willing to also write code to deal with all the browser types and technoloy scenarios. Then you need to deal with authentication from a web browser for some services to let you access their HTTP Streaming API. DataSift will class your connection as a 2nd class client if you don’t authenticate:

Authorization is not required on the streaming API. However, should you chose not to authenticate, you may be subject to stricter rate limiting and other disadvantages over an authorized user. Therefore, authorization is strongly recommended

What this means is that DataSift as yet don’t support real-time updates in web browser clients fully.

Then you need to consider other technologies like .NET, Java, C or C++ desktop applications, mobile applications on iPhone, iPad, Windows Mobile 7 or Palm using webOS. Each of these technologies requires a client API.

• DataSift API Client Libraries

A Scenario

Say I were building a new service which focused on real-time data and delivery of that data in our custom application to desktop app, mobile apps and web apps. My service is focused on curating real-time data from different sources. My first port of call would be DataSift or Superfeedr (I’m going to leave Kwwika out of the equation here). I know these guys can deal with large volumes of data and can provide some filtering. Maybe if filtering was absolutely key I would go for DataSift (maybe Julien would tell me off for that).

Now I’ve made my choice and I’ve got all the data I want streaming through my chosen service and I start to build my client applications. If I choose to build my web application first I’m stumped as I’ve already explained, but let’s say I want to build a Windows desktop application first. Simple, I use the HttpWebRequest object to create a HTTP Streaming connection to DataSift, follow the connection guidelines that DataSift follow (Twitter’s HTTP Streaming guidelines including a backing off process and handle reconnection) and I’ve not got my real-time Windows desktop application.

I now decide to build an iPhone application so that people on the move can access my service. I need to start from scratch by writing iOS (Objective-C) code to connect to DataSift’s HTTP streaming API, following the back off process and deal with reconnection. Once I’ve done that I move on to Windows 7 Mobile and I have to do the same again.

You get my point.

Now, once DataSift becomes popular, and it will, there will be loads of open source client APIs to choose from. However, these APIs are likely to have been built by a number of different developers providing different ways of doing the same things. Some might be DataSift specific but hopefully they’ll be generic HTTP Streaming libraries. At this point building my service applications will become a little easier but, in my opinion, still not as easy as it should be.

Using Kwwika

If I were to build this service, and not exclude Kwwika, I would set up all my real-time data streaming through DataSift and write a single library, probably reusing the Kwwika Tweet Streamer code which handles the HTTP Streaming connection and the Twitter connection guidelines, and push the data from DataSift into Kwwika. For each of my client applications I could simply download/use the relevant Kwwika API (JavaScript, .NET, Silverlight, Flash, Objective-C, Java, C) to subscribe to the DataSift data on any web-enabled device. And because these APIs are are developed and maintained by Kwwika they are consistent across platforms e.g. if you’ve used the .NET API you will be familiar with the JavaScript API.

My point is that the client APIs are as important as the server APIs. A quality real-time client API adds a lot of value to the service such as handling reconnection, failover, subscription management, connection monitoring, error handling, bi-directional messaing and request batching (not relevant in this scenario). As well as real-time data distrubtion/messaging/client push this is a key part of what Kwwika offers.

Related posts:

1. Kwwika-Superfeedr real-time demo available
2. Delivery as a Service and Data as a Service in 2011
3. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub

Real-time client push APIs have actually been around for quite a while (around 10 years) as they are shipped with Comet servers but only recently have these been moved into the cloud and offered as a service. The service flavour of these APIs give the developer the ability to instantly push information from their web server, through their chosen push service and into a web browser viewing their website.

Real-time client push is intended to replace the previous pull, or polling, mechanism that has been used for many years to mimic live data on a website. Using push via a dedicated Comet server is generally more resource efficient than polling a web server, and by using a service the resource load and complexity involved in setting up and running a Comet service is completely taken away from the developer’s considerations. This means that the web server is under much less strain, the developer can concentrate on building a killer real-time application and the website user gets the benefit of a truly real-time experience.

These services have only recently started to pop up due to a number of technology advancements. To be able to use a real-time client push service the web browser needs to be able to maintain a persistent connection back to the web server so that the web server can push information to it as soon as it becomes available. This has been achievable for a number of years via what some developer have labelled as “hacks” but is now easier than it has ever been. Most of the new services use the JavaScript WebSocket object to achieve this and fallback to using Flash if WebSockets are not supported by the browser.

The web browser also needs to be able to maintain a cross domain connection from the JavaScript code running in the website to the service e.g. a connection from blog.programmableweb.com to www.example.com. In older browsers cross domain connections were not allowed but the introduction of client access policy files (crossdomain.xml and clientaccesspolicy.xml) and more recently the Access-Control-Allow-Origin HTTP header have made cross domain calls from JavaScript possible (You can find more information and a demo of this in action here).

All of the real-time client push services have adopted a data publisher subscriber model with the web server code generally acting as the data publisher and the JavaScript code running in the web browser acting as the data subscriber. Subscriptions are made to a channel (or topic), that either exists or will be created, within the service and are identified by a name e.g. “my_channel” or “/PW/CHAT”. The publishers then simply publish data to that channel or topic using a service API and the information is instantly received by all subscribers, again via an API.

The real-time client push services that we know of at the moment are:

• Beacon
• Hookbox
• Kwwika (disclosure: author is a founder)
• PubNub
• Pusher
• WebSync

And some examples of their use include:

• A real-time news reader
• A Facebook “meh” button
• Interactive games
• Real-time sports statistics (requires HTML5 support)

You can also check out the demos on each of the services websites.

Real-time is already a big topic and users are starting to demand data and results as fast as possible. There is also the expectation that they should be informed as soon as new data is available or the existing data changes. Google are already pressing ahead with new real-time advancements such as Google Real-Time search (which actually uses polling) and Google Instant but the good news is that with the availability of real-time client push services any developer can now add real-time to their website.

Let us know if you are interested in finding out more about these real-time client push APIs and services and we’ll cover each one in more detail.

Photo via Blake Patterson

Originally posted on Programmable Web

Related posts:

1. Yahoo’s Open Sourced S4 Could be a Real-time Cloud Platform
2. Why client APIs are an important part of any real-time service
3. Hosted realtime services – making the realtime web more accessible