Realtime Web Technology Accessibility

One of the problems with real-time technologies over the past 10+ years has been the accessibility. Comet servers were difficult to install, a pain to maintain and upgrade and generally difficult to use due to numerous configuration options and technology quirks, mainly exposed by different web browsers. There have been a number of very important changes have happened over the past few years that now make real-time technologies much more accessible.

Web Browser Wars

For such a long time Internet Explorer 6 was by far the best web browser available. It’s the reason that Microsoft “dropped the ball” and why Firefox and Chrome are where they are today. But now we are in a great position of having a number of actively developed web browsers being driven by real competition to offer the best browsing experience. This has lead to a general improvements in browser speed (JavaScript and rendering engines) but also improvements in web technology available to web browsers and, possibly most important of all, web standardisation. HTML5 does indeed rock!

Cloud Hosted Services

Cloud hosted services, frequently referred to as Infrastructure as a Service (IaaS). These hosted solutions take away the pain of installation, maintenance and scaling, and frequently deliver them at a fraction of the cost possible if you were self-hosting.

Better self hosted solutions

Cloud hosted services are awesome (I would say that, but they are!) but the self hosted realtime web solutions that are available and the ease of installation has massively improved. As an example, you can have a socket.io example page up and running on node.js within minutes. These improvements have been driven by an increased demand for

WebSockets

WebSockets is a standard for continuous bi-directional single-connection communication between client and server. Real-time technologies, and in particular real-time messaging within a web browser, has been possible before but not until WebSockets has there been a standard that is so likely to succeed.

Unfortunately WebSockets don’t mean that the pain has completely gone away. The technology, whilst more accessible, still isn’t as easy as it could be for developers to use. On some platforms WebSocket server installation can still take too long, maintenance is just something developers don’t really want to do, and scaling takes time, effort and lots of configuration. So, we turn to the first important change as a solution to this problem – hosted realtime services.

Hosted realtime services

Hosted realtime services, like Pusher who I work for, make it much faster and easier for developers to start adding real-time functionality (push notifications, activity streams, realtime chat, collaboration, realtime data visualisations, multiplayer games, 2nd screen/Social TV experiences etc.) to their applications within minutes. For bigger organisations, the same solution also means that you know your developers are busy adding value, not installing, configuring and maintaining. This means they are happy and productive.

The web is only around 20 years old and the real-time web is younger still. We don’t know exactly where the next few years will take the web but I’m sure that HTML5, WebSockets and real-time messaging will be a key component, and services like Pusher in a great position to help provide the infrastructure that developers need to easily use these technologies.

Where next?

• If you found this interesting you might find my Realtime Web Technology Guide/List of interest. If you know of any technologies that I’ve missed, please get in touch.
• Why not take a look at Pusher and the Pusher Blog

Related posts:

1. Google realtime search isn’t real-time
2. Client Push Services Open Up Real-Time to Everyone
3. Making cross domain JavaScript requests using XMLHttpRequest or XDomainRequest

In this tutorial I’ll show how to:

• subscribe to a presence channel
• authenticate a subscription to a channel
• provide Pusher with additional information about a user
• display presence information on a product page for the “Who’s shopping?” widget

If you are desperate to see the demo in action you can see the Real-Time Web Store demo here.

Pusher Presence

To achieve the “Who’s shopping?” functionality I’m going to be using a feature in Pusher called presence. Presence provides you with additional information about a channel you are subscribed to so that you know:

• who is subscribed to that channel
• when new users subscribe
• when existing users unsubscribe (by either actually unsubscribing or navigating away from the page).

We are going to have a presence channel per product so that we know who is viewing each product.

User Info & Authentication

Subscribing to a presence channel

You subscribe to a presence channel in the same way that you do to any other channel but the name of the channel must have a presence- prefix. Presence channels are normal channels with two additions; authentication and presence information. With this in mind we are just going to update our application to use a presence channel.

The JavaScript that makes the subscription in our Razor view looks like this:

var productId = "@Model.ProductId";
var pusher = new Pusher("APP_KEY");
var channel = pusher.subscribe("presence-" + productId);

We also need to update the code in our StoreController to publish our stock events on the new presence channel:

var stockEvent = new StockUpdatedEvent(model, socketId);
ObjectPusherRequest request = new ObjectPusherRequest("presence-" + stockEvent.ProductId, "stockUpdated", stockEvent);
_provider.Trigger(request);

Note: If you are continuing where we left off in our last blog post there are a final couple of updates that are required to change the app to use the latest version of the Pusher JavaScript API. We recently released version 1.9 which introduced new connection state functionality and also a new connection object. So, update your Pusher script tag as follows:

<script src="http://js.pusherapp.com/1.9/pusher.js"></script>

And you’ll also need to update any pieces of code that access the socket_id via the Pusher instance. It should now be accessed via the new connection object as follows:

var socketId = pusher.connection.socket_id;

Getting User information

If Pusher is to send events about users subscribing to and unsubscribing from presence channels it needs information about the users. It gets this information from your application when the subscription request to the channel is made (pusher.subscribe('presence-channel')). Since we can’t really trust the web browser/client (it’s so easy to hack JavaScript running in a web browser) the Pusher library requests this information from your web server by making an AJAX call. By default this call goes to /pusher/auth and passes two parameters; channel_name, which is the name of the channel being subscribed to, and socket_id, which is a unique identifier for the current user’s connection to Pusher.

/pusher/auth/?channel_name=presence-pusher-tshirt&socket_id=<unique_socket_id>

When our application responds to this request we must provide an authentication signature to confirm that the user can subscribe to the channel and, importantly for our “Who’s shopping?” widget, information about the current user. The way we’ll handle this within our ASP.NET MVC application is by creating a PusherController with an Auth(string socket_id, string channel_name) action, and by using the authentication functionality within the PusherRESTDotNet library. This library is also available as a NuGet package.

Note: If you got the NuGet package as part of the last tutorial you’ll need to update it since the authentication functionality has just been added. You should also check that the .NET 3.5 runtime version of Newtonsoft.Json is added.

Handling the authentication request

As mentioned above, the Pusher JavaScript library will make a request to /pusher/auth when making the authentication request. Our new PusherController with Auth action does the following:

1. Fetches our Pusher credentials from the Web.config file.
2. Creates a new PusherProvider using the Pusher credentials
3. Creates a unique user_id for the presence channel
4. Creates an authentication string and returns that string as the Content of a ContentResult' with theContentTypeset toapplication/json` in response to the AJAX request.

For the moment this code doesn’t do any user authentication or provide any additional information about the current user.

using System;
using System.Configuration;
using System.Web.Mvc;
using PusherRESTDotNet;
using PusherRESTDotNet.Authentication;

namespace RealTimeWebStore.Controllers
{
    public class PusherController : Controller
    {
        public ActionResult Auth(string channel_name, string socket_id)
        {
            var applicationId = ConfigurationManager.AppSettings["application_id"];
            var applicationKey = ConfigurationManager.AppSettings["application_key"];
            var applicationSecret = ConfigurationManager.AppSettings["application_secret"];

            var channelData = new PresenceChannelData()
            {
                user_id = Guid.NewGuid().ToString()
            };

            var provider = new PusherProvider(applicationId, applicationKey, applicationSecret);
            string authJson = provider.Authenticate(channel_name, socket_id, channelData);

            return new ContentResult { Content = authJson, ContentType = "application/json" };
        }
    }
}

If we use one of the many web browser development tools available to us to inspect the authentication call within the browser we’ll see the JSON response coming back.

You’ll see the response contains a channel_data property which itself has a user_id with a unique Guid value and a user_info property with a null value. Pusher uses this user_id value to uniquely identify the user subscription to the presence channel. So it’s very important to make sure that each user has a unique ID.

Adding authentication

We’ve mentioned authentication a few times but as yet we haven’t authenticated the user. If the user has already logged (our app doesn’t have this functionality, but most do) in we can use the existing User.Identity or else we can just assign a guest identity to the user. Once we have a unique ID for the user we’ll also add some additional user_info to the channelData. The value of user_info can be anything you like from a simple string to a complex object. This gives you the ability to push as much additional information through Pusher and to the web page as you like. In our case we’ll just send through a timestamp which identifies how long the user has been on the site.

public ActionResult Auth(string channel_name, string socket_id)
{
    var channelData = new PresenceChannelData();
    if (User.Identity.IsAuthenticated)
    {
        channelData.user_id = User.Identity.Name;
    }
    else
    {
        channelData.user_id = GetUniqueUserId();
    }
    channelData.user_info = GetUserInfo();

    var provider = new PusherProvider(applicationId, applicationKey, applicationSecret);
    string authJson = provider.Authenticate(channel_name, socket_id, channelData);

    return new ContentResult { Content = authJson, ContentType = "application/json" };
}

Note: In our case we don’t really need to authorise a user but in other situations where the user needs to be logged in we can return a 401 HttpStatusCodeResult.

Who’s Shopping?

Now that we’ve got a PusherController that gives Pusher information about the user, we can start showing information about the user on the product page. You can get information about the users subscribed to presence channels by binding to the pusher:subscription_succeeded event on the presence channel object. The callback method for this event receives a members parameter which contains all the information about users subscribed to the channel.

First we’ll create some HTML within our web page where we are going to show “Who’s shopping?”. Then we’ll add the users to the HTML when pusher notifies us of them.

HTML

<div class="whos-shopping">
    <h3>Who's shopping?</h3>
    <ul></ul>
</div>

JavaScript

var pusher = new Pusher("006c79b1fe1700c6c10d");
var channel = pusher.subscribe("presence-" + productId);
channel.bind("pusher:subscription_succeeded", function(members) {

    members.each(function(member) {
        addMember(member);
    });

});

function addMember(member) {
    var enteredSite = new Date(member.info.timestamp);
    var now = new Date();
    var timeOnSite = (now - enteredSite);
    var li = $("<li data-user-id='" + member.id + "'>" +
                    member.id + " here for " +
                    toReadableTime(timeOnSite) +
               "</li>");
    $(".whos-shopping ul").append(li);
};

Note: The members object comes with a handy each method to make iterating the members collection really easy.

Of course new users can navigate to the page and existing users can leave it so the Pusher JavaScript library also exposes pusher:member_added and pusher:member_removed events on the presence channel object. When these events fire we should add or remove the user as required.

channel.bind("pusher:member_added", function(member) {
    addMember(member);
});
channel.bind("pusher:member_removed", function(member) {
    removeMember(member);
});

function addMember(member) {
    /* as before */
};

function removeMember(member) {
    $(".whos-shopping ul li[data-user-id='" + member.id + "']").remove();
};

With this in place we now have a fully functioning “Who’s shopping?” widget that shows the current user who else is viewing the same product as they are.

As mentioned in the opening paragraph, the theory here is that if shoppers can see that others users are viewing the same product it might give them that little push they need to take the plunge and make that purchase “while stocks last”.

There are a few refinements and enhancements that could be made to this widget such as filtering out the current user from the “Who’s shopping?” list or possibly showing them which one they are. You could also use the notification system from last time to notify the shopper when another shopper joins or leaves the product page. And, of course, you could add some user chat functionality to get the users discussing the product and really engaging. You could also have a staff member user who could answer any questions that the shoppers may have.

Just as last time all the code from this post is available in the real-time web store github repo. You can also see the Real-Time Web Store application up and running on AppHarbor. I’ve tried to link to relevant parts of the Pusher documentation throughout the post but if there anything that isn’t clear, if there’s anything that I’ve not provided enough detail on and it all just seems too ‘magical’, then please leave a comment or send an email to me (phil@pusher.com).

Here are some links to the key things covered in this post:

• Presence channels
• Presence channel events
• Authenticating Users
• Pusher REST .NET Library | NuGet package
• ASP.NET real-time web store in github
• Real-Time Web Store application up and running
• ASP.NET MVC 3

Addendum: What about WebForms?

The post above shows how to user the Pusher REST .NET library within an ASP.NET MVC application but it can just as easily be used within an ASP.NET WebForms app. The way I achieved this was by adding a new Generic HTTP Handler to our web app which will handle the authentication AJAX call.

In the code below the ProcessRequest method does the following things:

1. Fetches our Pusher credentials from the Web.config file.
2. Gets the values of the channel_name and socket_id parameters from the context.Request
3. Creates a new PusherProvider using the Pusher credentials
4. Creates a unique user_id for the presence channel
5. Creates an authentication string and returns that string as the response body of the AJAX request.

For the moment this code doesn’t do any user authentication or provide any additional information about the current user.

using System.Configuration;
using System.Web;
using PusherRESTDotNet;
using PusherRESTDotNet.Authentication;
using System;

namespace RealTimeWebStore
{
    public class AuthHandler : IHttpHandler
    {
        public void ProcessRequest(HttpContext context)
        {
            var applicationId = ConfigurationManager.AppSettings["pusher-application-id"];
            var applicationKey = ConfigurationManager.AppSettings["pusher-application-key"];
            var applicationSecret = ConfigurationManager.AppSettings["pusher-application-secret"];

            var socketID = context.Request["socket_id"].ToString();
            var channelName = context.Request["channel_name"].ToString();
            var channelData = new PresenceChannelData()
            {
                user_id = Guid.NewGuid().ToString()
            };

            var provider = new PusherProvider(applicationId, applicationKey, applicationSecret);
            string authJson = provider.Authenticate(channelName, socketId,  channelData);

            context.Response.Write(authJson);
        }

        public bool IsReusable
        {
            get
            {
                return false;
            }
        }
    }
}

Finally we need to configure our handler in the application Web.config file. We want the ProcessRequest method of our handler to be invoked for any call to /pusher/auth. To do this we just add a handler to the httpHandlers element and specify our handler, RealTimeWebStore.AuthHandler as the handler:

<system.web>
  <!-- other config -->
  <httpHandlers>
    <add verb="*"
         path="/pusher/auth/"
         type="RealTimeWebStore.AuthHandler" />
  </httpHandlers>
</system.web>

Related posts:

1. The easiest way to add real-time functionality to an ASP.NET e-commerce application
2. Kwwika Powered Real-Time Opta Sports Cricket Widget
3. Recent article in .net magazine: WebSockets – Code a real-time survey

One of the problems with existing e-commerce solutions can be trying to buy something which, whilst you’ve been online, has sold out. This can be a big problem for high demand products like gig tickets or sporting events such as the Olympics. A really good way of avoiding this is to show the customers just how fast tickets are selling. The added benefit of this, from a business point of view, is that it can give users that little push they needed to make that purchase. In this tutorial I’ll show how to:

• establish a connection to Pusher in the view
• subscribe to a product channel
• bind events from this channel to our stock indicator in the view and update the stock level
• send stock level update from our server when someone buys something

Getting started

To demonstrate adding real-time functionality to an ASP.NET website I’m going to start with a simple one page e-commerce site which has one product: our cool Pusher t-shirt (Sorry, they’re not really for sale. You’ve got to earn them!). On the product page there is a counter which tells the customer how many t-shirts we have in stock.

To start off with our e-commerce website is a simple ASP.NET MVC3 app with:

• A StoreController with two actions. One to display the default view and one to handle the “Buy” button being clicked and the stock level being decreased.
• A single Razor view displaying the single product
• Two models; one for the product called ProductModel and one for images called ProductImage
• A ProductRepository where we store our products

Connect to Pusher

As I’ve already said, adding real-time functionality is really simple – that’s the point! Firstly we have to include the Pusher JavaScript client library. We’ll add this to the view for the moment although we might want to make the include part of the site layout in the future.

Note: If your e-commerce page was being served up over HTTPS you would include our library over HTTPS too to avoid any browser warnings.

Since we are using ASP.NET MVC we’ve already got jQuery included from the _Layout.cs.html so we can wait until the page has loaded and connect to Pusher. The script include and the code that connects to Pusher looks as follows:

<script src="http://js.pusherapp.com/1.8/pusher.min.js"></script>
<script>
    $(function() {
        var pusher = new Pusher("006c79b1fe1700c6c10d");
    });
</script>

Subscribe to the product channel

Now that we’ve connected to Pusher we can push events from our StoreController to all connected browsers if any information changes about the product. The main piece of information that customers might want to know about is if the stock level changes. These t-shirts could sell out fast!

We need to choose a channel name to subscribe and publish to and the obvious choice is to use the unique product id for this. We’ll also prefix it with product- just so it’s really clear that the channels is for a product. We subscribe to this channel on the client as follows getting the product id from the Model that has been bound to the page:

var channel = pusher.subscribe("product-@Model.ProductId");

Bind to stock update events

Once we’ve got a channel object we can bind to events on it. We want to be informed when the stock level changes so let’s bind to an event called stockUpdated. We’ll also expect a JSON object to be passed which represents the ProductModel and will have the same properties on it. That way we can simply update the stock level in the page to match the value on the server by accessing the ProductModel.StockLevel value. We can also update the stock status:

channel.bind("stockUpdated", function(product) {
    $(".product .stock .level").html(product.StockLevel);
    $(".product .stock .status").html(product.StockStatus);
});

Trigger stock update events

As you can see, this is really easy to do. It’s just as easy to trigger the event on the server too. To do this we’re going to be using the C# REST API library within our StoreController. You can get the PusherRESTDotNet library from github or via NugGet.

Once we’ve added our PusherRESTDotNet reference we can add a few lines of code to our StoreController to instantly push any changes in stock level to all connected web browsers viewing our t-shirt page.

The first thing to do is to create a PusherProvider and pass in our Pusher details. In this example we are storing these details in the ASP.NET MVC3 Web.config.

Web.config

<configuration>
  <appSettings>
    <add key="application_id" value="APP_ID" />
    <add key="application_key" value="APP_KEY" />
    <add key="application_secret" value="APP_SECRET" />

    <!-- more settings -->
  </appSettings>

  <!-- more config -->
</configuration>

StoreController.cs

public class StoreController : Controller
{
  private IPusherProvider _provider;

  public StoreController()
  {
      string applicationKey = ConfigurationManager.AppSettings["application_key"];
      string applicaitonSecret = ConfigurationManager.AppSettings["application_secret"];
      string applicationId = ConfigurationManager.AppSettings["application_id"];
      _provider = new PusherProvider(applicationId, applicationKey, applicaitonSecret);
  }

  // more code here...
}

Once we’ve created our instance of the PusherProvider we can create a PusherObjectRequest to push the ProductModel, with an updated StockLevel following the purchase, to the clients. The serialisation of the model object is handled for us by the PusherRESTDotNet library. The full StoreController.Index HTTP POST handling action looks like this:

[HttpPost]
public ActionResult Index()
{
  bool bought = MvcApplication.ProductRepository.Buy(MvcApplication.BLUE_TSHIRT_ID);
  var model = MvcApplication.ProductRepository.GetProductById(MvcApplication.BLUE_TSHIRT_ID);

  if (bought)
  {
      ViewBag.Info = model.Title + " successfully bought";

      ObjectPusherRequest request = new ObjectPusherRequest("product-" + model.ProductId, "stockUpdated", model);
      _provider.Trigger(request);
  }
  else
  {
      ViewBag.Error = "There was a problem buying " + model.Title;
  }

  return View("Index", model);
}

Now, if you run your application and open two or more windows, as soon as a user in one window clicks the “Buy” button you’ll see that the stock level value instantly updates in all other windows.

Next time – displaying real-time “Who’s shopping” info with Pusher presence

We now have the stock level indicator changing in real-time. This tells the users that these t-shirts are selling out fast and gives them an added incentive to make that impulse buy. This is a simple yet effective change but there’s more we can do. What if we also show the user how many other users are viewing that product at the same time so that they know there is competition for the last few t-shirts? We can do that easily using Pusher’s presence functionality and I’ll cover that in my next Real-Time ASP.NET blog post which will also go into how to add Pusher user authentication to your ASP.NET web application.

In the meantime you can download the source of the Real-Time Web Store from github. The solution contains a project with the basic web store without any real-time Pusher functionality as well as a solution project. Here are some additional things you you might also want to try out:

• Changing the form submission when the user clicks “Buy” so that the POST to the Buy action it is made using a jQuery ajax call
• Making the change in stock levels more visually interesting. Try adding an effect to the number change or providing a growl-like notification
• Changing the CSS – it’s not my strong point

Related posts:

1. Adding a real-time "Who’s shopping?" widget to an ASP.NET Web App
2. Recent article in .net magazine: WebSockets – Code a real-time survey
3. Which Rich Internet Application Technology will dominate?

A very important question, that I recently heard from Roan Laverly of FreeAgent: “Is your product solving a hard enough problem?” Well, what problems was Collecta solving?

The first problem Collecta was solving was search, and the problem with building a search product in general is that the market is massively dominated by Google. Even Microsoft’s Bing and Yahoo! can’t really make a significant dent in the search market, so how can anybody compete? Collecta tried to compete by using real-time search results as their unique selling point – Google now has a real-time search option, and if a topic is trending real-time search results can form part of the general search results too.

Another problem that Collecta was solving was making real-time data accessible. So many companies that have real-time data are now opening up their own APIs and increasing the ease at which anybody can access that data. This means individuals and companies can now got direct to source and build their own search offering, or just extract just the data they need themselves. I’m quite sure that Collecta has some very clever technology that delivered their best guess (algorithmically computed) search results in real-time. But again, it doesn’t sound like that was enough.

Creating real-time technology components capable of consuming, processing and distributing large amounts of data was another problem that Collecta had to answer. Initially building the technology capable of doing this was difficult but now, a few years later, real-time technology components are much more widely accessible. So whilst the difficulty level is still high the problem isn’t as difficult as it once was. This in itself has meant that data companies can build their own APIs.

Then there’s the problem of creating a compelling application and products. A website showing real-time search results in a intuitive and informative way. Widgets showing search results in an easy to digest and embeddable format. The post on ReadWriteWeb provides a comment by one of their own writers, Jared Smith, on Collecta’s applications:

Collecta’s emphasis on a search experience that went beyond Twitter into photos and videos made it a great tool to truly watch a story unfold in real time. Their embeddable widget, which I used regularly on ReadWriteWeb’s event sites, was far more powerful than what Twitter provided and is still unmatched in my mind.

So, even though they were building a useful products it wasn’t enough. Did Jared pay for the widget, and if not would he have been willing to? Would enough users have been willing to pay?

The good news out of all of this is that it sounds like the Collecta technology is going to be open sourced. A quote on the ReadWriteWeb post states:

Collecta now says it will open source its software and is working with a variety of organizations to do so – including United Nations crisis relief projects.

Is the real-time search problem hard enough? We should keep our eyes on Topsy and Social Mention to see how they get on. What we can be sure of though is that with Collecta open sourcing their technology the real-time search technology question will now be easier than ever to answer.

Originally written by me and posted on Programmable Web

Related posts:

1. Problem solving lessons relearnt
2. How I approach problem solving in code?
3. Is It Finally the End for Real-time Search Engines?

It is technology that has made it possible for us to produce, share and consume so much information. We are living in a time that is often referred to as an era of digital overload. It’s therefore very interesting and essential that the cause of this data overload should also be the solution: Technology, in combination with human input, can help us solve these new problems through automated curation of data.

SXSW ran a panel covering this subject: “Humans Versus Robots: Who Curates the Real-Time Web?“. The panel consisted of representatives from curation services including Henry Nothhaft from Trapit, Jim England from Keepstream, Sam Decker from Mass Relevance, Xavier Damman from Storify, and Megan McCarthy from Media Gazer who chaired the discussion. Unsurprisingly there wasn’t a definitive answer to this question. The answer really depends on the data that is being curated. It was agreed that automated curation (robots) was required in cases where there was so much data that it would not be possible for humans to possibly curate. However, the consensus was that generally once the amount of data had been reduced to a more acceptable level that it was humans who could provide the best form of curation since they can understand and apply more context and objectivity to any data. Technology will of course improve and we’ll see the robots getting smarter and smarter.

Curation cannot be solved by any one technology and it will require smooth interaction and integration between numerous technologies in order to solve this problem. APIs will play a massive part in the curation process and we’ve already seen this directly through the Cadmus API and within services such as DataSift, where they use third party services such as Klout, Peer Index and Salience for influence and sentiment analysis. DataSift provide a delivery and curation service but they can’t do everything on their own which is why they have outsourced some parts of the process to other services. As more curation techniques and algorithms are identified we’ll see these techniques exposed through additional services and APIs and in turn these APIs will be integrated into other curation services.

The amount of information being published to the real-time web is going to increase and as it does so will the importance of curation. It might not be long until it will be essential in ensuring that we don’t all suffer digital overload. How long will it be before Twitter.com and the ever increasing list of official Twitter clients, who presently only offer manual curation through selective follows and lists, join in and add curation to their service? Will they build it themselves, use a number of the available APIs (unlikely based on the amount of data) or make another acquisition?

Photo via Blake Patterson

Originally written by me for Programmable Web

Related posts:

1. Browse, Build and Share Real-time Streams with DataSift
2. Why client APIs are an important part of any real-time service
3. The benefits of Real-Time Push-Once

Essential

• C#
• jQuery
• TDD/BDD
• ASP.NET MVC
• ADO.NET Entity Framework or other ORM

Great to have

• Passionate about the real-time web and real-time data (essential for the longer term)
• Knowledge of building scalable systems on Amazon Infrastructure
• Experience of integrating with 3rd party payment systems and building basic ecommerce functionality
• Experience of building a SaaS or IaaS offering
• Interest/experience in API and Web Service development
• Advanced JavaScript
• CSS
• Excited about HTML5
• Part of a tech community

Location

The role will be primarily home-based but at times we may need to meet around 2 to 3 times a week at a location that we agree on anywhere between or around Dundee and Edinburgh.

Moving forward I’m hopeful that this could develop into a fulltime role so it would be ideal if we could get somebody who was interested in this also.

If you are interested please get in touch with your availability and your rate.

Related posts:

1. Drag Drop bug in JQuery UI 1.5.3
2. Real-Time Rich Internet Applications (RTRIA)
3. The Real-Time Web: TechMeetup Aberdeen – 19/01/2011

A big thanks to @aboynejames for suggesting I do the talk and Bruce Scharlau for inviting me along. I’d also like to thank everybody who attended and made me feel very welcome and certainly seemed interested.

Feedback

If you where there then please leave a comment letting me know what you thought or just to let me know who you are. If you’d rather email you can get me using phil@leggetter.co.uk.

Video

Here’s a video of the first 10 minutes of the talk. Unfortunately the video recorder ran out of batteries.

Presentation and Notes

Unfortunately I can’t find a way of showing my slide notes in the above embeded Google Presentation so here they are in pretty rough form.

What is Real-Time?

Respond or react within a predictable time after an event has occurred.

There may also be a maximum time constraint.

Because after that maximum time the information loses value and context.

Maybe we should really call it:
Right-Time

When does Real-Time matter?

When does NOW matter?

When do you think being informed of something the instant it occurs really matters?

• Critical systems.
• Live events where things lose context.
• Systems requiring synchronisation:
  The 2Screen experience e.g. TellyLinks.com
• Interactive/collaborative things e.g. chat, Google Wave, Google Docs (docs, spreadsheets, drawing)
• Sensor data e.g. Car ABS breaks
• Trading: You’ve got to be sure you are seeing the correct trade price in order to be able to make a decision on whether to trade or not.
• Betting: You want to make sure you bet at the right odds and you don’t want clients to be able to bed when the odds have changed.

Evolution of the Web

From Pull to Push

Although I’m going to use the Web Browser as the example client the client could of course be anything. Another web server any desktop application, a mobile application, an embedded app on any device.

I’m also going to focus on HTTP but the mechanisms could be applied to standard TCP socket connections.

HTTP Pull (Request, Response)

Note: client. This can be a web browser but doesn’t have to be.

1) Each request would result in a page reload.
2) Event occurs on server there’s no way of the server telling the browser.
3) e.g. chat pages where you would need to click the “Refresh” button to see what other users are saying

HTTP Polling

A user hitting F5 or clicking the refresh button is one manual example.

1) REFRESH-META
2) Then frames
3) XMLHttpRequest -> Outlook web access (ActiveX object)

Check if the information has changed by polling.

The problem with this is you can easily waste requests and it can be resource intensive for both the client and the server.

A large amount of services still take this approach although the web, and the technologies, have evolved to allow for real-time push.

For quite some time I’ve said if a service uses HTTP Polling when push technology is available and a much better option that it’s not real-time

HTTP Push

In this diagram you could replace the Client with Server. The initialization could be classed as a subscription or webhook.

Various techniques have been developed to hold a connection open to a web server from a browser.

Difficulties :

* Proxies/Firewall
* Connection reselience
* Browsers
* Messaging API

Related posts:

1. Delivery as a Service and Data as a Service in 2011
2. XMPP PubSub or PubSubHubbub for real-time server push?
3. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub

DataSift launched its Alpha service at TechCrunch Disrupt in September and describe it as a “real time social media filtering engine.” The initial buzz around DataSift was generated when Twitter agreed to give them access to the Twitter firehose, but they now have access to a much wider range of data including the Google Buzz, MySpace, SixApart, WordPress, Facebook and Digg. These sources of data within DataSift, sometimes called input services, are defined as Targets in the DataSift knowledge base.

Users can use Targets to create their own streams from within the My Streams section of the DataSift dashboard using a language called FSDL (Filtered Stream Definition Language). The Web editor used to define your streams is pretty simple but it does provide some basic syntax highlighting as well as validating your syntax whenever you save. FSDL also provides access to augmentation Targets through services such as Salience, TweetMeme, Peer Index, Klout and InfoChimps that allow streams to be augmented with third party data.

Once a stream has been defined you can choose to build your feed. At present this takes up to 60 minutes and the dashboard will show you the progress of the build as well as a host of other features including a data preview, a live example of the data, a graph showing matched stream items (iterations per minute) and the history of the feed definition.

Update: Nick Halstead (CEO of DataSift) has provided a clarification about the stream preview, the live stream feature and also when the stream can be used via the API:

The stream ‘preview’ does not need to be built for anything to work, you can define + attach to API immediately (or just hit ‘live’ tab to see live results) – the preview was to allow stream owners to demonstrate what the stream would offer.

You can see Nick’s full comment in the comments section below.

DataSift is encouraging its users to build feeds that are discoverable and accessible to other users, although it does offer a private feed option, by providing a number of options on the stream page including tagging, an area to encourage you to tell others about it on common social networks and a comments area to encourage users to interact and visits to the stream page. Recently added, most commented and top rated streams are also featured on the home section of the DataSift dashboard.

Once the stream has been built it can also be used in the definition of another user stream, and it in another stream and so on. DataSift really have exposed a lot of functionality and capabilities within their user dashboard and the documentation that they provide is quite thorough and really helps a user get to grips with creating streams reasonably quickly.

All these rich features would be a waste unless there was a way of accessing the data and using it with an application. Unsurprisingly, the DataSift API delivers by providing three endpoints, including access to filtered data in a paged manner, HTTP Streaming and an RSS endpoint. It also recently introduced a WebSockets Streaming API.

Although DataSift is still in Alpha it is offering what seems an affordable and very impressive service which should excite any developer interested in real-time technologies and data. The company’s access to a wide range of data sources, engaging and intuitive user dashboard and range of API endpoints should mean that most developers will have their technology needs met.

A good starting point to learn about DataSift is an interview by Robert Scoble with Nick Halstead, the CEO of DataSift (embedded above). The video is a little old but provides a good overview and an example of creating a stream. If you’ve any comments or questions about DataSift please leave a comment here. After that you should head over to http://datasift.net and register for the DataSift Alpha program.

This post was originally written by me for ProgrammableWeb

Related posts:

1. Who Curates the Real-Time Web?
2. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
3. Why client APIs are an important part of any real-time service

Kate – another very insightful post. And another set of predictions that I completely agree with you about and really excite me because I’m working on technology that makes this possible and more easily accessible. For multi-user games and collaborative applications the information about a player move, new document update or chat notification generally needs to be instantly sent to all other users. Even where real-time is not absolutely essential to the application a push notification is still more beneficial (UX and resource usage) than notification via polling.

During the last year we started to see, or at least notice, a lot more real-time collaboration applications and games which used the Internet as the network for the messages. In this post I wrote for Programmable Web on real-time client push services the first screenshot is of a web-based scrabble-like game.

Some other well know examples are Google Docs documents, drawing and spreadsheets real-time collaboration, Google Wave, PBWorks and real-time chat services such as Nurph (although they currently use polling). I also think that services such as Balsamiq, a UI mocking app, plan to introduce real-time collaboration functionality.

The user experience side of things is also very interesting. When real-time data delivery becomes a possibility the application developer really needs to consider how this impacts the UX. Just because you can stream 1000′s of tweets a minute into an application it doesn’t mean you should. search.twitter.com is a good example of this; they’ve chose to just show a notification that new tweets are available rather than actually push them into view. These two examples also show notifications being pushed into view:

• Twitter Fall
• Simple real-time streaming of Tweets demo on Kwwika (click on NOWPLAYING)

I’d love to know what people think about this and hear any idea about how we can solve this real-time notification UX problem.

During 2011 I absolutely believe that we will start to see a lot more real-time and cross platform collaborative applications and games where one user may be on a smartphone, another on a PC running in a web browser and another on a native iPad app. These applications will be powered by real-time technologies either hosted by the game or application developers or using one of the real-time push notification and messaging services that started up last year.

My hope is that people will choose to use a service for that infrastructure and that Kwwika is well placed to fill that role by having a foundation of technology developed over 10 years and APIs for the majority of technologies (JavaScript [Web], Objective-C [iPhone/iPad], C, Java, .NET, Flash, Silverlight).

Related posts:

1. Comment on Real-Time Delivery Explained post by feedmyinbox
2. Real-Time Rich Internet Applications (RTRIA)
3. Using Fiddler to help develop cross domain capable JavaScript web applications

As 2011 continues we might to see some of these companies either partner up or start to tread on each others toes as they look at delivering data to alternative platforms and clients. Ultimately we are going to see services offering real-time communication of any type to any web enabled device using the web as the network.

The following lists aren’t 100% comprehensive and as time goes by I’ll probably create some pages on my site dedicated to this information.

Here’s the list of services I’m aware of and the type of service they offer and technology they use.

• BeaconPush - [Web browser]
• Google Channel API (Python/Java) – Part of Google App Engine – [Web browser]
• DataSift – [Server to server, WebSockets, HTTP Streaming]
• Kwwika – [Web browser, mobile, server to server, Desktop]
• Hookbox – [Web browser]
• Notifo – [Mobile]
• Notify.io – [Desktop, IM, Email]
• Notify.me – [Mobile, Desktop, IM, Email, XMPP]
• PubNub – [Web browser, mobile, server to server, Desktop]
• Pusher – [Web browser]
• Superfeedr – [Server to server, PubSubHubbub, WebSockets, HTTP Streaming]
• Twilio – [Mobile]
• Urban Airship – [Moblie]

A quick not of the key technologies behind these types of service:

• Comet (Note: this version by Jacobolus (Jacob Rus) is by far the best description. The current wikipedia description, IMHO, is inaccurate and incomplete and was ruined by Damiens.rf. Damiens.rf even put Greg Wilkins, who contributes to Jetty and Cometd, off contributing to Wikipedia all together.)
• Growl
• HTTP Streaming
• PubSubHubbub
• XMPP
• Webhooks
• WebSockets

Note: I originally wrote this as a comment over on Anil Dash’s post on Delivery as a Service but I thought it was good content so it deserved a post of its own.

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. The Real-Time Web: TechMeetup Aberdeen – 19/01/2011
3. Why client APIs are an important part of any real-time service