There’s so much data being generated by social networks that it wasn’t surprising that a lot of the focus was around analysing the data. The key words from the event in relation to the data were relevance, sentiment, curation, authority and location. All of these terms play off each other. Relevance can be determined by authority and location. Sentiment, relevance and authority can be determined by a curation process. Some solutions to curation pitched at the event were based on language and text analysis, whilst others suggested crowdsourcing and using manual intervention as a means of determining the quality of data.

Of all the presentations, panels and startup pitches there was not all that much time spent talking about underlying technology used by any of the companies. James Whittaker of TweetDeck reiterated Iain Dodsworths words from Le Web in saying TweetDeck would stay with the Adobe Air platform. He also mentioned that they were looking forward to the version 2.0 release of Air – I wish I’d asked why. Marco Kaiser of Seesmic, in my eyes TweetDeck’s main rival, talked about moving to the Windows platform to increase application performance and provide a plugin development framework. Other than that the only other mention of technology that I can remember were a couple of mentions of PUSH by David Maher Roberts of The Filter, who said none of their clients were using PUSH at the moment but there were requests in place and a POC ongoing, and Tim Morgan of Mint Digital who have recently released Football3’s which clearly relies on true real-time in order to be interactive.

It was a very interesting event and I’m really pleased that I attended but I would really have liked to have gained some insight into the technologies that companies are using and would have liked to see people talking about and moving to “real-time” in the true sense of the term rather than it being used as a marketing term.

You can view videos of most of the presentations on TechCrunch. Look out for one lady finding it particularly funny when I went to ask Marco of Seesmic a question and introduced myself:

Hi, I’m Phil Leggetter and I’m a Real-Time Web Software Consultant… I just made that up!

This was a joke, I really am a “Real-Time Web Software Consultant”, but I thought it might be quite funny, but not THAT funny. You can see this at just after 7:00 on this video:

Related posts:

1. The ChristmasCrunch nearly real-time web event
2. Microsoft Bing.com round table thoughts
3. When does NOW matter?

Our real-time search enables you to discover breaking news the moment it’s happening, even if it’s not the popular news of the day, and even if you didn’t know about it beforehand.

Although I’d still argue that we’re not seeing the real-time web with this solution I thought I’d go in to a bit of detail about how Google are achieving this HTTP polling solution.

Summary

• Google are consuming and indexing real-time data from multiple social networks
• It’s not real-time HTTP PUSH yet, it’s HTTP PULL using a polling interval
• The polling interval seems to vary but this needs to be confirmed.
• The polling result returns an encoded JSON response containing the next request to be made and the HTML of any new results. These results are injected into the “Latest results” section of the page
• The HTML returned is quite verbose and could be refined
• 87% from a sample of 30 polled requests returned no new results meaning the request was a waste of resources

Data Sources

Google now has an agreement in place with Twitter, Facebook, MySpace, FriendFeed, Identi.ca and Jaiku to consume and index their real-time data and make the results available in a “Latest Results” section of the Google search results. A number of social networks have APIs and some of these APIs have a real-time streaming feed and it’s likely that Google is simply connecting to this. For example, MySpace have released a Real-Time Stream API and Twitter has a Firehose feed, both of which Google will be using in their real-time search results. The Twitter Firehose feed is “not generally available” at the moment but Ryan Sarver, who is Director of Platform at Twitter, announced at Le Web that the Firehose stream would become generally available in the future.

How Google put the real-time data into their search results?

Since Google now have access to the data as soon as it is generated by the social network the next thing they need to do is get that into your search results. As soon as you make your search Google can put information that has just become available straight into the page. However, how do they get information that becomes available after your search results page has already loaded? Simple. They poll for new results regular intervals (as I mentioned in my How real-time does real-time have to be? blog post). So far I’ve seen a 20 second polling interval but recently, with a search for #leweb, I’ve seen a 30 second polling interval. It’s possible that they may have an idea how likely it is that your search subject will have new information available and change this polling interval accordingly.

These polling search requests look like this:

http://www.google.co.uk/realtimejs?q=%23leweb

&hl=en
&polltype=mb
&since=1260622409000000
&sessionstart=1260623982000000
&ei=bpgjS5zOG8j54Ab5kMzpCQ
&usg=05e5

And the responses:

mbrt0.insert('{\x22nextRequest\x22:\x22/realtimejs?q\\x3d%23leweb\\x26hl\\x3den\\x26output\\x3dsearch\\x26since\\x3d1260623734161591\\x26sessionstart\\x3d1260623385000000\\x26ei\\x3dbpgjS5zOG8j54Ab5kMzpCQ\\x26usg\\x3d3760\x22,\x22results\x22:[{\x22date\x22:\x221260623734161590\x22,\x22html\x22:\x22\\x3cli class\\x3d\\x22g s\\x22 style\\x3d\\x22line-height:16px;margin:0 0 8px;overflow:hidden\\x22\\x3e\\x3ctable class\\x3d\\x22ts\\x22\\x3e\\x3ctr\\x3e\\x3ctd colspan\\x3d3 class\\x3drt2 style\\x3d\\x22overflow:none\\x22\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-4px 0;width:6px;height:4px;float:right\\x22\\x3e\\x3c/div\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22width:4px;height:4px\\x22\\x3e\\x3c/div\\x3e\\x3ctr\\x3e\\x3ctd class\\x3drt1 style\\x3d\\x22background-position:-10px 0;background-repeat:repeat\\x22\\x3e\\x3cdiv style\\x3d\\x22width:2px\\x22\\x3e\\x3c/div\\x3e\\x3ctd style\\x3d\\x22padding:0 5px 1px;width:100%\\x22\\x3eFinally back at home after a long week in Paris. Great time at \\x3ca href\\x3d\\x22/search?q\\x3d%23LeWeb+site%3Atwitter.com\\x26amp;tbs\\x3dmbl:1\\x26amp;tbo\\x3d1\\x26amp;hl\\x3den\\x26amp;output\\x3dsearch\\x26amp;ei\\x3dcZgjS8PMNZbSjAfnl_nYBw\\x26amp;sa\\x3dX\\x26amp;oi\\x3dmicroblog_result\\x26amp;resnum\\x3d1\\x26amp;ct\\x3dresult\\x26amp;cd\\x3d1\\x26amp;ved\\x3d0CAMQoAQoADAA\\x22\\x3e#\\x3cem\\x3eLeWeb\\x3c/em\\x3e\\x3c/a\\x3e. Reconnected with lots of old friends etc,\\x3ctd class\\x3drt1 style\\x3d\\x22background-position:-14px 0;background-repeat:repeat\\x22\\x3e\\x3cdiv style\\x3d\\x22width:3px\\x22\\x3e\\x3c/div\\x3e\\x3ctr\\x3e\\x3ctd colspan\\x3d3 class\\x3drt2 style\\x3d\\x22background-position:-4px\\x22\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-4px -4px;width:6px;height:6px;float:right\\x22\\x3e\\x3c/div\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:0 -4px;width:4px;height:6px\\x22\\x3e\\x3c/div\\x3e\\x3c/table\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-17px 0;margin-top:-3px;margin-left:5px;padding:4px 0 0 23px\\x22\\x3e\\x3cspan class\\x3dgl\\x3e\\x3ca href\\x3d\\x22http://twitter.com/bdesarnauts\\x22 class\\x3dl onmousedown\\x3d\\x22return clk(this.href,\\x27microblog_result\\x27,\\x27\\x27,\\x27result\\x27,\\x271\\x27,\\x27\\x27,\\x270CAIQoAQwAA\\x27)\\x22\\x3ebdesarnauts\\x3c/a\\x3e\\x3c/span\\x3e - \\x3cspan class\\x3da\\x3etwitter.com\\x3c/span\\x3e - \\x3ca href\\x3d\\x22/url?q\\x3dhttp://twitter.com/bdesarnauts/status/6598672912\\x26amp;ei\\x3dcZgjS8PMNZbSjAfnl_nYBw\\x26amp;sa\\x3dX\\x26amp;oi\\x3dmicroblog_result\\x26amp;resnum\\x3d1\\x26amp;ct\\x3dresult\\x26amp;cd\\x3d2\\x26amp;ved\\x3d0CAQQoAQoADAA\\x26amp;usg\\x3dAFQjCNEyjEaOrJexPm6wUgqqVEDg24BOFQ\\x22 style\\x3d\\x22text-decoration:none\\x22\\x3e\\x3cspan class\\x3d\\x22f rtdm\\x22\\x3e4 minutes ago\\x3c/span\\x3e\\x3c/a\\x3e\\x3c/div\\x3e\x22}]}');

The response is an encoded JSON response and contains two things:

• nextRequest: The next polling search request to make
• results: An Array of Encoded HTML for any results that have been found for the current polling request.

In the example response above we get a next request of:

/realtimejs?
q\\x3d%23leweb\\x26hl\\x3den\\x26output\\x3dsearch\\x26since\\x3d1260623734161591\\x26sessionstart\\x3d1260623385000000\\x26ei\\x3dbpgjS5zOG8j54Ab5kMzpCQ\\x26usg\\x3d3760

Which translates to the following URL:

http://www.google.co.uk/realtimejs?q=%23leweb&hl=en&output=search&since=1260623734161591&sessionstart=1260623385000000&ei=bpgjS5zOG8j54Ab5kMzpCQ&usg=3760

From looking at the parameters it would appear that the since parameter would be very useful and would allow you to retrieve historical updates.

Each encoded HTML result in the results array will be decoded on the client using JavaScript and injected into the “Latest Results” section within the search result listings. It’s quite surprising that they return quite so much encoded HTML content and not just the information and then generate the HTML on the client. I suppose this is a new feature and it will be refined over time.

One point worth mentioning is that because Google are using a polling technique they are performing a lot of HTTP PULL requests to the server that return no results. This is a wasted request. From a sample size of 30 polled requests only 4 requests actually returned new results. This means that 87% of requests are a complete waste of resources.

Related posts:

1. Google realtime search isn’t real-time
2. Add real-time news to your site with Google. Add any real-time data to your site with Kwwika
3. Real-Time Rich Internet Applications (RTRIA)

The Code

First, and this is Important:

Now that you are aware of that, you will also need the Silverlight development environment. You can get everything you need via the Silverlight Getting Started page.

You’ve now got everything you need to run the RTRIA example. To run the sample application you should set the MSDNFlashRTRIAExample.Web project as the startup project and the MSDNFlashRTRIAExampleTestPage.html page as the startup page.

Setting up the solution to run the application

If you’d like to find out a bit more about the code then read on. If you’d rather jump straight into the code you can download it from the TweetStreamer Google Code project.

The streaming connection

The following extracts of code may be slightly modified but that has been done to be able to explain what the code does in general a bit better.

The following extract is used to establish a connection to the Twitter real-time data stream using a HttpWebRequest. The important thing to note is the use of request.AllowReadStreamBuffering = false; which is required since we are requesting a streaming feed. Without setting the AllowReadStreamBuffering property to false the ConnectionResponseCallback callback will not be invoked because the response will be continuously buffering.

Since the Twitter real-time data stream requires authentication, and we can’t set Credentials on the HttpWebRequest in Silverlight, the browser will prompt the user for a username and password.

private const string SPRITZER_URL = "http://stream.twitter.com/1/statuses/sample.json";

/// <summary>
/// Starts the connection to the Twitter real-time data stream.
/// </summary>
public void Connect()
{
    this.InternalConnectionStatus = ConnectionStatus.Connecting;

    try
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(SPRITZER_URL));
        request.AllowReadStreamBuffering = false;
        request.BeginGetResponse(ConnectionResponseCallback, request);
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
}

Within the callback method we ensure that we are connected and then call the ReadResponseStream method which will not return until we call Disconnect().

/// <summary>
/// Called when the initial connection has been established.
/// </summary>
private void ConnectionResponseCallback(IAsyncResult asynchronousResult)
{
    try
    {
        HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
        using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult))
        {
            if (response.StatusCode == HttpStatusCode.OK)
            {
                this.InternalConnectionStatus = ConnectionStatus.Connected;

                this.ReadResponseStream(request, response);

                if (this.InternalConnectionStatus != ConnectionStatus.Disconnecting)
                {
                    // unexpected status
                    Debug.WriteLine("unexpected connection status: " + this.InternalConnectionStatus);
                }
            }
            else
            {
                Debug.WriteLine("unexpected status code: " + response.StatusCode);
            }
        }

        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
    }
}

In the ReadResponseStream method we continually read the stream until we are interrupted by the user setting the connection status to a value other than ConnectionStatus.Connected, by calling the Disconnect() method, or an exception is thrown whilst reading from the stream. For each read from the stream we parse the data to get the tweets from the real-time data feed in JSON format.

/// <summary>
/// Reads the information received from the Twitter real-time data stream.
/// </summary>
private void ReadResponseStream(HttpWebRequest request, HttpWebResponse response)
{
    byte[] buffer = new byte[65536];
    using (Stream stream = response.GetResponseStream())
    {
        while (this.InternalConnectionStatus == ConnectionStatus.Connected)
        {
            int read = stream.Read(buffer, 0, buffer.Length);
            UTF8Encoding encoding = new UTF8Encoding();
            string data = encoding.GetString(buffer, 0, read);
            ParseResponseChunk(data);
        }

        // need to call request.Abort or the the thread will block at the end of
        // the using block.
        request.Abort();
    }
}

The ParseResponseChunk checks the data it’s passed and ensures that the data contains at least one full status message (tweet). I’ll not go into the details of that here since it’s just a matter of string parsing.

I chose to use the JSON format simply because the content passed over the wire is smaller than the XML feed. This should mean that the application has to do less work to read all the data. What we really should also do is benchmark the deserialisation of JSON against the deserialisation of XML to see which performs best within a Silverlight application.

Deserialising the JSON

The following JavaScript JSON snipped shows an example of a single Tweet that we get back from the Twitter real-time data feed.

{
    "in_reply_to_status_id":9999999,
    "in_reply_to_user_id":00000000,
    "favorited":false,
    "in_reply_to_screen_name":"leggetter",
    "text":"@leggetter Wow! A Real-Time Rich Internet Application (RTRIA)",
    "id":2820354600,
    "created_at":"Fri Nov 4 09:39:33 +0000 2009",
    "truncated":false,
    "source":"<a href=\"http:\/\/tweetdeck.com\/\">TweetDeck<\/a>"
}

The JSON can be deserialised as an instance of a C# class using the DataContract attribute on the class and the DataMember attributes on properties.

using System;
using System.Runtime.Serialization.Json;
using System.Runtime.Serialization;
using System.ComponentModel;

namespace TweetStreamer
{
    [DataContract]
    public class StatusMessage : IStatusMessage, INotifyPropertyChanged
    {
        [DataMember(Name = "text")]
        public string Text {get;set}
    }
}

A single Tweet, or in TweetStreamer a StatusMessage, can be deserialised using an instance of the DataContractJsonSerializer.

/// <summary>
/// Creates a single message from json string.
/// </summary>
/// <param name="messageData">The message data.</param>
/// <returns></returns>
private static IStatusMessage CreateMessageFromJsonString(string messageData)
{
    Debug.WriteLine(String.Format("Creating StatusMessage for: {0}", messageData));

    IStatusMessage message = null;

    using (MemoryStream stream = new MemoryStream(UTF8Encoding.UTF8.GetBytes(messageData)))
    {
        DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(StatusMessage));
        message = ser.ReadObject(stream) as StatusMessage;
    }

    if (string.IsNullOrEmpty(message.Id))
    {
        message = null;
        Debug.WriteLine("message had no ID. Assuming to be a delete message so nulling message object");
    }

    return message;
}

Binding the data to the grid

To bind the data to a DataGrid the grid needs to be defined in XAML. In addition we can specify the properties on the StatusMessage that we want to bind to columns. In the XAML below we are binding the CreatedAtString property to a Time column, a User.ScreenName to a User column, and a Text property to a Message column. Notice the cool binding of User.ScreenName. The StatusMessage.User property returns an instance of another class and we are actually binding to a property on the returned class.

<data:DataGrid Grid.Row="0" x:Name="Tweets" AutoGenerateColumns="False">
    <data:DataGrid.Columns>
        <data:DataGridTextColumn Header="Time"
            Binding="{Binding CreatedAtString}" />
        <data:DataGridTextColumn Header="User"
            Binding="{Binding User.ScreenName}" />
        <data:DataGridTextColumn Header="Message"
            Binding="{Binding Text}" />
    </data:DataGrid.Columns>
</data:DataGrid>

Next, the code to set up the binding and registering for StatusMessage updates using the StatusMessageReceived event. As you many have noticed, the StatusMessage object implements the INotifyPropertyChanged interface. This was used so that we could add each StatusMessage to an ObservableCollection<T> and then bind it to a DataGrid to display the Tweets in real-time.

Connection _twitterConnection;
ObservableCollection<IStatusMessage> _messages = new ObservableCollection<IStatusMessage>();
public MainPage()
{
    InitializeComponent();

    Tweets.ItemsSource = _messages;

    _twitterConnection = new Connection();
    _twitterConnection.StatusMessageReceived += new Connection.OnStatusMessageReceivedEventHandler
        (twitterConnection_StatusMessageReceived);
    _twitterConnection.Connect();
}

Finally, whenever we get a StatusMessageReceived callback we need to add the new StatusMessage to the ObservableCollection collection. This has to be done on the UI thread and via the Tweets.ItemsSource property or the UI will not update.

/// <summary>
/// Status message received event handler.
/// </summary>
/// <param name="sender"></param>
/// <param name="args"></param>
void twitterConnection_StatusMessageReceived(object sender, IStatusMessageReceivedEventArgs args)
{
    Dispatcher.BeginInvoke(() =>
        AddMessage(args.Message)
    );
}

/// <summary>
/// Adds a message to the observable message list which updates the UI.
/// </summary>
/// <param name="message"></param>
void AddMessage(IStatusMessage message)
{
    ObservableCollection<IStatusMessage> messageList = ((ObservableCollection<IStatusMessage>)Tweets.ItemsSource);
    messageList.Insert(0, message);
}

Hopefully this will have helped you understand how the Twitter real-time data stream is consumed and an example of how you can use it within a Real-Time Rich Internet Application. Now, why now download the TweetStreamer library and example application and have a play.

Download

You can download the source from the TweetStreamer Google Code project.

Screencast

This screencast was supposed to be short but ended up being just shy of 10 minutes. In it I provide some technical detail of how I built the application, show the basics of how Fiddler is used to give access to the Twitter real-time data stream, and give a demo of the application.

Related posts:

1. Real-Time Rich Internet Applications (RTRIA)
2. Which Rich Internet Application Technology will dominate?
3. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request

How Silverlight makes a web request

If you were making a web request from Silverlight to download the following page:

http://www.leggetter.co.uk/2009/10/23/how-i-approach-problem-solving-in-code.html

The Silverlight runtime would first request

http://www.leggetter.co.uk/clientaccesspolicy.xml

If this request fails it will then request

http://www.leggetter.co.uk/crossdomain.xml

If both of these requests fail (return a 404 HTTP status) the Silverlight runtime will not allow you to make your web request. *

* Note: The web request failure due to a failure to download a security policy file can manifest itself in a number of ways. I’ll try to remember to put a post up about this later.

The clientaccesspolicy.xml file is Microsoft’s own security policy file. The crossdomain.xml file is used by Adobe Flash but is also supported by Silverlight. For more information see master-policy file.

Now that we’ve got the background information out of the way we can get on to the interesting stuff.

An example using the Twitter Streaming Feed

If you wanted to write your own Real-Time Rich Internet Application (RTRIA) that consumed data from the Twitter real-time data stream you’d be stuck. Well, you’ll be stuck until Twitter puts a security policy file up **. Until then you can trick the Silverlight runtime by using Fiddler to detect the security policy file request and fake a response.

** The following URLs are where Twitter would put their security policy files should they decide to allow cross domain access. Please not that you’ll be prompted for your Twitter username and password if you click on them by the browser because you need to sign in with valid twitter credentials to access the twitter stream domain: http://stream.twitter.com/clientaccesspolicy.xml and http://stream.twitter.com/crossdomain.xml)

To do this you will need to set up an AutoResponder in Fiddler that intercepts the request for the security policy file by the Silverlight runtime and returns a fake security policy file. In the example below I have an AutoResponder set up for http://stream.twitter.com/crossdomain.xml.

Using Fiddler to AutoRespond with a fake Security Policy File

Here’s an example of a crossdomain.xml security policy file which grants access to requests from all domains.

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy
  SYSTEM "http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
  <allow-access-from domain="*" />
</cross-domain-policy>

Once you’ve set up the AutoResponder start Fiddler. Then start your Silverlight application that is trying to stream data from Twitter and bingo! You should have real-time data from Twitter in your RIA – a Real-Time Rich Internet Application (RTRIA).

Fiddler HTTP Proxy Screencast

Related posts:

1. How to make a cross domain web request with SilverLight 2
2. Silverlight uses XCP tmp files for Web Requests
3. Using Fiddler to help develop cross domain capable JavaScript web applications

I mentioned RTRIAs for the first time back in April 2009 in a post called “What is the real-time web” on the Caplin Systems Platformability blog, and since then there has been a real-time web explosion. The “real-time web” is now the buzz phrase around the Internet that everybody is jumping on. Back in April I set up a Google Alert for the term “real-time web”. Back then I got maybe one Google Alert a day for this term, probably less. Now, I get at least two dense emails a day from Google with people using the term for all sorts of things; real-time web stats, real-time analytics, real-time search, rssCloud, pubsubhubbub, the list goes on. This is not the real-time web! Whilst some of the things on this list will help  the web become truly real-time (rssCloud and pubsubhubbub may even form the back bone), none of these things give the user a truly real-time web experience.

My opinion about what the real-time web is has not changed from that first blog post. For me, it’s not just about data being available as soon as it’s been published. In this situation the user still has to actively make a request to see if any data is available. Instead it’s about making a single request for data (a subscription) and that data being pushed to your RIA to be consumed by the application user in real-time. Quite a few applications have implemented a polling solution (more on this later) but the experience presented to the user is not truly real-time.

"Real-Time?"

In my opinion the only way to guarantee being able to deliver truly real-time data (TRTD) to a web application is by maintaining a streaming connection to the source of the data using a continuous streaming HTTP connection. Most web applications use a technique called polling or long polling to retrieve their data which means that although the data is initially “fresh”, within seconds it could be out of date, or more information may be available. I’ll discuss server technologies further in a later post.

At the moment there are a number of technologies that allow you to develop RTRIAs. Ajax, Silverlight and Flash are the three mainstream solutions. The core barriers to RTRIAs are the scalability of server technology and access to truly real-time data.

Related posts:

1. Which Rich Internet Application Technology will dominate?
2. The ChristmasCrunch nearly real-time web event
3. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request