My main interest in this event is to see if anybody is going to start delivering a “truly real-time” experience to users. Will TweetDeck still be making polling requests to the various services it uses to check if any more data is available or will they move to using PUSH technology? Are other companies going to start adopting PUSH to give the user a truly real-time web experience?

I’ve already mentioned my opinion time and time and time again that what we presently see as the real-time web isn’t in fact giving any of us a truly real-time experience, and this won’t be the case until data is pushed to us in real-time. If we have to actively check to see if there is new data available then how can anybody claim that we are receiving data in real-time? As soon as the new data is available we should not only be informed about that new data, but also given it.

Let me use Google search as an example: Google is getting really good at almost instantly picking up new content on the web – what everybody is calling the “real-time web” experience. I’ve also observed that it looks like they’ve started to rank new content higher, so they are really embracing and valuing the freshness of information. If I were to go to Google.com and search for “Real-Time Rich Internet Applications” I’d get a number of relevant and potentially fresh results.

If just after Google returns these results to me Robert Scoble decides that RTRIAs are the next big and publishes a blog post on the subject, leading to a number of other blog posts and tweets, I’d be none the wiser until I resubmit my search to see the updated results. Google would no doubt pick up this new, high profile bloggers, blog post and it would be available, maybe even top in the search listings, when I next searched for “Real-Time Rich Internet Applications”. But, I’ve done my search and I’d believe that I have got the most relevant information available at that moment so my search results would be out of date.

Until companies move away from the old PULL web model and move to PUSH we are going to be stuck with a nearly-real-time web experience. I hope I get the opportunity to ask a few people at the ChristmasCrunch about when, not if, they will be moving to a PUSH model to deliver a truly real-time web experience to their users.

Related posts:

1. Real-Time Rich Internet Applications (RTRIA)
2. Thoughts on the TechCrunch ChristmasCrunch (#xmascrunch)
3. How real-time does real-time have to be?

The Code

First, and this is Important:

Now that you are aware of that, you will also need the Silverlight development environment. You can get everything you need via the Silverlight Getting Started page.

You’ve now got everything you need to run the RTRIA example. To run the sample application you should set the MSDNFlashRTRIAExample.Web project as the startup project and the MSDNFlashRTRIAExampleTestPage.html page as the startup page.

Setting up the solution to run the application

If you’d like to find out a bit more about the code then read on. If you’d rather jump straight into the code you can download it from the TweetStreamer Google Code project.

The streaming connection

The following extracts of code may be slightly modified but that has been done to be able to explain what the code does in general a bit better.

The following extract is used to establish a connection to the Twitter real-time data stream using a HttpWebRequest. The important thing to note is the use of request.AllowReadStreamBuffering = false; which is required since we are requesting a streaming feed. Without setting the AllowReadStreamBuffering property to false the ConnectionResponseCallback callback will not be invoked because the response will be continuously buffering.

Since the Twitter real-time data stream requires authentication, and we can’t set Credentials on the HttpWebRequest in Silverlight, the browser will prompt the user for a username and password.

private const string SPRITZER_URL = "http://stream.twitter.com/1/statuses/sample.json";

/// <summary>
/// Starts the connection to the Twitter real-time data stream.
/// </summary>
public void Connect()
{
    this.InternalConnectionStatus = ConnectionStatus.Connecting;

    try
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(SPRITZER_URL));
        request.AllowReadStreamBuffering = false;
        request.BeginGetResponse(ConnectionResponseCallback, request);
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
}

Within the callback method we ensure that we are connected and then call the ReadResponseStream method which will not return until we call Disconnect().

/// <summary>
/// Called when the initial connection has been established.
/// </summary>
private void ConnectionResponseCallback(IAsyncResult asynchronousResult)
{
    try
    {
        HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
        using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult))
        {
            if (response.StatusCode == HttpStatusCode.OK)
            {
                this.InternalConnectionStatus = ConnectionStatus.Connected;

                this.ReadResponseStream(request, response);

                if (this.InternalConnectionStatus != ConnectionStatus.Disconnecting)
                {
                    // unexpected status
                    Debug.WriteLine("unexpected connection status: " + this.InternalConnectionStatus);
                }
            }
            else
            {
                Debug.WriteLine("unexpected status code: " + response.StatusCode);
            }
        }

        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
    }
}

In the ReadResponseStream method we continually read the stream until we are interrupted by the user setting the connection status to a value other than ConnectionStatus.Connected, by calling the Disconnect() method, or an exception is thrown whilst reading from the stream. For each read from the stream we parse the data to get the tweets from the real-time data feed in JSON format.

/// <summary>
/// Reads the information received from the Twitter real-time data stream.
/// </summary>
private void ReadResponseStream(HttpWebRequest request, HttpWebResponse response)
{
    byte[] buffer = new byte[65536];
    using (Stream stream = response.GetResponseStream())
    {
        while (this.InternalConnectionStatus == ConnectionStatus.Connected)
        {
            int read = stream.Read(buffer, 0, buffer.Length);
            UTF8Encoding encoding = new UTF8Encoding();
            string data = encoding.GetString(buffer, 0, read);
            ParseResponseChunk(data);
        }

        // need to call request.Abort or the the thread will block at the end of
        // the using block.
        request.Abort();
    }
}

The ParseResponseChunk checks the data it’s passed and ensures that the data contains at least one full status message (tweet). I’ll not go into the details of that here since it’s just a matter of string parsing.

I chose to use the JSON format simply because the content passed over the wire is smaller than the XML feed. This should mean that the application has to do less work to read all the data. What we really should also do is benchmark the deserialisation of JSON against the deserialisation of XML to see which performs best within a Silverlight application.

Deserialising the JSON

The following JavaScript JSON snipped shows an example of a single Tweet that we get back from the Twitter real-time data feed.

{
    "in_reply_to_status_id":9999999,
    "in_reply_to_user_id":00000000,
    "favorited":false,
    "in_reply_to_screen_name":"leggetter",
    "text":"@leggetter Wow! A Real-Time Rich Internet Application (RTRIA)",
    "id":2820354600,
    "created_at":"Fri Nov 4 09:39:33 +0000 2009",
    "truncated":false,
    "source":"<a href=\"http:\/\/tweetdeck.com\/\">TweetDeck<\/a>"
}

The JSON can be deserialised as an instance of a C# class using the DataContract attribute on the class and the DataMember attributes on properties.

using System;
using System.Runtime.Serialization.Json;
using System.Runtime.Serialization;
using System.ComponentModel;

namespace TweetStreamer
{
    [DataContract]
    public class StatusMessage : IStatusMessage, INotifyPropertyChanged
    {
        [DataMember(Name = "text")]
        public string Text {get;set}
    }
}

A single Tweet, or in TweetStreamer a StatusMessage, can be deserialised using an instance of the DataContractJsonSerializer.

/// <summary>
/// Creates a single message from json string.
/// </summary>
/// <param name="messageData">The message data.</param>
/// <returns></returns>
private static IStatusMessage CreateMessageFromJsonString(string messageData)
{
    Debug.WriteLine(String.Format("Creating StatusMessage for: {0}", messageData));

    IStatusMessage message = null;

    using (MemoryStream stream = new MemoryStream(UTF8Encoding.UTF8.GetBytes(messageData)))
    {
        DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(StatusMessage));
        message = ser.ReadObject(stream) as StatusMessage;
    }

    if (string.IsNullOrEmpty(message.Id))
    {
        message = null;
        Debug.WriteLine("message had no ID. Assuming to be a delete message so nulling message object");
    }

    return message;
}

Binding the data to the grid

To bind the data to a DataGrid the grid needs to be defined in XAML. In addition we can specify the properties on the StatusMessage that we want to bind to columns. In the XAML below we are binding the CreatedAtString property to a Time column, a User.ScreenName to a User column, and a Text property to a Message column. Notice the cool binding of User.ScreenName. The StatusMessage.User property returns an instance of another class and we are actually binding to a property on the returned class.

<data:DataGrid Grid.Row="0" x:Name="Tweets" AutoGenerateColumns="False">
    <data:DataGrid.Columns>
        <data:DataGridTextColumn Header="Time"
            Binding="{Binding CreatedAtString}" />
        <data:DataGridTextColumn Header="User"
            Binding="{Binding User.ScreenName}" />
        <data:DataGridTextColumn Header="Message"
            Binding="{Binding Text}" />
    </data:DataGrid.Columns>
</data:DataGrid>

Next, the code to set up the binding and registering for StatusMessage updates using the StatusMessageReceived event. As you many have noticed, the StatusMessage object implements the INotifyPropertyChanged interface. This was used so that we could add each StatusMessage to an ObservableCollection<T> and then bind it to a DataGrid to display the Tweets in real-time.

Connection _twitterConnection;
ObservableCollection<IStatusMessage> _messages = new ObservableCollection<IStatusMessage>();
public MainPage()
{
    InitializeComponent();

    Tweets.ItemsSource = _messages;

    _twitterConnection = new Connection();
    _twitterConnection.StatusMessageReceived += new Connection.OnStatusMessageReceivedEventHandler
        (twitterConnection_StatusMessageReceived);
    _twitterConnection.Connect();
}

Finally, whenever we get a StatusMessageReceived callback we need to add the new StatusMessage to the ObservableCollection collection. This has to be done on the UI thread and via the Tweets.ItemsSource property or the UI will not update.

/// <summary>
/// Status message received event handler.
/// </summary>
/// <param name="sender"></param>
/// <param name="args"></param>
void twitterConnection_StatusMessageReceived(object sender, IStatusMessageReceivedEventArgs args)
{
    Dispatcher.BeginInvoke(() =>
        AddMessage(args.Message)
    );
}

/// <summary>
/// Adds a message to the observable message list which updates the UI.
/// </summary>
/// <param name="message"></param>
void AddMessage(IStatusMessage message)
{
    ObservableCollection<IStatusMessage> messageList = ((ObservableCollection<IStatusMessage>)Tweets.ItemsSource);
    messageList.Insert(0, message);
}

Hopefully this will have helped you understand how the Twitter real-time data stream is consumed and an example of how you can use it within a Real-Time Rich Internet Application. Now, why now download the TweetStreamer library and example application and have a play.

Download

You can download the source from the TweetStreamer Google Code project.

Screencast

This screencast was supposed to be short but ended up being just shy of 10 minutes. In it I provide some technical detail of how I built the application, show the basics of how Fiddler is used to give access to the Twitter real-time data stream, and give a demo of the application.

Related posts:

1. Real-Time Rich Internet Applications (RTRIA)
2. Which Rich Internet Application Technology will dominate?
3. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request

How Silverlight makes a web request

If you were making a web request from Silverlight to download the following page:

http://www.leggetter.co.uk/2009/10/23/how-i-approach-problem-solving-in-code.html

The Silverlight runtime would first request

http://www.leggetter.co.uk/clientaccesspolicy.xml

If this request fails it will then request

http://www.leggetter.co.uk/crossdomain.xml

If both of these requests fail (return a 404 HTTP status) the Silverlight runtime will not allow you to make your web request. *

* Note: The web request failure due to a failure to download a security policy file can manifest itself in a number of ways. I’ll try to remember to put a post up about this later.

The clientaccesspolicy.xml file is Microsoft’s own security policy file. The crossdomain.xml file is used by Adobe Flash but is also supported by Silverlight. For more information see master-policy file.

Now that we’ve got the background information out of the way we can get on to the interesting stuff.

An example using the Twitter Streaming Feed

If you wanted to write your own Real-Time Rich Internet Application (RTRIA) that consumed data from the Twitter real-time data stream you’d be stuck. Well, you’ll be stuck until Twitter puts a security policy file up **. Until then you can trick the Silverlight runtime by using Fiddler to detect the security policy file request and fake a response.

** The following URLs are where Twitter would put their security policy files should they decide to allow cross domain access. Please not that you’ll be prompted for your Twitter username and password if you click on them by the browser because you need to sign in with valid twitter credentials to access the twitter stream domain: http://stream.twitter.com/clientaccesspolicy.xml and http://stream.twitter.com/crossdomain.xml)

To do this you will need to set up an AutoResponder in Fiddler that intercepts the request for the security policy file by the Silverlight runtime and returns a fake security policy file. In the example below I have an AutoResponder set up for http://stream.twitter.com/crossdomain.xml.

Using Fiddler to AutoRespond with a fake Security Policy File

Here’s an example of a crossdomain.xml security policy file which grants access to requests from all domains.

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy
  SYSTEM "http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
  <allow-access-from domain="*" />
</cross-domain-policy>

Once you’ve set up the AutoResponder start Fiddler. Then start your Silverlight application that is trying to stream data from Twitter and bingo! You should have real-time data from Twitter in your RIA – a Real-Time Rich Internet Application (RTRIA).

Fiddler HTTP Proxy Screencast

Related posts:

1. How to make a cross domain web request with SilverLight 2
2. Silverlight uses XCP tmp files for Web Requests
3. Using Fiddler to help develop cross domain capable JavaScript web applications

I mentioned RTRIAs for the first time back in April 2009 in a post called “What is the real-time web” on the Caplin Systems Platformability blog, and since then there has been a real-time web explosion. The “real-time web” is now the buzz phrase around the Internet that everybody is jumping on. Back in April I set up a Google Alert for the term “real-time web”. Back then I got maybe one Google Alert a day for this term, probably less. Now, I get at least two dense emails a day from Google with people using the term for all sorts of things; real-time web stats, real-time analytics, real-time search, rssCloud, pubsubhubbub, the list goes on. This is not the real-time web! Whilst some of the things on this list will help  the web become truly real-time (rssCloud and pubsubhubbub may even form the back bone), none of these things give the user a truly real-time web experience.

My opinion about what the real-time web is has not changed from that first blog post. For me, it’s not just about data being available as soon as it’s been published. In this situation the user still has to actively make a request to see if any data is available. Instead it’s about making a single request for data (a subscription) and that data being pushed to your RIA to be consumed by the application user in real-time. Quite a few applications have implemented a polling solution (more on this later) but the experience presented to the user is not truly real-time.

"Real-Time?"

In my opinion the only way to guarantee being able to deliver truly real-time data (TRTD) to a web application is by maintaining a streaming connection to the source of the data using a continuous streaming HTTP connection. Most web applications use a technique called polling or long polling to retrieve their data which means that although the data is initially “fresh”, within seconds it could be out of date, or more information may be available. I’ll discuss server technologies further in a later post.

At the moment there are a number of technologies that allow you to develop RTRIAs. Ajax, Silverlight and Flash are the three mainstream solutions. The core barriers to RTRIAs are the scalability of server technology and access to truly real-time data.

Related posts:

1. Which Rich Internet Application Technology will dominate?
2. The ChristmasCrunch nearly real-time web event
3. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request