Podcast

Phil Leggetter talks about creating a Silverlight application that consumes real-time data

Here’s a link to Eric’s original podcast post.

E-Book

Here’s a link to Eric’s original e-book post

Related posts:

1. Silverlight uses XCP tmp files for Web Requests
2. Thoughts on an SEO question raised in the Social Media White Noise podcast
3. Book review: Juggle! Rethink work, reclaim your life

The Code

First, and this is Important:

Now that you are aware of that, you will also need the Silverlight development environment. You can get everything you need via the Silverlight Getting Started page.

You’ve now got everything you need to run the RTRIA example. To run the sample application you should set the MSDNFlashRTRIAExample.Web project as the startup project and the MSDNFlashRTRIAExampleTestPage.html page as the startup page.

Setting up the solution to run the application

If you’d like to find out a bit more about the code then read on. If you’d rather jump straight into the code you can download it from the TweetStreamer Google Code project.

The streaming connection

The following extracts of code may be slightly modified but that has been done to be able to explain what the code does in general a bit better.

The following extract is used to establish a connection to the Twitter real-time data stream using a HttpWebRequest. The important thing to note is the use of request.AllowReadStreamBuffering = false; which is required since we are requesting a streaming feed. Without setting the AllowReadStreamBuffering property to false the ConnectionResponseCallback callback will not be invoked because the response will be continuously buffering.

Since the Twitter real-time data stream requires authentication, and we can’t set Credentials on the HttpWebRequest in Silverlight, the browser will prompt the user for a username and password.

private const string SPRITZER_URL = "http://stream.twitter.com/1/statuses/sample.json";

/// <summary>
/// Starts the connection to the Twitter real-time data stream.
/// </summary>
public void Connect()
{
    this.InternalConnectionStatus = ConnectionStatus.Connecting;

    try
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(SPRITZER_URL));
        request.AllowReadStreamBuffering = false;
        request.BeginGetResponse(ConnectionResponseCallback, request);
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
}

Within the callback method we ensure that we are connected and then call the ReadResponseStream method which will not return until we call Disconnect().

/// <summary>
/// Called when the initial connection has been established.
/// </summary>
private void ConnectionResponseCallback(IAsyncResult asynchronousResult)
{
    try
    {
        HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
        using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult))
        {
            if (response.StatusCode == HttpStatusCode.OK)
            {
                this.InternalConnectionStatus = ConnectionStatus.Connected;

                this.ReadResponseStream(request, response);

                if (this.InternalConnectionStatus != ConnectionStatus.Disconnecting)
                {
                    // unexpected status
                    Debug.WriteLine("unexpected connection status: " + this.InternalConnectionStatus);
                }
            }
            else
            {
                Debug.WriteLine("unexpected status code: " + response.StatusCode);
            }
        }

        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
    }
}

In the ReadResponseStream method we continually read the stream until we are interrupted by the user setting the connection status to a value other than ConnectionStatus.Connected, by calling the Disconnect() method, or an exception is thrown whilst reading from the stream. For each read from the stream we parse the data to get the tweets from the real-time data feed in JSON format.

/// <summary>
/// Reads the information received from the Twitter real-time data stream.
/// </summary>
private void ReadResponseStream(HttpWebRequest request, HttpWebResponse response)
{
    byte[] buffer = new byte[65536];
    using (Stream stream = response.GetResponseStream())
    {
        while (this.InternalConnectionStatus == ConnectionStatus.Connected)
        {
            int read = stream.Read(buffer, 0, buffer.Length);
            UTF8Encoding encoding = new UTF8Encoding();
            string data = encoding.GetString(buffer, 0, read);
            ParseResponseChunk(data);
        }

        // need to call request.Abort or the the thread will block at the end of
        // the using block.
        request.Abort();
    }
}

The ParseResponseChunk checks the data it’s passed and ensures that the data contains at least one full status message (tweet). I’ll not go into the details of that here since it’s just a matter of string parsing.

I chose to use the JSON format simply because the content passed over the wire is smaller than the XML feed. This should mean that the application has to do less work to read all the data. What we really should also do is benchmark the deserialisation of JSON against the deserialisation of XML to see which performs best within a Silverlight application.

Deserialising the JSON

The following JavaScript JSON snipped shows an example of a single Tweet that we get back from the Twitter real-time data feed.

{
    "in_reply_to_status_id":9999999,
    "in_reply_to_user_id":00000000,
    "favorited":false,
    "in_reply_to_screen_name":"leggetter",
    "text":"@leggetter Wow! A Real-Time Rich Internet Application (RTRIA)",
    "id":2820354600,
    "created_at":"Fri Nov 4 09:39:33 +0000 2009",
    "truncated":false,
    "source":"<a href=\"http:\/\/tweetdeck.com\/\">TweetDeck<\/a>"
}

The JSON can be deserialised as an instance of a C# class using the DataContract attribute on the class and the DataMember attributes on properties.

using System;
using System.Runtime.Serialization.Json;
using System.Runtime.Serialization;
using System.ComponentModel;

namespace TweetStreamer
{
    [DataContract]
    public class StatusMessage : IStatusMessage, INotifyPropertyChanged
    {
        [DataMember(Name = "text")]
        public string Text {get;set}
    }
}

A single Tweet, or in TweetStreamer a StatusMessage, can be deserialised using an instance of the DataContractJsonSerializer.

/// <summary>
/// Creates a single message from json string.
/// </summary>
/// <param name="messageData">The message data.</param>
/// <returns></returns>
private static IStatusMessage CreateMessageFromJsonString(string messageData)
{
    Debug.WriteLine(String.Format("Creating StatusMessage for: {0}", messageData));

    IStatusMessage message = null;

    using (MemoryStream stream = new MemoryStream(UTF8Encoding.UTF8.GetBytes(messageData)))
    {
        DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(StatusMessage));
        message = ser.ReadObject(stream) as StatusMessage;
    }

    if (string.IsNullOrEmpty(message.Id))
    {
        message = null;
        Debug.WriteLine("message had no ID. Assuming to be a delete message so nulling message object");
    }

    return message;
}

Binding the data to the grid

To bind the data to a DataGrid the grid needs to be defined in XAML. In addition we can specify the properties on the StatusMessage that we want to bind to columns. In the XAML below we are binding the CreatedAtString property to a Time column, a User.ScreenName to a User column, and a Text property to a Message column. Notice the cool binding of User.ScreenName. The StatusMessage.User property returns an instance of another class and we are actually binding to a property on the returned class.

<data:DataGrid Grid.Row="0" x:Name="Tweets" AutoGenerateColumns="False">
    <data:DataGrid.Columns>
        <data:DataGridTextColumn Header="Time"
            Binding="{Binding CreatedAtString}" />
        <data:DataGridTextColumn Header="User"
            Binding="{Binding User.ScreenName}" />
        <data:DataGridTextColumn Header="Message"
            Binding="{Binding Text}" />
    </data:DataGrid.Columns>
</data:DataGrid>

Next, the code to set up the binding and registering for StatusMessage updates using the StatusMessageReceived event. As you many have noticed, the StatusMessage object implements the INotifyPropertyChanged interface. This was used so that we could add each StatusMessage to an ObservableCollection<T> and then bind it to a DataGrid to display the Tweets in real-time.

Connection _twitterConnection;
ObservableCollection<IStatusMessage> _messages = new ObservableCollection<IStatusMessage>();
public MainPage()
{
    InitializeComponent();

    Tweets.ItemsSource = _messages;

    _twitterConnection = new Connection();
    _twitterConnection.StatusMessageReceived += new Connection.OnStatusMessageReceivedEventHandler
        (twitterConnection_StatusMessageReceived);
    _twitterConnection.Connect();
}

Finally, whenever we get a StatusMessageReceived callback we need to add the new StatusMessage to the ObservableCollection collection. This has to be done on the UI thread and via the Tweets.ItemsSource property or the UI will not update.

/// <summary>
/// Status message received event handler.
/// </summary>
/// <param name="sender"></param>
/// <param name="args"></param>
void twitterConnection_StatusMessageReceived(object sender, IStatusMessageReceivedEventArgs args)
{
    Dispatcher.BeginInvoke(() =>
        AddMessage(args.Message)
    );
}

/// <summary>
/// Adds a message to the observable message list which updates the UI.
/// </summary>
/// <param name="message"></param>
void AddMessage(IStatusMessage message)
{
    ObservableCollection<IStatusMessage> messageList = ((ObservableCollection<IStatusMessage>)Tweets.ItemsSource);
    messageList.Insert(0, message);
}

Hopefully this will have helped you understand how the Twitter real-time data stream is consumed and an example of how you can use it within a Real-Time Rich Internet Application. Now, why now download the TweetStreamer library and example application and have a play.

Download

You can download the source from the TweetStreamer Google Code project.

Screencast

This screencast was supposed to be short but ended up being just shy of 10 minutes. In it I provide some technical detail of how I built the application, show the basics of how Fiddler is used to give access to the Twitter real-time data stream, and give a demo of the application.

Related posts:

1. Real-Time Rich Internet Applications (RTRIA)
2. Which Rich Internet Application Technology will dominate?
3. The easiest way to add real-time functionality to an ASP.NET e-commerce application