This is particularly noticeable and reproducible if you are using the HttpWebRequest class to stream data (for more information on streaming data from Silverlight you can listen to my podcast and read my article in a free MSDN book). You can then reproduce the loss of connection using Fiddler (which seriously rocks and is becoming more and more useful) by right-clicking on the streaming connection and selecting “Abort Session”.

Fiddler - Abort Session

The best solution to resolve this that I’ve found is to manually invoke the garbage collector whenever you detect the connection loss.

// Manually invoke Garbage collection
GC.Collect();

Related posts:

1. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request
2. C# – Get Windows Temporary Directory
3. How to make a cross domain web request with SilverLight 2

Podcast

Phil Leggetter talks about creating a Silverlight application that consumes real-time data

Here’s a link to Eric’s original podcast post.

E-Book

Here’s a link to Eric’s original e-book post

Related posts:

1. Silverlight uses XCP tmp files for Web Requests
2. Book review: Juggle! Rethink work, reclaim your life

I think the UI revolution’s already here; now we see which ones bubble to the top!

This got me thinking about the criteria that may determine which RIA technology would be most successful.

I think there are a number of things that will determine which one will ultimately win, or at least become the dominant choice. At the moment I can think of three categories; Technology buy-in, Development environment and Data access capabilities and choice.

Technology buy-in

Some companies buy-in to a technology because they have always used that technology. For example, a company that have always used Java will be more likely to opt for the Adobe solution where they can write their server components using Java and their client in Flash/ActionScript/Flex (this terminology is still a bit unclear to me. I think this is why Adobe are renaming Flex Builder to Flash Builder) but a company that have traditionally used Microsoft products will undoubtedly go for Silverlight. More traditional web developers, who probably build upon the LAMP stack, that have been hand coding PHP on the server and JavaScript on the client tend to go for creating RIAs using Ajax.

Development environment

I believe that Microsoft have a massive advantage in having Visual Studio as a server and client IDE. It has it’s faults but in general it’s a fantastic development environment. Those choosing Flex (Java -> ActionScript/Flash) will more than likely choose the Eclipse IDE, which is powerful, but it’s ActionScript/Flex development tools are lacking maturity. I’ve recently used the Flash Builder 4 beta and it’s miles behind Visual Studio in terms of coding productivity features. I’ve heard that the Flex UI component development features may be better than Visual Studio’s support for Silverlight UI. The LAMP with Ajax stack suffers much the same problems as Flex with, as far as I know, Eclipse being the IDE of choice and client-side productivity tools still very much in their infancy. It’s clearly very difficult to write an IDE to enhance JavaScript/Ajax productivity.

Data access capabilities and choice

I think this has been one of the last things to be considered in the RIA stack. Previously browsers have locked down where you can get your data from. Developers were always trying to implement hacks to allow them to do things such as make a web request to a server on a different domain or cross-site scripting to implement HTTP streaming. As the web has opened up to the idea of Mashups, browser and RIA plugin vendors have had to change their stance on cross site security. It’s a difficult line to walk but I believe that the technology that makes retrieving data from multiple sources the easiest, without having to go via a proxy, will have a distinct advantage. Flash and Silverlight are winning the race at the moment with Silverlight’s use of security policy files and Flash’s support for master policy files. For the Ajax solution I still need to research information on HTTP cross domain access control headers and which browsers have implemented support for them. I still think there will be an argument for mashing up your data on the server and I have a blog post planned on RTRIA server technologies.

The important questions around RIA data are:

• Which RIA will be more efficient at consuming data?
• Which one will prove to be the most capable as a Real-Time Rich Internet Application (RTRIA)?
• Will RIA technology vendors try to lock developers into using a client-server technology stack, such as Adobe may be trying to do with LCDS and Flex, and will this put developers off?
• Is decoupling of server and client technology, and communication protocol, important? I think it is, do you?

Although RIAs have been around for a while now, as Patrick said, the race to dominate the Rich Internet Application Technology space is on.

Related posts:

1. Real-Time Rich Internet Applications (RTRIA)
2. Using real-time web software and technology to distribute events
3. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request

The Code

First, and this is Important:

Now that you are aware of that, you will also need the Silverlight development environment. You can get everything you need via the Silverlight Getting Started page.

You’ve now got everything you need to run the RTRIA example. To run the sample application you should set the MSDNFlashRTRIAExample.Web project as the startup project and the MSDNFlashRTRIAExampleTestPage.html page as the startup page.

Setting up the solution to run the application

If you’d like to find out a bit more about the code then read on. If you’d rather jump straight into the code you can download it from the TweetStreamer Google Code project.

The streaming connection

The following extracts of code may be slightly modified but that has been done to be able to explain what the code does in general a bit better.

The following extract is used to establish a connection to the Twitter real-time data stream using a HttpWebRequest. The important thing to note is the use of request.AllowReadStreamBuffering = false; which is required since we are requesting a streaming feed. Without setting the AllowReadStreamBuffering property to false the ConnectionResponseCallback callback will not be invoked because the response will be continuously buffering.

Since the Twitter real-time data stream requires authentication, and we can’t set Credentials on the HttpWebRequest in Silverlight, the browser will prompt the user for a username and password.

private const string SPRITZER_URL = "http://stream.twitter.com/1/statuses/sample.json";

/// <summary>
/// Starts the connection to the Twitter real-time data stream.
/// </summary>
public void Connect()
{
    this.InternalConnectionStatus = ConnectionStatus.Connecting;

    try
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(SPRITZER_URL));
        request.AllowReadStreamBuffering = false;
        request.BeginGetResponse(ConnectionResponseCallback, request);
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
}

Within the callback method we ensure that we are connected and then call the ReadResponseStream method which will not return until we call Disconnect().

/// <summary>
/// Called when the initial connection has been established.
/// </summary>
private void ConnectionResponseCallback(IAsyncResult asynchronousResult)
{
    try
    {
        HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
        using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult))
        {
            if (response.StatusCode == HttpStatusCode.OK)
            {
                this.InternalConnectionStatus = ConnectionStatus.Connected;

                this.ReadResponseStream(request, response);

                if (this.InternalConnectionStatus != ConnectionStatus.Disconnecting)
                {
                    // unexpected status
                    Debug.WriteLine("unexpected connection status: " + this.InternalConnectionStatus);
                }
            }
            else
            {
                Debug.WriteLine("unexpected status code: " + response.StatusCode);
            }
        }

        this.InternalConnectionStatus = ConnectionStatus.Disconnected;
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
    }
}

In the ReadResponseStream method we continually read the stream until we are interrupted by the user setting the connection status to a value other than ConnectionStatus.Connected, by calling the Disconnect() method, or an exception is thrown whilst reading from the stream. For each read from the stream we parse the data to get the tweets from the real-time data feed in JSON format.

/// <summary>
/// Reads the information received from the Twitter real-time data stream.
/// </summary>
private void ReadResponseStream(HttpWebRequest request, HttpWebResponse response)
{
    byte[] buffer = new byte[65536];
    using (Stream stream = response.GetResponseStream())
    {
        while (this.InternalConnectionStatus == ConnectionStatus.Connected)
        {
            int read = stream.Read(buffer, 0, buffer.Length);
            UTF8Encoding encoding = new UTF8Encoding();
            string data = encoding.GetString(buffer, 0, read);
            ParseResponseChunk(data);
        }

        // need to call request.Abort or the the thread will block at the end of
        // the using block.
        request.Abort();
    }
}

The ParseResponseChunk checks the data it’s passed and ensures that the data contains at least one full status message (tweet). I’ll not go into the details of that here since it’s just a matter of string parsing.

I chose to use the JSON format simply because the content passed over the wire is smaller than the XML feed. This should mean that the application has to do less work to read all the data. What we really should also do is benchmark the deserialisation of JSON against the deserialisation of XML to see which performs best within a Silverlight application.

Deserialising the JSON

The following JavaScript JSON snipped shows an example of a single Tweet that we get back from the Twitter real-time data feed.

{
    "in_reply_to_status_id":9999999,
    "in_reply_to_user_id":00000000,
    "favorited":false,
    "in_reply_to_screen_name":"leggetter",
    "text":"@leggetter Wow! A Real-Time Rich Internet Application (RTRIA)",
    "id":2820354600,
    "created_at":"Fri Nov 4 09:39:33 +0000 2009",
    "truncated":false,
    "source":"<a href=\"http:\/\/tweetdeck.com\/\">TweetDeck<\/a>"
}

The JSON can be deserialised as an instance of a C# class using the DataContract attribute on the class and the DataMember attributes on properties.

using System;
using System.Runtime.Serialization.Json;
using System.Runtime.Serialization;
using System.ComponentModel;

namespace TweetStreamer
{
    [DataContract]
    public class StatusMessage : IStatusMessage, INotifyPropertyChanged
    {
        [DataMember(Name = "text")]
        public string Text {get;set}
    }
}

A single Tweet, or in TweetStreamer a StatusMessage, can be deserialised using an instance of the DataContractJsonSerializer.

/// <summary>
/// Creates a single message from json string.
/// </summary>
/// <param name="messageData">The message data.</param>
/// <returns></returns>
private static IStatusMessage CreateMessageFromJsonString(string messageData)
{
    Debug.WriteLine(String.Format("Creating StatusMessage for: {0}", messageData));

    IStatusMessage message = null;

    using (MemoryStream stream = new MemoryStream(UTF8Encoding.UTF8.GetBytes(messageData)))
    {
        DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(StatusMessage));
        message = ser.ReadObject(stream) as StatusMessage;
    }

    if (string.IsNullOrEmpty(message.Id))
    {
        message = null;
        Debug.WriteLine("message had no ID. Assuming to be a delete message so nulling message object");
    }

    return message;
}

Binding the data to the grid

To bind the data to a DataGrid the grid needs to be defined in XAML. In addition we can specify the properties on the StatusMessage that we want to bind to columns. In the XAML below we are binding the CreatedAtString property to a Time column, a User.ScreenName to a User column, and a Text property to a Message column. Notice the cool binding of User.ScreenName. The StatusMessage.User property returns an instance of another class and we are actually binding to a property on the returned class.

<data:DataGrid Grid.Row="0" x:Name="Tweets" AutoGenerateColumns="False">
    <data:DataGrid.Columns>
        <data:DataGridTextColumn Header="Time"
            Binding="{Binding CreatedAtString}" />
        <data:DataGridTextColumn Header="User"
            Binding="{Binding User.ScreenName}" />
        <data:DataGridTextColumn Header="Message"
            Binding="{Binding Text}" />
    </data:DataGrid.Columns>
</data:DataGrid>

Next, the code to set up the binding and registering for StatusMessage updates using the StatusMessageReceived event. As you many have noticed, the StatusMessage object implements the INotifyPropertyChanged interface. This was used so that we could add each StatusMessage to an ObservableCollection<T> and then bind it to a DataGrid to display the Tweets in real-time.

Connection _twitterConnection;
ObservableCollection<IStatusMessage> _messages = new ObservableCollection<IStatusMessage>();
public MainPage()
{
    InitializeComponent();

    Tweets.ItemsSource = _messages;

    _twitterConnection = new Connection();
    _twitterConnection.StatusMessageReceived += new Connection.OnStatusMessageReceivedEventHandler
        (twitterConnection_StatusMessageReceived);
    _twitterConnection.Connect();
}

Finally, whenever we get a StatusMessageReceived callback we need to add the new StatusMessage to the ObservableCollection collection. This has to be done on the UI thread and via the Tweets.ItemsSource property or the UI will not update.

/// <summary>
/// Status message received event handler.
/// </summary>
/// <param name="sender"></param>
/// <param name="args"></param>
void twitterConnection_StatusMessageReceived(object sender, IStatusMessageReceivedEventArgs args)
{
    Dispatcher.BeginInvoke(() =>
        AddMessage(args.Message)
    );
}

/// <summary>
/// Adds a message to the observable message list which updates the UI.
/// </summary>
/// <param name="message"></param>
void AddMessage(IStatusMessage message)
{
    ObservableCollection<IStatusMessage> messageList = ((ObservableCollection<IStatusMessage>)Tweets.ItemsSource);
    messageList.Insert(0, message);
}

Hopefully this will have helped you understand how the Twitter real-time data stream is consumed and an example of how you can use it within a Real-Time Rich Internet Application. Now, why now download the TweetStreamer library and example application and have a play.

Download

You can download the source from the TweetStreamer Google Code project.

Screencast

This screencast was supposed to be short but ended up being just shy of 10 minutes. In it I provide some technical detail of how I built the application, show the basics of how Fiddler is used to give access to the Twitter real-time data stream, and give a demo of the application.

Related posts:

1. Real-Time Rich Internet Applications (RTRIA)
2. Which Rich Internet Application Technology will dominate?
3. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request

How Silverlight makes a web request

If you were making a web request from Silverlight to download the following page:

http://www.leggetter.co.uk/2009/10/23/how-i-approach-problem-solving-in-code.html

The Silverlight runtime would first request

http://www.leggetter.co.uk/clientaccesspolicy.xml

If this request fails it will then request

http://www.leggetter.co.uk/crossdomain.xml

If both of these requests fail (return a 404 HTTP status) the Silverlight runtime will not allow you to make your web request. *

* Note: The web request failure due to a failure to download a security policy file can manifest itself in a number of ways. I’ll try to remember to put a post up about this later.

The clientaccesspolicy.xml file is Microsoft’s own security policy file. The crossdomain.xml file is used by Adobe Flash but is also supported by Silverlight. For more information see master-policy file.

Now that we’ve got the background information out of the way we can get on to the interesting stuff.

An example using the Twitter Streaming Feed

If you wanted to write your own Real-Time Rich Internet Application (RTRIA) that consumed data from the Twitter real-time data stream you’d be stuck. Well, you’ll be stuck until Twitter puts a security policy file up **. Until then you can trick the Silverlight runtime by using Fiddler to detect the security policy file request and fake a response.

** The following URLs are where Twitter would put their security policy files should they decide to allow cross domain access. Please not that you’ll be prompted for your Twitter username and password if you click on them by the browser because you need to sign in with valid twitter credentials to access the twitter stream domain: http://stream.twitter.com/clientaccesspolicy.xml and http://stream.twitter.com/crossdomain.xml)

To do this you will need to set up an AutoResponder in Fiddler that intercepts the request for the security policy file by the Silverlight runtime and returns a fake security policy file. In the example below I have an AutoResponder set up for http://stream.twitter.com/crossdomain.xml.

Using Fiddler to AutoRespond with a fake Security Policy File

Here’s an example of a crossdomain.xml security policy file which grants access to requests from all domains.

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy
  SYSTEM "http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
  <allow-access-from domain="*" />
</cross-domain-policy>

Once you’ve set up the AutoResponder start Fiddler. Then start your Silverlight application that is trying to stream data from Twitter and bingo! You should have real-time data from Twitter in your RIA – a Real-Time Rich Internet Application (RTRIA).

Fiddler HTTP Proxy Screencast

Related posts:

1. How to make a cross domain web request with SilverLight 2
2. Silverlight uses XCP tmp files for Web Requests
3. Which Rich Internet Application Technology will dominate?

I mentioned RTRIAs for the first time back in April 2009 in a post called “What is the real-time web” on the Caplin Systems Platformability blog, and since then there has been a real-time web explosion. The “real-time web” is now the buzz phrase around the Internet that everybody is jumping on. Back in April I set up a Google Alert for the term “real-time web”. Back then I got maybe one Google Alert a day for this term, probably less. Now, I get at least two dense emails a day from Google with people using the term for all sorts of things; real-time web stats, real-time analytics, real-time search, rssCloud, pubsubhubbub, the list goes on. This is not the real-time web! Whilst some of the things on this list will help  the web become truly real-time (rssCloud and pubsubhubbub may even form the back bone), none of these things give the user a truly real-time web experience.

My opinion about what the real-time web is has not changed from that first blog post. For me, it’s not just about data being available as soon as it’s been published. In this situation the user still has to actively make a request to see if any data is available. Instead it’s about making a single request for data (a subscription) and that data being pushed to your RIA to be consumed by the application user in real-time. Quite a few applications have implemented a polling solution (more on this later) but the experience presented to the user is not truly real-time.

"Real-Time?"

In my opinion the only way to guarantee being able to deliver truly real-time data (TRTD) to a web application is by maintaining a streaming connection to the source of the data using a continuous streaming HTTP connection. Most web applications use a technique called polling or long polling to retrieve their data which means that although the data is initially “fresh”, within seconds it could be out of date, or more information may be available. I’ll discuss server technologies further in a later post.

At the moment there are a number of technologies that allow you to develop RTRIAs. Ajax, Silverlight and Flash are the three mainstream solutions. The core barriers to RTRIAs are the scalability of server technology and access to truly real-time data.

Related posts:

1. Which Rich Internet Application Technology will dominate?
2. The ChristmasCrunch nearly real-time web event
3. Why Comet is key to the Truly Real-Time Web

Here’s the code. It’s obviously part of a full solution containing a Web Application and a SilverLight Application project. I’ve also uploaded it.

Useful Information:

• You can not do a cross domain request over HTTPS.
• You can do a cross domain request over HTTP.
• To be able to make the cross domain request the server that you are making the request to needs to have either (requires clarification)  crossdomain.xml and/or clientaccesspolicy.xml in the root of the webserver.

Resources/links:

The code:

private void RequestButton_Click(object sender, RoutedEventArgs e)
{
    try
    {
        // Create and being making the request/getting the response
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri("http://www.leggetter.co.uk/"));
        request.BeginGetResponse(HandleResponse, request);
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
    }
}

private void HandleResponse(IAsyncResult result)
{
    try
    {
        // Finish getting the response and then read the response in chunks
        HttpWebRequest request = (HttpWebRequest)result.AsyncState;
        HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result);

        StringBuilder readText = new StringBuilder();
        using (Stream responseStream = response.GetResponseStream())
        {
            byte[] buffer = new byte[1024];
            int read = responseStream.Read(buffer, 0, buffer.Length);
            while (read > 0)
            {
                readText.Append(Encoding.UTF8.GetString(buffer, 0, read));
                read = responseStream.Read(buffer, 0, buffer.Length);
            }
        }

        // Display the response text
        Dispatcher.BeginInvoke(delegate()
        {
            ResponseText.Text = readText.ToString();
        });
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
    }
}

Related posts:

1. Using Fiddler to trick Silverlight into allowing a crossdomain Web Request
2. How much is your domain worth?
3. Silverlight uses XCP tmp files for Web Requests