Our real-time search enables you to discover breaking news the moment it’s happening, even if it’s not the popular news of the day, and even if you didn’t know about it beforehand.

Although I’d still argue that we’re not seeing the real-time web with this solution I thought I’d go in to a bit of detail about how Google are achieving this HTTP polling solution.

Summary

• Google are consuming and indexing real-time data from multiple social networks
• It’s not real-time HTTP PUSH yet, it’s HTTP PULL using a polling interval
• The polling interval seems to vary but this needs to be confirmed.
• The polling result returns an encoded JSON response containing the next request to be made and the HTML of any new results. These results are injected into the “Latest results” section of the page
• The HTML returned is quite verbose and could be refined
• 87% from a sample of 30 polled requests returned no new results meaning the request was a waste of resources

Data Sources

Google now has an agreement in place with Twitter, Facebook, MySpace, FriendFeed, Identi.ca and Jaiku to consume and index their real-time data and make the results available in a “Latest Results” section of the Google search results. A number of social networks have APIs and some of these APIs have a real-time streaming feed and it’s likely that Google is simply connecting to this. For example, MySpace have released a Real-Time Stream API and Twitter has a Firehose feed, both of which Google will be using in their real-time search results. The Twitter Firehose feed is “not generally available” at the moment but Ryan Sarver, who is Director of Platform at Twitter, announced at Le Web that the Firehose stream would become generally available in the future.

How Google put the real-time data into their search results?

Since Google now have access to the data as soon as it is generated by the social network the next thing they need to do is get that into your search results. As soon as you make your search Google can put information that has just become available straight into the page. However, how do they get information that becomes available after your search results page has already loaded? Simple. They poll for new results regular intervals (as I mentioned in my How real-time does real-time have to be? blog post). So far I’ve seen a 20 second polling interval but recently, with a search for #leweb, I’ve seen a 30 second polling interval. It’s possible that they may have an idea how likely it is that your search subject will have new information available and change this polling interval accordingly.

These polling search requests look like this:

http://www.google.co.uk/realtimejs?q=%23leweb

&hl=en
&polltype=mb
&since=1260622409000000
&sessionstart=1260623982000000
&ei=bpgjS5zOG8j54Ab5kMzpCQ
&usg=05e5

And the responses:

mbrt0.insert('{\x22nextRequest\x22:\x22/realtimejs?q\\x3d%23leweb\\x26hl\\x3den\\x26output\\x3dsearch\\x26since\\x3d1260623734161591\\x26sessionstart\\x3d1260623385000000\\x26ei\\x3dbpgjS5zOG8j54Ab5kMzpCQ\\x26usg\\x3d3760\x22,\x22results\x22:[{\x22date\x22:\x221260623734161590\x22,\x22html\x22:\x22\\x3cli class\\x3d\\x22g s\\x22 style\\x3d\\x22line-height:16px;margin:0 0 8px;overflow:hidden\\x22\\x3e\\x3ctable class\\x3d\\x22ts\\x22\\x3e\\x3ctr\\x3e\\x3ctd colspan\\x3d3 class\\x3drt2 style\\x3d\\x22overflow:none\\x22\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-4px 0;width:6px;height:4px;float:right\\x22\\x3e\\x3c/div\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22width:4px;height:4px\\x22\\x3e\\x3c/div\\x3e\\x3ctr\\x3e\\x3ctd class\\x3drt1 style\\x3d\\x22background-position:-10px 0;background-repeat:repeat\\x22\\x3e\\x3cdiv style\\x3d\\x22width:2px\\x22\\x3e\\x3c/div\\x3e\\x3ctd style\\x3d\\x22padding:0 5px 1px;width:100%\\x22\\x3eFinally back at home after a long week in Paris. Great time at \\x3ca href\\x3d\\x22/search?q\\x3d%23LeWeb+site%3Atwitter.com\\x26amp;tbs\\x3dmbl:1\\x26amp;tbo\\x3d1\\x26amp;hl\\x3den\\x26amp;output\\x3dsearch\\x26amp;ei\\x3dcZgjS8PMNZbSjAfnl_nYBw\\x26amp;sa\\x3dX\\x26amp;oi\\x3dmicroblog_result\\x26amp;resnum\\x3d1\\x26amp;ct\\x3dresult\\x26amp;cd\\x3d1\\x26amp;ved\\x3d0CAMQoAQoADAA\\x22\\x3e#\\x3cem\\x3eLeWeb\\x3c/em\\x3e\\x3c/a\\x3e. Reconnected with lots of old friends etc,\\x3ctd class\\x3drt1 style\\x3d\\x22background-position:-14px 0;background-repeat:repeat\\x22\\x3e\\x3cdiv style\\x3d\\x22width:3px\\x22\\x3e\\x3c/div\\x3e\\x3ctr\\x3e\\x3ctd colspan\\x3d3 class\\x3drt2 style\\x3d\\x22background-position:-4px\\x22\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-4px -4px;width:6px;height:6px;float:right\\x22\\x3e\\x3c/div\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:0 -4px;width:4px;height:6px\\x22\\x3e\\x3c/div\\x3e\\x3c/table\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-17px 0;margin-top:-3px;margin-left:5px;padding:4px 0 0 23px\\x22\\x3e\\x3cspan class\\x3dgl\\x3e\\x3ca href\\x3d\\x22http://twitter.com/bdesarnauts\\x22 class\\x3dl onmousedown\\x3d\\x22return clk(this.href,\\x27microblog_result\\x27,\\x27\\x27,\\x27result\\x27,\\x271\\x27,\\x27\\x27,\\x270CAIQoAQwAA\\x27)\\x22\\x3ebdesarnauts\\x3c/a\\x3e\\x3c/span\\x3e - \\x3cspan class\\x3da\\x3etwitter.com\\x3c/span\\x3e - \\x3ca href\\x3d\\x22/url?q\\x3dhttp://twitter.com/bdesarnauts/status/6598672912\\x26amp;ei\\x3dcZgjS8PMNZbSjAfnl_nYBw\\x26amp;sa\\x3dX\\x26amp;oi\\x3dmicroblog_result\\x26amp;resnum\\x3d1\\x26amp;ct\\x3dresult\\x26amp;cd\\x3d2\\x26amp;ved\\x3d0CAQQoAQoADAA\\x26amp;usg\\x3dAFQjCNEyjEaOrJexPm6wUgqqVEDg24BOFQ\\x22 style\\x3d\\x22text-decoration:none\\x22\\x3e\\x3cspan class\\x3d\\x22f rtdm\\x22\\x3e4 minutes ago\\x3c/span\\x3e\\x3c/a\\x3e\\x3c/div\\x3e\x22}]}');

The response is an encoded JSON response and contains two things:

• nextRequest: The next polling search request to make
• results: An Array of Encoded HTML for any results that have been found for the current polling request.

In the example response above we get a next request of:

/realtimejs?
q\\x3d%23leweb\\x26hl\\x3den\\x26output\\x3dsearch\\x26since\\x3d1260623734161591\\x26sessionstart\\x3d1260623385000000\\x26ei\\x3dbpgjS5zOG8j54Ab5kMzpCQ\\x26usg\\x3d3760

Which translates to the following URL:

http://www.google.co.uk/realtimejs?q=%23leweb&hl=en&output=search&since=1260623734161591&sessionstart=1260623385000000&ei=bpgjS5zOG8j54Ab5kMzpCQ&usg=3760

From looking at the parameters it would appear that the since parameter would be very useful and would allow you to retrieve historical updates.

Each encoded HTML result in the results array will be decoded on the client using JavaScript and injected into the “Latest Results” section within the search result listings. It’s quite surprising that they return quite so much encoded HTML content and not just the information and then generate the HTML on the client. I suppose this is a new feature and it will be refined over time.

One point worth mentioning is that because Google are using a polling technique they are performing a lot of HTTP PULL requests to the server that return no results. This is a wasted request. From a sample size of 30 polled requests only 4 requests actually returned new results. This means that 87% of requests are a complete waste of resources.

Related posts:

1. Google realtime search isn’t real-time
2. Add real-time news to your site with Google. Add any real-time data to your site with Kwwika
3. Is It Finally the End for Real-time Search Engines?

I think the UI revolution’s already here; now we see which ones bubble to the top!

This got me thinking about the criteria that may determine which RIA technology would be most successful.

I think there are a number of things that will determine which one will ultimately win, or at least become the dominant choice. At the moment I can think of three categories; Technology buy-in, Development environment and Data access capabilities and choice.

Technology buy-in

Some companies buy-in to a technology because they have always used that technology. For example, a company that have always used Java will be more likely to opt for the Adobe solution where they can write their server components using Java and their client in Flash/ActionScript/Flex (this terminology is still a bit unclear to me. I think this is why Adobe are renaming Flex Builder to Flash Builder) but a company that have traditionally used Microsoft products will undoubtedly go for Silverlight. More traditional web developers, who probably build upon the LAMP stack, that have been hand coding PHP on the server and JavaScript on the client tend to go for creating RIAs using Ajax.

Development environment

I believe that Microsoft have a massive advantage in having Visual Studio as a server and client IDE. It has it’s faults but in general it’s a fantastic development environment. Those choosing Flex (Java -> ActionScript/Flash) will more than likely choose the Eclipse IDE, which is powerful, but it’s ActionScript/Flex development tools are lacking maturity. I’ve recently used the Flash Builder 4 beta and it’s miles behind Visual Studio in terms of coding productivity features. I’ve heard that the Flex UI component development features may be better than Visual Studio’s support for Silverlight UI. The LAMP with Ajax stack suffers much the same problems as Flex with, as far as I know, Eclipse being the IDE of choice and client-side productivity tools still very much in their infancy. It’s clearly very difficult to write an IDE to enhance JavaScript/Ajax productivity.

Data access capabilities and choice

I think this has been one of the last things to be considered in the RIA stack. Previously browsers have locked down where you can get your data from. Developers were always trying to implement hacks to allow them to do things such as make a web request to a server on a different domain or cross-site scripting to implement HTTP streaming. As the web has opened up to the idea of Mashups, browser and RIA plugin vendors have had to change their stance on cross site security. It’s a difficult line to walk but I believe that the technology that makes retrieving data from multiple sources the easiest, without having to go via a proxy, will have a distinct advantage. Flash and Silverlight are winning the race at the moment with Silverlight’s use of security policy files and Flash’s support for master policy files. For the Ajax solution I still need to research information on HTTP cross domain access control headers and which browsers have implemented support for them. I still think there will be an argument for mashing up your data on the server and I have a blog post planned on RTRIA server technologies.

The important questions around RIA data are:

• Which RIA will be more efficient at consuming data?
• Which one will prove to be the most capable as a Real-Time Rich Internet Application (RTRIA)?
• Will RIA technology vendors try to lock developers into using a client-server technology stack, such as Adobe may be trying to do with LCDS and Flex, and will this put developers off?
• Is decoupling of server and client technology, and communication protocol, important? I think it is, do you?

Although RIAs have been around for a while now, as Patrick said, the race to dominate the Rich Internet Application Technology space is on.

Related posts:

1. Real-Time Rich Internet Applications (RTRIA)
2. Government invests £200m in technology centres but no mention of Software or Information Technology
3. Marriage, Work/life integration, real-time web technology evangelism & Kwwika: A retrospective of my 2010

The phrase “real-time web” has been streaming its way around the Internet for a while now. It’s presently being used to describe information being available in search results as soon as it has been published by its author. Examples of this are Twitter or FriendFeed search.

As far as I’m aware it was Robert Scoble who made this phrase mainstream. Here’s an example from the start of February, but by this time Robert had been using the term for a while and a quick Google finds the term used as early as June 2008:

Let’s do a search for anyone who has written about the Canon 5D MK II but lets constrain that to posts that have at least one like and at least four comments. Here’s the search. Note that the post I wrote just one minute ago is already in the results page. This is the real-time web. – Robert Scoble (http://scobleizer.com/2009/02/09/is-the-real-time-web-a-threat-to-google-search/)

Whilst this is an example of something being almost instantly available from search does it really qualify as being “real-time”?

The first thing to do, as others have done, is understand what the term “real-time” means. Rather than going into this I recommend you have a read of the wikipedia definition of real-time and watch a Kaazing presentation on What is the Real-Time Web. There have been fewer attempts at defining “real-time web”. Even Wikipedia presently (19/04/2009) lacks a definition.

For me for something to be real-time web it should meet the following criteria:

1. Get up to the second information on a chosen subject.
2. Make an initial request for information to be delivered across the Internet. Subsequent updates to that information, or information stream, should be delivered as soon as possible without the need to make a additional requests.
3. Initial information and subsequent updates should be delivered in an acceptable amount of time.

The real-time web experience

If I go to my twitter homepage I don’t want to have to refresh the page, by clicking the refresh button, or hitting the F5 key, in order to see new messages from the people that I’m following. Twitter doesn’t really want users doing this either as it puts load on their server – but let’s leave this for another time.

Twitter

This lack of real-time updates on the Twitter site has lead to a surge in Twitter client applications such as TweetDeck and Thwirl. These clients simulate real-time by polling the Twitter API at intervals. I’d like to highlight the word “simulate” again and clarify what polling is. What the Twitter client applications are actually doing is making requests to Twitter at periodic intervals to see if the user has had any messages and if they have it displays them, if there are no new messages then it’s a wasted request.

TweetDeck

Twhirl

A much better model would be for Twitter to notify the client that new messages are available by passing the additional messages straight to the Twitter client.

Real-time web delivery time

What determines the amount of time that is acceptable between a message being published and it being received by the consumer (user) depends on the context of its use. For something such as instant messaging or a Twitter conversation the time between publication and consumption can be reasonably large, maybe 10 seconds. However, if the publisher is a trading system sending a price and the consumer is a trader 10 seconds is far too long. By the time the price reaches the trader it may be out of date.

Real-time technology

XMPP and SUP have been touted as the technology to deliver the real-time web but they appear to be server side solutions or require clients with knowledge of the specific technologies. For a web solution people start to turn to Comet server solutions. Caplin Systems – the real-time web company – have had the technology available to deliver real-time data for many years now so I wanted to understand why, only now, has the idea of a “real-time web” really taken off.

Caplin Systems

Caplin Systems have a high performance scalable real-time comet server called Liberator that can push updates to subscribing clients as soon as it receives information. The clients are:

• Java clients: standalone application or applet
• .NET clients such a windows desktop application or web page embedded SilverLight widget
• JavaScript clients: any website such as http://twitter.com or http://friendfeed.com

Twitter and FriendFeed don’t use Liberator but they could easily do so. All that needs to be done to integrate Real-time Rich Internet Application (RTRIA ?) functionality to an existing web site is include a JavaScript library called SL4B (StreamLink for Browsers). The RTRIA functionality can then be added to an existing site in the same way that RIA functionality can be introduced to a website using other JavaScript libraries such as JQuery. However, in this case the RIA functionality that can be added is a true real-time web experience of data being pushed to the client as soon as it becomes available, without any need for the user to check to see if any new information is available.

Conclusion

My opinion is that the instant availability of information via search straight after it’s published doesn’t really justify the term “real-time web”. To be truely real-time the information needs to be pushed to the user. The true real-time web experience and the technology to achieve this is already here but at the moment few people are aware of it and therefore it’s not in mainstream use. The ideas behind Comet, the Liberator server and Liberator clients provide technology and infrastructure to push data to clients without the need for polling or a user driven information request. Scalability and side by side integration with existing systems are already possible which means there’s no reason why existing clients such as websites can’t start taking advantage of it.

Some relevant links:

• http://www.readwriteweb.com/archives/sorry_google_you_missed_the_real_time_web.php
• http://www.gnipcentral.com/
• http://scobleizer.com/2009/02/09/is-the-real-time-web-a-threat-to-google-search/
• http://scobleizer.com/2008/12/21/rss-shows-its-age-in-real-time-web-sup-and-xmpp-to-the-rescue/
• Kaazing presentation on “What is real-time web“

Related posts:

1. Caplin Systems Blog Post – What is the real-time web?
2. XMPP PubSub or PubSubHubbub for real-time server push?