Phil Leggetter - Real-Time Web Software and Technology Evangelist » Google

Google doesn’t always get it right

Phil Leggetter — Tue, 28 Sep 2010 10:52:00 +0000

Here’s an interesting example where Google really isn’t returning the most relevant information.

Search for “real-time web” and realtime3w.com gets returned in 9th position. All they have is a slightly relevant URL.

Maybe this is actually an indicator that tells us the Google algorithm puts a massive weighting on the URL/domain.

See and download the full gallery on posterous

Permalink

| Leave a comment »

Google Adsense could easily be better

Phil Leggetter — Fri, 24 Sep 2010 08:47:38 +0000

I was just looking at my web stats and was presented with a Chrome advertisement at the top of the page. But I'm already using Chrome so what's the point in showing me an advert for it. That's just a wasted advert impression.

Google Adsense uses JavaScript to display each banner and there are a tonne of ways you can use JavaScript to detect what web browser the page is rendered in. So, why doesn't Adsense use this information when it displays an advert. An advert for Internet Explorer would have been much more valid – but wouldn't have made me switch my browser.

Permalink

| Leave a comment »

Add real-time news to your site with Google. Add any real-time data to your site with Kwwika

Phil Leggetter — Tue, 18 May 2010 09:05:00 +0000

It’s been revealed that Google are going to release a version of their API that supports Real-Time news push. Real-time push technology has been around for ages and there are a number of technologies that allow you to host your own real-time push server. However, more recently hosted solutions have come about. Kwwika is one of these solutions.

However, why restrict yourself to just news updates when there are services such a Kwwika that allow you to add real-time push containing any data to your website!

If you have data that you want to push your data in real-time then you can use Kwwika to easily and instantly distribute your data to thousands of users. This can be great for showing the quality of your data and proving you use the most up to date technologies to give your data consumers a massive advantage. If you want to add real-time data to your website then you can easily subscribe to data and have it update instantly in your website.

Why not get in touch with Kwwika or sign up for the beta programme and see how you can either real-time push your data or add real-time push to your website, Rich Internet Application (RIA) or desktop application.

Permalink

| Leave a comment »

Google have integrated Facebook fan page status messages into latest results

Phil Leggetter — Sat, 27 Feb 2010 10:02:18 +0000

According to this article by Read Write Web, Bing have exclusive access to the data created by Facebook user status updates and Google have access to the data from Facebook fan page updates. It’s interesting to see that this data is so highly valued that there is clearly a bidding war over access to certain types of data. Which is more valuable?

As of today, Facebook Fan Page status updates will begin appearing in Google search results, according to a tweet by Google. The announcement means that we will begin seeing results from the nearly 3 million fan pages, but not from the more than 400 million users.

How does Google Real-Time Search work?

Phil Leggetter — Sat, 12 Dec 2009 14:18:20 +0000

On or around the 12th of December 2009 Google released a feature they are calling “real-time search”.

Our real-time search enables you to discover breaking news the moment it’s happening, even if it’s not the popular news of the day, and even if you didn’t know about it beforehand.

Although I’d still argue that we’re not seeing the real-time web with this solution I thought I’d go in to a bit of detail about how Google are achieving this HTTP polling solution.

Summary

Google are consuming and indexing real-time data from multiple social networks
It’s not real-time HTTP PUSH yet, it’s HTTP PULL using a polling interval
The polling interval seems to vary but this needs to be confirmed.
The polling result returns an encoded JSON response containing the next request to be made and the HTML of any new results. These results are injected into the “Latest results” section of the page
The HTML returned is quite verbose and could be refined
87% from a sample of 30 polled requests returned no new results meaning the request was a waste of resources

Data Sources

Google now has an agreement in place with Twitter, Facebook, MySpace, FriendFeed, Identi.ca and Jaiku to consume and index their real-time data and make the results available in a “Latest Results” section of the Google search results. A number of social networks have APIs and some of these APIs have a real-time streaming feed and it’s likely that Google is simply connecting to this. For example, MySpace have released a Real-Time Stream API and Twitter has a Firehose feed, both of which Google will be using in their real-time search results. The Twitter Firehose feed is “not generally available” at the moment but Ryan Sarver, who is Director of Platform at Twitter, announced at Le Web that the Firehose stream would become generally available in the future.

How Google put the real-time data into their search results?

Since Google now have access to the data as soon as it is generated by the social network the next thing they need to do is get that into your search results. As soon as you make your search Google can put information that has just become available straight into the page. However, how do they get information that becomes available after your search results page has already loaded? Simple. They poll for new results regular intervals (as I mentioned in my How real-time does real-time have to be? blog post). So far I’ve seen a 20 second polling interval but recently, with a search for #leweb, I’ve seen a 30 second polling interval. It’s possible that they may have an idea how likely it is that your search subject will have new information available and change this polling interval accordingly.

These polling search requests look like this:


http://www.google.co.uk/realtimejs?q=%23leweb

&hl=en
&polltype=mb
&since=1260622409000000
&sessionstart=1260623982000000
&ei=bpgjS5zOG8j54Ab5kMzpCQ
&usg=05e5

And the responses:

mbrt0.insert('{\x22nextRequest\x22:\x22/realtimejs?q\\x3d%23leweb\\x26hl\\x3den\\x26output\\x3dsearch\\x26since\\x3d1260623734161591\\x26sessionstart\\x3d1260623385000000\\x26ei\\x3dbpgjS5zOG8j54Ab5kMzpCQ\\x26usg\\x3d3760\x22,\x22results\x22:[{\x22date\x22:\x221260623734161590\x22,\x22html\x22:\x22\\x3cli class\\x3d\\x22g s\\x22 style\\x3d\\x22line-height:16px;margin:0 0 8px;overflow:hidden\\x22\\x3e\\x3ctable class\\x3d\\x22ts\\x22\\x3e\\x3ctr\\x3e\\x3ctd colspan\\x3d3 class\\x3drt2 style\\x3d\\x22overflow:none\\x22\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-4px 0;width:6px;height:4px;float:right\\x22\\x3e\\x3c/div\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22width:4px;height:4px\\x22\\x3e\\x3c/div\\x3e\\x3ctr\\x3e\\x3ctd class\\x3drt1 style\\x3d\\x22background-position:-10px 0;background-repeat:repeat\\x22\\x3e\\x3cdiv style\\x3d\\x22width:2px\\x22\\x3e\\x3c/div\\x3e\\x3ctd style\\x3d\\x22padding:0 5px 1px;width:100%\\x22\\x3eFinally back at home after a long week in Paris. Great time at \\x3ca href\\x3d\\x22/search?q\\x3d%23LeWeb+site%3Atwitter.com\\x26amp;tbs\\x3dmbl:1\\x26amp;tbo\\x3d1\\x26amp;hl\\x3den\\x26amp;output\\x3dsearch\\x26amp;ei\\x3dcZgjS8PMNZbSjAfnl_nYBw\\x26amp;sa\\x3dX\\x26amp;oi\\x3dmicroblog_result\\x26amp;resnum\\x3d1\\x26amp;ct\\x3dresult\\x26amp;cd\\x3d1\\x26amp;ved\\x3d0CAMQoAQoADAA\\x22\\x3e#\\x3cem\\x3eLeWeb\\x3c/em\\x3e\\x3c/a\\x3e. Reconnected with lots of old friends etc,\\x3ctd class\\x3drt1 style\\x3d\\x22background-position:-14px 0;background-repeat:repeat\\x22\\x3e\\x3cdiv style\\x3d\\x22width:3px\\x22\\x3e\\x3c/div\\x3e\\x3ctr\\x3e\\x3ctd colspan\\x3d3 class\\x3drt2 style\\x3d\\x22background-position:-4px\\x22\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-4px -4px;width:6px;height:6px;float:right\\x22\\x3e\\x3c/div\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:0 -4px;width:4px;height:6px\\x22\\x3e\\x3c/div\\x3e\\x3c/table\\x3e\\x3cdiv class\\x3drt1 style\\x3d\\x22background-position:-17px 0;margin-top:-3px;margin-left:5px;padding:4px 0 0 23px\\x22\\x3e\\x3cspan class\\x3dgl\\x3e\\x3ca href\\x3d\\x22http://twitter.com/bdesarnauts\\x22 class\\x3dl onmousedown\\x3d\\x22return clk(this.href,\\x27microblog_result\\x27,\\x27\\x27,\\x27result\\x27,\\x271\\x27,\\x27\\x27,\\x270CAIQoAQwAA\\x27)\\x22\\x3ebdesarnauts\\x3c/a\\x3e\\x3c/span\\x3e - \\x3cspan class\\x3da\\x3etwitter.com\\x3c/span\\x3e - \\x3ca href\\x3d\\x22/url?q\\x3dhttp://twitter.com/bdesarnauts/status/6598672912\\x26amp;ei\\x3dcZgjS8PMNZbSjAfnl_nYBw\\x26amp;sa\\x3dX\\x26amp;oi\\x3dmicroblog_result\\x26amp;resnum\\x3d1\\x26amp;ct\\x3dresult\\x26amp;cd\\x3d2\\x26amp;ved\\x3d0CAQQoAQoADAA\\x26amp;usg\\x3dAFQjCNEyjEaOrJexPm6wUgqqVEDg24BOFQ\\x22 style\\x3d\\x22text-decoration:none\\x22\\x3e\\x3cspan class\\x3d\\x22f rtdm\\x22\\x3e4 minutes ago\\x3c/span\\x3e\\x3c/a\\x3e\\x3c/div\\x3e\x22}]}');

The response is an encoded JSON response and contains two things:

nextRequest: The next polling search request to make
results: An Array of Encoded HTML for any results that have been found for the current polling request.

In the example response above we get a next request of:

/realtimejs?
q\\x3d%23leweb\\x26hl\\x3den\\x26output\\x3dsearch\\x26since\\x3d1260623734161591\\x26sessionstart\\x3d1260623385000000\\x26ei\\x3dbpgjS5zOG8j54Ab5kMzpCQ\\x26usg\\x3d3760

Which translates to the following URL:


http://www.google.co.uk/realtimejs?q=%23leweb&hl=en&output=search&since=1260623734161591&sessionstart=1260623385000000&ei=bpgjS5zOG8j54Ab5kMzpCQ&usg=3760

From looking at the parameters it would appear that the since parameter would be very useful and would allow you to retrieve historical updates.

Each encoded HTML result in the results array will be decoded on the client using JavaScript and injected into the “Latest Results” section within the search result listings. It’s quite surprising that they return quite so much encoded HTML content and not just the information and then generate the HTML on the client. I suppose this is a new feature and it will be refined over time.

One point worth mentioning is that because Google are using a polling technique they are performing a lot of HTTP PULL requests to the server that return no results. This is a wasted request. From a sample size of 30 polled requests only 4 requests actually returned new results. This means that 87% of requests are a complete waste of resources.

Microsoft Bing.com round table thoughts

Phil Leggetter — Wed, 01 Jul 2009 19:07:02 +0000

A couple of days ago I blogged that I was going to attend a Microsoft bing round table. I’ve now attended and here are the key points and interesting facts that I took from the event. You can also search for #meetbing on twitter for relevant tweets.

Bing.com is a re-brand of Live Search

And Microsoft openly admitted this. Basically Live Search wasn’t getting any traction so Microsoft had to do something (see my next point).

“Microsoft believe they will provide as relevant search results as Google in the UK in a number of months #meetbing” @leggetter

What they’ve done is go with a “web 2.0 name” of “Bing” to try and jazz things up a little. They’ve also gone back to basics and want to concentrate on the quality and relevance of search results and believe they will provide as relevant results as Google in a few months and are judging the quality of their results on something called NDCG. I certainly didn’t use Live Search because I found the results pretty useless. Bing definitely returns better results than Live did. An example on this focus on quality is that Bing has only presently indexed around 10 Billion pages which in the scheme of things isn’t very much.

“Only 1 in 4 searches deliver a successful result. Data from MSN search, tool bar in IE. Could be searching Google, yahoo etc.” @karenblakeman

Microsoft also believe they’ve improved the UX a bit to allow users to refine their search results with the introduction of the quick tabs/suggested links in a menu on the left hand side of the user interface. They’ve done this because they believe that only 1 in 4 searches deliver a successful result so this allows users to more easily find things faster.

Microsoft seem to think the re-branding and basic search tweaks have worked as they showed us data that they believe proves Bing has overtaken Yahoo! as No.2 in the search market. Their source of data to determine where users were going to perform their web search was quite interesting. Their source was the MSN toolbar which is only available to those with Internet Explorer. I guess since IE still has a large share of the market (W3Schools Stats | Wikipedia Stats) that the figures are more or less correct but some people at the event did question this as a suitable source.

The other thing that Microsoft are doing is to integrate Bing as a “search brand” into all their products in the same way that Google integrate Google search into Gmail, Google Docs etc. So, you will now see “Bing” used to power Hotmail search and most probably all other Microsoft sites or web applications over time.

Microsoft tried to buy Yahoo! to become No.2 in search

“Microsoft tried to buy Yahoo to become No.2 in search. Fact admitted here” @leggetter

This may not be a big surprise to some people but I didn’t know that this was Microsoft’s main motivation to buy Yahoo! and I did read a few articles about the attempted purchase. Microsoft admitted that Yahoo! had the No.2 slot and if they bought them they would gain this position in addition to a well-known brand. They also admitted that they may have kept the Yahoo! brand for their search and ditched Live altogether.

Bing UK is in Beta. Bing US is not

Bing UK is lacking a lot of the functionality of Bing US since it’s still in beta. Microsoft wanted to launch Bing and felt it was better to do it world wide so stuck with the beta. Bing UK will move out of Beta when more of the functionality you can see in the US is available in the UK.

Only the standard verticals are being targeted

Local, travel, maps, videos, shopping and news will be the verticals that Microsoft initially target. This is a little disappointing as it would be great to see Microsoft break the mould and do something innovative. The only unique vertical that was mentioned is that a music search function is being worked on by the Bing China team.

All in all I thought it was a worthwhile exercise by Microsoft to tell people what they are doing and gather a number of opinions. The consensus of the people at the event is that Bing is going to have to something different to take a significant portion of people away from Google and their current offering doesn’t offer too much to do this.

I’m simply happy that Microsoft have upped their game and will at least have Google looking over their shoulders albiet way back into the distance. Hopefully this competition will provoke innovation in search that has been generally stagnent for the past few years up until the introduction of the real-time web. Maybe this could be something that Microsoft could excel at?

Click here to view all the #meetbing tweets