Phil Leggetter - Real-Time Web Software and Technology Evangelist » Facebook

Should Cloud APIs Focus on Client Libraries More Than Endpoints?

Phil Leggetter — Thu, 03 Feb 2011 08:00:47 +0000

Cloud APIs are all about the endpoints: some services follow the current trend of providing a RESTful end point, others use older protocols such as RPC or SOAP, some use newer – push focused – endpoints like WebSockets or HTTP Streaming, others may offer a number of different endpoints to meet different requirements and some just use what seems to be best for a specific job which might mean not strictly following protocol rules. But is providing an endpoint to a service alone good enough? Should a developer really have to care about how a service is built or accessed when they can use a client library?

This post was very much inspired by William Vambenepe’s blog post called Cloud APIs are like military parades which focuses on the current trend of REST APIs but it also covers some interesting questions and one very key one:

How many developers actually directly access an endpoint and how many access a service through a library?

So, should cloud services just be providing endpoints and relying on community, open source and developers to build libraries that make access to their service easier? Should they just provide samples or small helper libraries? Or should a library be provide by the cloud service vendor and be considered a key part of that service?

There may not be right or wrong answers to these question, it may very well depend on the complexity involved in using the service API.

A look at the all time most popular APIs used by mashups in our API directory unsurprisingly lists Google Maps, Flickr, YouTube, Twitter, Amazon and Facebook. The most popular mashups listing also confirms this with a heavy focus on the mashups using mapping APIs.

Mapping APIs are quite complex due to the UI aspect of the service and the majority of libraries are provided by the cloud service vendor. Twitter has developed a large array of libraries built by an “ecosystem” of developers, Facebook provides a suite of SDKs and Tools, Flickr provides API Kits, the YouTube API consists of libraries for visual-oriented functionality and raw access to a data API and finally Amazon offer SDKs for access to AWS APIs and provide code samples for accessing their product advertising one.

Big cloud API players are clearly making an effort to give developers a running start when using their APIs. In the majority of cases some raw API access is available but samples, SDKs and libraries can be found in abundance and have quite frequently been developed by the company itself.

So, do cloud services offer API endpoints just to meet the needs of a very small percentage of developers who want to make raw calls to an API? Do they do it in order to enforce good development practice? Or is the main benefit that it encourages developers and open source advocates to get involved with a service and built a community? What do you think?

Photo by Kevin Dooley

This post was originally written by me for Programmable Web.

Browse, Build and Share Real-time Streams with DataSift

Phil Leggetter — Mon, 17 Jan 2011 08:00:20 +0000

A core feature of the real-time web is the continuously updating real-time streams of information. These streams are commonly generated by social networks and with the continued uptake of social networking the amount of information is only going to increase. This will continue to introduce opportunities for companies to create products and services that extract value from that vast amount of data. Some of the most common services built around these streams include trend and sentiment analysis, data storage, aggregation, sorting, search and filtering. DataSift is a service that offers a host of exciting features including the ability to let users browse, build and share their own real-time streams using social media data drawn from a host of sources.

DataSift launched its Alpha service at TechCrunch Disrupt in September and describe it as a “real time social media filtering engine.” The initial buzz around DataSift was generated when Twitter agreed to give them access to the Twitter firehose, but they now have access to a much wider range of data including the Google Buzz, MySpace, SixApart, WordPress, Facebook and Digg. These sources of data within DataSift, sometimes called input services, are defined as Targets in the DataSift knowledge base.

Users can use Targets to create their own streams from within the My Streams section of the DataSift dashboard using a language called FSDL (Filtered Stream Definition Language). The Web editor used to define your streams is pretty simple but it does provide some basic syntax highlighting as well as validating your syntax whenever you save. FSDL also provides access to augmentation Targets through services such as Salience, TweetMeme, Peer Index, Klout and InfoChimps that allow streams to be augmented with third party data.

Once a stream has been defined you can choose to build your feed. At present this takes up to 60 minutes and the dashboard will show you the progress of the build as well as a host of other features including a data preview, a live example of the data, a graph showing matched stream items (iterations per minute) and the history of the feed definition.

Update: Nick Halstead (CEO of DataSift) has provided a clarification about the stream preview, the live stream feature and also when the stream can be used via the API:

The stream ‘preview’ does not need to be built for anything to work, you can define + attach to API immediately (or just hit ‘live’ tab to see live results) – the preview was to allow stream owners to demonstrate what the stream would offer.

You can see Nick’s full comment in the comments section below.

DataSift is encouraging its users to build feeds that are discoverable and accessible to other users, although it does offer a private feed option, by providing a number of options on the stream page including tagging, an area to encourage you to tell others about it on common social networks and a comments area to encourage users to interact and visits to the stream page. Recently added, most commented and top rated streams are also featured on the home section of the DataSift dashboard.

Once the stream has been built it can also be used in the definition of another user stream, and it in another stream and so on. DataSift really have exposed a lot of functionality and capabilities within their user dashboard and the documentation that they provide is quite thorough and really helps a user get to grips with creating streams reasonably quickly.

All these rich features would be a waste unless there was a way of accessing the data and using it with an application. Unsurprisingly, the DataSift API delivers by providing three endpoints, including access to filtered data in a paged manner, HTTP Streaming and an RSS endpoint. It also recently introduced a WebSockets Streaming API.

Although DataSift is still in Alpha it is offering what seems an affordable and very impressive service which should excite any developer interested in real-time technologies and data. The company’s access to a wide range of data sources, engaging and intuitive user dashboard and range of API endpoints should mean that most developers will have their technology needs met.

A good starting point to learn about DataSift is an interview by Robert Scoble with Nick Halstead, the CEO of DataSift (embedded above). The video is a little old but provides a good overview and an example of creating a stream. If you’ve any comments or questions about DataSift please leave a comment here. After that you should head over to http://datasift.net and register for the DataSift Alpha program.

This post was originally written by me for ProgrammableWeb

Real-Time Web Posts 03/03/2010

Phil Leggetter — Wed, 03 Mar 2010 09:12:45 +0000

I’m doing my best to keep up to speed on any news related to the real-time web. I follow interesting people on Twitter who tweet out links, I subscribe to RSS feeds and I’ve got Google Alerts set up on a number of phrases. Since I’m doing all this work I thought it would be a good idea to start sharing it by publishing the findings in an occasional summary blog post. I generally don’t like blog posts which just say “this is an interesting article” but sometimes, if it raises awareness of good articles, it can be useful. Hopefully it will be.

I tend to first post the links on the Real-Time Web Software and Technologies LinkedIn group so if you are interested on getting the news as I find it, or contributing your own real-time web finds then please join the group. Also feel free to Tweet at me or contact me via the website if you find anything interesting and relevant.

WebSockets

5 Signs You Need HTML5 Web Sockets (Part 1) – Kaazing – Part 2, 3, 4 and 5 are also available.
WebSocket Chat – A discussion about creating a Chat application using WebSockets – Greg Wilkins
Why we don’t need HTML5 WebSocket – a discussion on the problems with WebSockets – Martin Tyler

Real-Time Web in action

The Social Media Response to Disaster in Haiti – An article on how real-time web technologies were used in response to the Haiti disaster – Amy Sample Ward, NetSquared
When were you when Tendulkar scored 200? – Ok, this isn’t actually an example of Real-Time Web in action. But it should be! This is a great example of where adding a dash of real-time web to your website could actually help you scale (stop users hitting Refresh) – Martin Tyler

Real-Time Web technology

Building Real-Time web apps with Rails 3 – LakTEK
Using real-time web software and technology to distribute events – Phil Leggetter (that’s me)
It’s not just Twitter (who use Open Source real-time web technology) - Steven J. Vaughan-Nichols
The Real-Time Web (a list of web applications that are built on or user Real-Time Web technology – or claim to) – Robert D. Hof (Business Week)
Lightstreamer 3.6 is out! – Lightstreamer release new versions of their components including support for Silverlight, Flex AMF, and Bidirectional Web Streaming

General News/Discussion

Google have integrated Facebook fan page status messages into latest results

Phil Leggetter — Sat, 27 Feb 2010 10:02:18 +0000

According to this article by Read Write Web, Bing have exclusive access to the data created by Facebook user status updates and Google have access to the data from Facebook fan page updates. It’s interesting to see that this data is so highly valued that there is clearly a bidding war over access to certain types of data. Which is more valuable?

As of today, Facebook Fan Page status updates will begin appearing in Google search results, according to a tweet by Google. The announcement means that we will begin seeing results from the nearly 3 million fan pages, but not from the more than 400 million users.