Phil Leggetter - Real-Time Web Software and Technology Evangelist » cloud

PublishMyData Helps You Set Your Data Free

Phil Leggetter — Tue, 05 Apr 2011 16:00:01 +0000

The number of “as a Service” types continues to grow and we are even seeing services that help you build your own service. PublishMyData falls into this category as it offers Infrastructure as a Service (IaaS) which enables you to offer your Data as a Service (DaaS). The company’s focus is to help those with data share it in a standard format and in an accessible way.

PublishMyData’s website sums up its offering as:

We can help you publish accessible, queryable, Linked Data on the Web so that it’s easy for people to find, understand and re-use.

This statement not only defines what PublishMyData are trying to achieve but also the point of Linked Data. Linked Data is described on Wikipedia as:

a method of publishing structured data, so that it can be interlinked and become more useful. It builds upon standard Web technologies, such as HTTP and URIs – but rather than using them to serve web pages for human readers, it extends them to share information in a way that can be read automatically by computers. This enables data from different sources to be connected and queried.

Due to the complexity involved in converting any data in any format into Linked Data the publishing process isn’t fully automated at present and there will be some consultancy involved. However, in the longer term there are plans to expose their publishing tools and move towards a much more self-service model: a kind of Content Management System for linked data driven sites.

Once the data has been analysed and processed to be more accessible and queryable as Linked Data it can then be made available on a customized portal, including a highly accessible and standardized SPARQL API (a query language for RDF), for the client. In addition it can be hosted, or at least referred to, in the PublishMyData service. If it is made available via PublishMyData it will be listed in their datasets list. Although PublishMyData will have data available directly on their site and through their hosted API it is not aiming to become a data market or data index themselves – rather an enabling platform for others who want to make their data accessible.

Although it might appear that PublishMyData is a similar service to Factual there are a number of differences. Factual are looking to make their site a destination, appear to be focusing on offering a service to host the data, defining the subject matter and get users using their API. PublishMyData in comparison are looking to offer a service that lets data owners offer their own portal where they can keep ownership of their data and control access to their API. Factual does not appear to be offering access to its data in a standardized format (i.e. something defined by the W3C) and its Factual API uses a customized query syntax, whilst PublishMyData is trying to offer access to data in a potentially more standardized format (Linked Data/RDF) and offer filtering and querying using a standard query language (SPARQL).

PublishMyData offers a nifty tool that lets you execute SPARQL queries and once you are happy with your query you can use the relevant SPARQL API endpoint to access the data from code. The API can return data in a number of formats including xml, json, text, csv and tsv. The data is also accessible following the basic HTTP-dereferencing approach that is core to Linked Data. In addition, the company is working on adding in some non-SPARQL, more ‘traditional’ HTTP API functions, to make some common functionality available without the user needing any knowledge of SPARQL which should really reduce the barrier to entry.

PublishMyData is a relatively new service and Bill Roberts of PublishMyData explains their focus for the past few months:

Our main focus over the last couple of months has been setting up some trials with data owners to publish their data. We’re aiming mainly at the UK public sector at the moment and in the process of setting up a few client-specific sites built on our platform.

More recently PublishMyData has teamed up with Aberdeen City council and a government department to publish their data. Bill explains further:

So far most of the government focus on open data and transparency has been around public spending, but these examples should move the discussion on a bit. With the council [Aberdeen City] we’re looking at how linked data can help the communication between citizens and council, and how the delivery of public services can be enhanced. With the government department, we’ll be presenting some important ’state of the nation’ statistics in a way that should be a lot more accessible than the usual government approaches.

The Aberdeen Council PublishMyData portal has just been made available and the number of DataSets will grow over time.

PublishMyData is constantly on the lookout for useful data so if you have any data that you would like to share or if you want some data but can’t get hold of it then PublishMyData would like you to get in touch. If you are interested in finding out more about Linked Data then the PublishMyData website, this post on Linked Data and JavaScript and Learn Linked Data (by PublishMyData) are good starting points.

Originally written by me for Programmable Web

NASDAQ Continues the Data as a Service Trend

Should Cloud APIs Focus on Client Libraries More Than Endpoints?

Phil Leggetter — Thu, 03 Feb 2011 08:00:47 +0000

Cloud APIs are all about the endpoints: some services follow the current trend of providing a RESTful end point, others use older protocols such as RPC or SOAP, some use newer – push focused – endpoints like WebSockets or HTTP Streaming, others may offer a number of different endpoints to meet different requirements and some just use what seems to be best for a specific job which might mean not strictly following protocol rules. But is providing an endpoint to a service alone good enough? Should a developer really have to care about how a service is built or accessed when they can use a client library?

This post was very much inspired by William Vambenepe’s blog post called Cloud APIs are like military parades which focuses on the current trend of REST APIs but it also covers some interesting questions and one very key one:

How many developers actually directly access an endpoint and how many access a service through a library?

So, should cloud services just be providing endpoints and relying on community, open source and developers to build libraries that make access to their service easier? Should they just provide samples or small helper libraries? Or should a library be provide by the cloud service vendor and be considered a key part of that service?

There may not be right or wrong answers to these question, it may very well depend on the complexity involved in using the service API.

A look at the all time most popular APIs used by mashups in our API directory unsurprisingly lists Google Maps, Flickr, YouTube, Twitter, Amazon and Facebook. The most popular mashups listing also confirms this with a heavy focus on the mashups using mapping APIs.

Mapping APIs are quite complex due to the UI aspect of the service and the majority of libraries are provided by the cloud service vendor. Twitter has developed a large array of libraries built by an “ecosystem” of developers, Facebook provides a suite of SDKs and Tools, Flickr provides API Kits, the YouTube API consists of libraries for visual-oriented functionality and raw access to a data API and finally Amazon offer SDKs for access to AWS APIs and provide code samples for accessing their product advertising one.

Big cloud API players are clearly making an effort to give developers a running start when using their APIs. In the majority of cases some raw API access is available but samples, SDKs and libraries can be found in abundance and have quite frequently been developed by the company itself.

So, do cloud services offer API endpoints just to meet the needs of a very small percentage of developers who want to make raw calls to an API? Do they do it in order to enforce good development practice? Or is the main benefit that it encourages developers and open source advocates to get involved with a service and built a community? What do you think?

Photo by Kevin Dooley

This post was originally written by me for Programmable Web.

Yahoo’s Open Sourced S4 Could be a Real-time Cloud Platform

Phil Leggetter — Fri, 31 Dec 2010 08:00:07 +0000

In a world where real-time data streams are becoming much more common, and with the volume of that data continuing to increase, it makes sense that a framework would be developed to increase the ease at which that data can be processed. Yahoo! S4 isn’t the first such framework to be concieved, or even open sourced, but it is likely to massively increase awareness that such frameworks exist, what problems they may help solve and get developers thinking about how they could use the technology and potentially increase the likelihood of somebody moving S4-like capabilities into the cloud and offering it as as service.

The requirement for a “distributed stream computing platform” came about for Yahoo! in order to be able to process thousands of search queries per second, from potentially millions of users per day, to facilitate the generation of highly personalized adverts for web search. A new framework was required because Yahoo! felt that MapReduce, which is commonly used to process large datasets in batch jobs, was “hard to apply to stream computational tasks”.

Yahoo! describe the S4 framework using a number of terms that have become common place in the world of cloud computing:

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.

Exactly what Yahoo! S4 is, and what it is capable of, has been discussed in a number of other places. The most commonly used term by comparable frameworks is Complex Event Processing with applications including filtering, correlation and pattern matching. These discussions will no doubt continue but ultimately a framework is something that can be put to multiple uses which is why Yahoo! chose to call it “general-purpose”.

Yahoo! have created a couple of examples to demonstrate some of the basic capabilities and clarify what S4 can do. One of the examples recieves data from the Twitter real-time Garden Hose stream, counts the number of times a hashtag is mentioned and keeps an ordered list of the most commonly mentioned hashtags. Each step of the process is performed in what Yahoo! are calling Processing Elements and it’s these elements that enforce the separation of each logical step of the process (e.g. recieve update, extract hashtags, count hashtags, order hastag count list) and allow the execution of the process to take place on a distributed system.

One potential thing holding S4 adoption back is that as yet it’s not offered as a service. As well as writing their own Processing Elements developers will have to host their own distributed stream computing platform. If S4 proves to be a useful and popular framework then we may start to see hosted distributed stream computing platform services in the same way that we’ve already seen MapReduce being offered as a service by Amazon.

Yahoo! S4 is yet another powerful real-time component now available to the Programmable Web. It opens up a number of possibilities for developers to start building exciting data-centric applications, mashups or hosted services which could integrate with other components such as real-time APIs, real-time client push services and DaaS services.

Originally posted on Programmable Web

NASDAQ Continues the Data as a Service Trend

Phil Leggetter — Mon, 08 Nov 2010 21:30:05 +0000

Originally post on Programmable Web

Over the past few months at PW we’ve noticed steady increase in the number of mentions of Data as a Service (DaaS) and this trend looks to continue. Our first post on the subject was by a guest author Pete Soderling who introduced us to the concept and discussed pricing models. Then we continued the theme by posting about a partnership between a stock data site and an open data platform, working together to deliver a stock historical data API. The latest big move in this area sees NASDAQ creating a Data-On-Demand service accessible through a Web API.

This is a progressive move for NASDAQ who have previously made the data available in a much less accessible and flexible format, as SearchCloudComputing writes:

The NASDAQ Data-On-Demand service will allow users to connect programmatically to NASDAQ’s massive data store instead of having to either look at it online or download giant chunks of raw data and sort it themselves.

It’s really interesting to see a large finance sector organization trust the cloud. Many banks and other financial institutions seem to be holding off from a move to the cloud. The general concensus about the reason for this is that they have a concern about cloud service security. This move by NASDAQ could potentially pave the way for other similar organizations to start considering the cloud as a viable, and secure, option. However, in this case it might have been cost saving that has been a real motivating factor.

Also from the SearchCloudComputing story:

Since NASDAQ has so much historical data, the costs to store and use the data on Amazon Web Services (AWS) can add up to hundreds or possibly thousands of dollars per month; spare change from an enterprise perspective on operating costs.

How long will it be until we start to see more financial institutions moving services to the cloud? Will the DaaS trend continue as more organizations realize the value of their data and build, or use third party services, to expose that data? And will we see a convergence with a massive trend of the past few years, the real-time web, as companies start to expose real-time data services?

Client Push Services Open Up Real-Time to Everyone

Phil Leggetter — Tue, 14 Sep 2010 04:00:58 +0000

The number of services offering real-time APIs is slowly but surely expanding and it looks like we’re going to have to add quite a few more. Since the start of the year a new type of service has started to appear–client push services, which help developers include real-time updates in their web apps.

Real-time client push APIs have actually been around for quite a while (around 10 years) as they are shipped with Comet servers but only recently have these been moved into the cloud and offered as a service. The service flavour of these APIs give the developer the ability to instantly push information from their web server, through their chosen push service and into a web browser viewing their website.

Real-time client push is intended to replace the previous pull, or polling, mechanism that has been used for many years to mimic live data on a website. Using push via a dedicated Comet server is generally more resource efficient than polling a web server, and by using a service the resource load and complexity involved in setting up and running a Comet service is completely taken away from the developer’s considerations. This means that the web server is under much less strain, the developer can concentrate on building a killer real-time application and the website user gets the benefit of a truly real-time experience.

These services have only recently started to pop up due to a number of technology advancements. To be able to use a real-time client push service the web browser needs to be able to maintain a persistent connection back to the web server so that the web server can push information to it as soon as it becomes available. This has been achievable for a number of years via what some developer have labelled as “hacks” but is now easier than it has ever been. Most of the new services use the JavaScript WebSocket object to achieve this and fallback to using Flash if WebSockets are not supported by the browser.

The web browser also needs to be able to maintain a cross domain connection from the JavaScript code running in the website to the service e.g. a connection from blog.programmableweb.com to www.example.com. In older browsers cross domain connections were not allowed but the introduction of client access policy files (crossdomain.xml and clientaccesspolicy.xml) and more recently the Access-Control-Allow-Origin HTTP header have made cross domain calls from JavaScript possible (You can find more information and a demo of this in action here).

All of the real-time client push services have adopted a data publisher subscriber model with the web server code generally acting as the data publisher and the JavaScript code running in the web browser acting as the data subscriber. Subscriptions are made to a channel (or topic), that either exists or will be created, within the service and are identified by a name e.g. “my_channel” or “/PW/CHAT”. The publishers then simply publish data to that channel or topic using a service API and the information is instantly received by all subscribers, again via an API.

The real-time client push services that we know of at the moment are:

Beacon
Hookbox
Kwwika (disclosure: author is a founder)
PubNub
Pusher
WebSync

And some examples of their use include:

A real-time news reader
A Facebook “meh” button
Interactive games
Real-time sports statistics (requires HTML5 support)

You can also check out the demos on each of the services websites.

Real-time is already a big topic and users are starting to demand data and results as fast as possible. There is also the expectation that they should be informed as soon as new data is available or the existing data changes. Google are already pressing ahead with new real-time advancements such as Google Real-Time search (which actually uses polling) and Google Instant but the good news is that with the availability of real-time client push services any developer can now add real-time to their website.

Let us know if you are interested in finding out more about these real-time client push APIs and services and we’ll cover each one in more detail.

Photo via Blake Patterson

Originally posted on Programmable Web

Using real-time web software and technology to distribute events

Phil Leggetter — Sat, 27 Feb 2010 11:08:36 +0000

I’ve just come across William Vambenepe’s blog and an article called “Waiting for events (in Cloud APIs)“ where he discusses how an event system is missing from cloud vendor APIs.

With my interest in Comet I straight away thought of this as a solution to the notification requirement and it’s interesting that the article goes on to talk about subscription management and then moves on to thinking about the delivery mechanism which are all key aspects of a good comet implementation:

How do you deliver notifications? Do you keep HTTP connections open through tricks similar to how self-updating web pages work (e.g. COMET, long polling and soon WebSockets)? Or do you just provide a listener endpoint to which the notifier tries to connect (which, in the case of public cloud deployments, means you need to have a publicly-addressable listener, but hopefully not on the same Cloud infrastructure). Do you use XMPP? AMQP? Email? Can I have you hold my events and let me come pull them?

However, when considering a notification system such as this you need to consider how frequently events would actually occur. How often would a new virtual machine in the cloud be spun up? How frequently do virtual machines unexpectedly crash? In relation to events in other scenarios such as stock rate updates or events within the Premier League between 15:00 and 17:00 on a Saturday, probably not all that frequently.

You also need to think about what would be consuming the event. Would it be an application with a user interface that prompts the user to take action or would it be an application with built in logic to handle a virtual machine failure event? If the application is running on a server then there is no reason why that server couldn’t be informed of the event via a single call to a web service, or maybe using something such as PUbSubHubbub, rather than holding open a connection to the cloud API – William describes this as a “listener endpoint”. If the application consuming the event is a web application, maybe the application is the cloud vendors VM dashboard, then Comet would be an ideal choice for instant notification.

Frequency of events and the event consuming application are two key things to consider when thinking about the best way to distribute and consume events using real-time web technologies. As an event-based API developer you may need to consider distributing your events in a number of ways to give application developers the choice.

Are there any notification systems out there that can be used by an API developer to easily distribute events in either way?

Compiling a cloud hosting vendor comparison list

Phil Leggetter — Tue, 09 Feb 2010 13:42:38 +0000

I’ve started building a list of cloud hosting vendors, including those offering platform as a service (PAAS). I’m trying to get vendors to contribute but as yet haven’t been too successful. I’ll need to directly approach them over the next few weeks to see if they’d be willing to provide me with a few details.

Update 27/02/2010:

A good friend of mine found a website that lists all cloud vendors. Although it doesn’t contain all technology specific information that I was looking for it’s quite a comprehensive list and a good starting point. Check out cloudxl.

You can still view my original list too:

View the list