Since PubSubHubbub has been around for a while it’s relatively easy for a developer to start using a hub such as Guzzle Ayup (Ayup for short). There are a good number of PubSubHubbub publisher and subscriber clients that can be used. There is also a reasonable amount of documentation on the subject including the Google PubSubHubbub home page and Ayup’s documentation.

Ayup was created by a web agency based in Paris called Lemonchik following the requirement to provide notifications to a real-time theme-based news aggregator called Guzzle and for future web and iOS applications that will require real-time notifications. They were so happy with the final implementation that they decided to build a front-end console and offer it as a public service.

Marca Tatem of Lemonchik (and Guzzle Ayup) explains a bit more about the internals of the the service:

The internals of Ayup are fun to look at. It’s a ruby application with a Sinatra web-service, and everything is happening in high-performance, RAM only, Resque queues (backed with Redis). Ayup structure is in itself completely scalable, adding a server with hundreds of new workers is a matter of minutes.

Ayup offer a simple pay for what you use pricing policy and allow you to set a notification limit which can be handy in keeping costs down when subscribed feeds update more than expected. They also offer something called Virtual hubs which Marca explains as:

Virtual Hubs are PubSubHubbub hubs content publishers can create to push free notifications to subscribers. For example, let’s say that you want people to be able to receive instant push notifications each time you publish a story, you simply create a virtual hub ([YourHubName].ayup.us) people can subscribe to (with a subscribe request) and notifications will be sent to their http callback for free.

Marca continues to explain why Virtual Hubs are a good way of encouraging simple and free access to your content:

The difference between a virtual hub and the actual ayup’s hub is that with a virtual hub, you can only send a subscribe request for a topic that belongs to the virtual hub’s owner. In other words, if you create a virtual hub, I can’t send a subscribe request for MacRumor’s RSS feed. If I want to subscribe to many different feeds without limitations, then I’ll have to create an Ayup account and pay for sent notifications.

Ayup’s focus for the near future is to keep things simple by delivering a good experience for developers and good quality of service such as fast and constant notifications and clean ATOM.

At the time of writing there are only two hosted PubSubHubbub hub services (Hub implementations and hosted services). The first well-known service Superfeedr and now there is the welcome addition of a second in Guzzle Ayup

Originally written by me and posted on Programmable Web

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. XMPP PubSub or PubSubHubbub for real-time server push?
3. Basic Authentication against the Superfeedr HTTP PubSubHubbub API using a .NET HttpWebRequest

It is technology that has made it possible for us to produce, share and consume so much information. We are living in a time that is often referred to as an era of digital overload. It’s therefore very interesting and essential that the cause of this data overload should also be the solution: Technology, in combination with human input, can help us solve these new problems through automated curation of data.

SXSW ran a panel covering this subject: “Humans Versus Robots: Who Curates the Real-Time Web?“. The panel consisted of representatives from curation services including Henry Nothhaft from Trapit, Jim England from Keepstream, Sam Decker from Mass Relevance, Xavier Damman from Storify, and Megan McCarthy from Media Gazer who chaired the discussion. Unsurprisingly there wasn’t a definitive answer to this question. The answer really depends on the data that is being curated. It was agreed that automated curation (robots) was required in cases where there was so much data that it would not be possible for humans to possibly curate. However, the consensus was that generally once the amount of data had been reduced to a more acceptable level that it was humans who could provide the best form of curation since they can understand and apply more context and objectivity to any data. Technology will of course improve and we’ll see the robots getting smarter and smarter.

Curation cannot be solved by any one technology and it will require smooth interaction and integration between numerous technologies in order to solve this problem. APIs will play a massive part in the curation process and we’ve already seen this directly through the Cadmus API and within services such as DataSift, where they use third party services such as Klout, Peer Index and Salience for influence and sentiment analysis. DataSift provide a delivery and curation service but they can’t do everything on their own which is why they have outsourced some parts of the process to other services. As more curation techniques and algorithms are identified we’ll see these techniques exposed through additional services and APIs and in turn these APIs will be integrated into other curation services.

The amount of information being published to the real-time web is going to increase and as it does so will the importance of curation. It might not be long until it will be essential in ensuring that we don’t all suffer digital overload. How long will it be before Twitter.com and the ever increasing list of official Twitter clients, who presently only offer manual curation through selective follows and lists, join in and add curation to their service? Will they build it themselves, use a number of the available APIs (unlikely based on the amount of data) or make another acquisition?

Photo via Blake Patterson

Originally written by me for Programmable Web

Related posts:

1. Browse, Build and Share Real-time Streams with DataSift
2. Why client APIs are an important part of any real-time service
3. The benefits of Real-Time Push-Once

Essential

• C#
• jQuery
• TDD/BDD
• ASP.NET MVC
• ADO.NET Entity Framework or other ORM

Great to have

• Passionate about the real-time web and real-time data (essential for the longer term)
• Knowledge of building scalable systems on Amazon Infrastructure
• Experience of integrating with 3rd party payment systems and building basic ecommerce functionality
• Experience of building a SaaS or IaaS offering
• Interest/experience in API and Web Service development
• Advanced JavaScript
• CSS
• Excited about HTML5
• Part of a tech community

Location

The role will be primarily home-based but at times we may need to meet around 2 to 3 times a week at a location that we agree on anywhere between or around Dundee and Edinburgh.

Moving forward I’m hopeful that this could develop into a fulltime role so it would be ideal if we could get somebody who was interested in this also.

If you are interested please get in touch with your availability and your rate.

Related posts:

1. Drag Drop bug in JQuery UI 1.5.3
2. Real-Time Rich Internet Applications (RTRIA)
3. The Real-Time Web: TechMeetup Aberdeen – 19/01/2011

A big thanks to @aboynejames for suggesting I do the talk and Bruce Scharlau for inviting me along. I’d also like to thank everybody who attended and made me feel very welcome and certainly seemed interested.

Feedback

If you where there then please leave a comment letting me know what you thought or just to let me know who you are. If you’d rather email you can get me using phil@leggetter.co.uk.

Video

Here’s a video of the first 10 minutes of the talk. Unfortunately the video recorder ran out of batteries.

Presentation and Notes

Unfortunately I can’t find a way of showing my slide notes in the above embeded Google Presentation so here they are in pretty rough form.

What is Real-Time?

Respond or react within a predictable time after an event has occurred.

There may also be a maximum time constraint.

Because after that maximum time the information loses value and context.

Maybe we should really call it:
Right-Time

When does Real-Time matter?

When does NOW matter?

When do you think being informed of something the instant it occurs really matters?

• Critical systems.
• Live events where things lose context.
• Systems requiring synchronisation:
  The 2Screen experience e.g. TellyLinks.com
• Interactive/collaborative things e.g. chat, Google Wave, Google Docs (docs, spreadsheets, drawing)
• Sensor data e.g. Car ABS breaks
• Trading: You’ve got to be sure you are seeing the correct trade price in order to be able to make a decision on whether to trade or not.
• Betting: You want to make sure you bet at the right odds and you don’t want clients to be able to bed when the odds have changed.

Evolution of the Web

From Pull to Push

Although I’m going to use the Web Browser as the example client the client could of course be anything. Another web server any desktop application, a mobile application, an embedded app on any device.

I’m also going to focus on HTTP but the mechanisms could be applied to standard TCP socket connections.

HTTP Pull (Request, Response)

Note: client. This can be a web browser but doesn’t have to be.

1) Each request would result in a page reload.
2) Event occurs on server there’s no way of the server telling the browser.
3) e.g. chat pages where you would need to click the “Refresh” button to see what other users are saying

HTTP Polling

A user hitting F5 or clicking the refresh button is one manual example.

1) REFRESH-META
2) Then frames
3) XMLHttpRequest -> Outlook web access (ActiveX object)

Check if the information has changed by polling.

The problem with this is you can easily waste requests and it can be resource intensive for both the client and the server.

A large amount of services still take this approach although the web, and the technologies, have evolved to allow for real-time push.

For quite some time I’ve said if a service uses HTTP Polling when push technology is available and a much better option that it’s not real-time

HTTP Push

In this diagram you could replace the Client with Server. The initialization could be classed as a subscription or webhook.

Various techniques have been developed to hold a connection open to a web server from a browser.

Difficulties :

* Proxies/Firewall
* Connection reselience
* Browsers
* Messaging API

Related posts:

1. Delivery as a Service and Data as a Service in 2011
2. XMPP PubSub or PubSubHubbub for real-time server push?
3. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub

DataSift launched its Alpha service at TechCrunch Disrupt in September and describe it as a “real time social media filtering engine.” The initial buzz around DataSift was generated when Twitter agreed to give them access to the Twitter firehose, but they now have access to a much wider range of data including the Google Buzz, MySpace, SixApart, WordPress, Facebook and Digg. These sources of data within DataSift, sometimes called input services, are defined as Targets in the DataSift knowledge base.

Users can use Targets to create their own streams from within the My Streams section of the DataSift dashboard using a language called FSDL (Filtered Stream Definition Language). The Web editor used to define your streams is pretty simple but it does provide some basic syntax highlighting as well as validating your syntax whenever you save. FSDL also provides access to augmentation Targets through services such as Salience, TweetMeme, Peer Index, Klout and InfoChimps that allow streams to be augmented with third party data.

Once a stream has been defined you can choose to build your feed. At present this takes up to 60 minutes and the dashboard will show you the progress of the build as well as a host of other features including a data preview, a live example of the data, a graph showing matched stream items (iterations per minute) and the history of the feed definition.

Update: Nick Halstead (CEO of DataSift) has provided a clarification about the stream preview, the live stream feature and also when the stream can be used via the API:

The stream ‘preview’ does not need to be built for anything to work, you can define + attach to API immediately (or just hit ‘live’ tab to see live results) – the preview was to allow stream owners to demonstrate what the stream would offer.

You can see Nick’s full comment in the comments section below.

DataSift is encouraging its users to build feeds that are discoverable and accessible to other users, although it does offer a private feed option, by providing a number of options on the stream page including tagging, an area to encourage you to tell others about it on common social networks and a comments area to encourage users to interact and visits to the stream page. Recently added, most commented and top rated streams are also featured on the home section of the DataSift dashboard.

Once the stream has been built it can also be used in the definition of another user stream, and it in another stream and so on. DataSift really have exposed a lot of functionality and capabilities within their user dashboard and the documentation that they provide is quite thorough and really helps a user get to grips with creating streams reasonably quickly.

All these rich features would be a waste unless there was a way of accessing the data and using it with an application. Unsurprisingly, the DataSift API delivers by providing three endpoints, including access to filtered data in a paged manner, HTTP Streaming and an RSS endpoint. It also recently introduced a WebSockets Streaming API.

Although DataSift is still in Alpha it is offering what seems an affordable and very impressive service which should excite any developer interested in real-time technologies and data. The company’s access to a wide range of data sources, engaging and intuitive user dashboard and range of API endpoints should mean that most developers will have their technology needs met.

A good starting point to learn about DataSift is an interview by Robert Scoble with Nick Halstead, the CEO of DataSift (embedded above). The video is a little old but provides a good overview and an example of creating a stream. If you’ve any comments or questions about DataSift please leave a comment here. After that you should head over to http://datasift.net and register for the DataSift Alpha program.

This post was originally written by me for ProgrammableWeb

Related posts:

1. Who Curates the Real-Time Web?
2. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
3. Why client APIs are an important part of any real-time service

As 2011 continues we might to see some of these companies either partner up or start to tread on each others toes as they look at delivering data to alternative platforms and clients. Ultimately we are going to see services offering real-time communication of any type to any web enabled device using the web as the network.

The following lists aren’t 100% comprehensive and as time goes by I’ll probably create some pages on my site dedicated to this information.

Here’s the list of services I’m aware of and the type of service they offer and technology they use.

• BeaconPush - [Web browser]
• Google Channel API (Python/Java) – Part of Google App Engine – [Web browser]
• DataSift – [Server to server, WebSockets, HTTP Streaming]
• Kwwika – [Web browser, mobile, server to server, Desktop]
• Hookbox – [Web browser]
• Notifo – [Mobile]
• Notify.io – [Desktop, IM, Email]
• Notify.me – [Mobile, Desktop, IM, Email, XMPP]
• PubNub – [Web browser, mobile, server to server, Desktop]
• Pusher – [Web browser]
• Superfeedr – [Server to server, PubSubHubbub, WebSockets, HTTP Streaming]
• Twilio – [Mobile]
• Urban Airship – [Moblie]

A quick not of the key technologies behind these types of service:

• Comet (Note: this version by Jacobolus (Jacob Rus) is by far the best description. The current wikipedia description, IMHO, is inaccurate and incomplete and was ruined by Damiens.rf. Damiens.rf even put Greg Wilkins, who contributes to Jetty and Cometd, off contributing to Wikipedia all together.)
• Growl
• HTTP Streaming
• PubSubHubbub
• XMPP
• Webhooks
• WebSockets

Note: I originally wrote this as a comment over on Anil Dash’s post on Delivery as a Service but I thought it was good content so it deserved a post of its own.

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. The Real-Time Web: TechMeetup Aberdeen – 19/01/2011
3. Why client APIs are an important part of any real-time service

So, six days into 2011 I’m writing this blog post. It’s purpose is all of the above but it will hopefully also act as a reminder to me about why I made some decisions. Why I’m here, and not there. What it is that drives me and what I want. And how my decisions were made after consulting and considering others. I also hope that it will explain a bit about work/life integration by demonstrating the problems that I’ve faced and that things can be achieved (ongoing) and opportunities created by following what you are passionate about (I’m midway through writing and reviewing this and there are definitely ups and downs).

The blog post is a personal post but since work is a big part of what I do, and I’ve made a decision to accept that rather than fight it (work life integration v work life balance), it’s also relevant to what I do to make a living. If you are truly passionate about it can you really drop it at 18:00? Can I?

January: The Work/Life Integration decision

I’ve been very aware that the standard 9 to 6 doesn’t suit me. Heck, it probably doesn’t suit a lot of people and most definitely doesn’t mean you work when you are most likely to do your best work. How many developers work in to the night because they are “in the zone”? If you are in the office in the middle of London and your partner is waiting for you to get home from work it’s highly unlikely that you are going to hang around until 2am. Especially when it takes over an hour to get home and the trains stop before midnight.

How many of us enjoy parts of our job but not all of it? How many only like a small portion or none at all, we just need it to pay the bills? I’ve had a nagging feeling about this for quite some time. In quite few of my jobs I’ve absolutely loved the work but disliked some of it. In others It’s just got to the point where the dislike overpowered the enjoyable moments for far too long so I left. The thing is I would go home and then start thinking, solving problems, generating ideas and writing code again. I’d leave my job where I’d be writing code to go home and write code! I couldn’t help but think there must be a better way.

Then in mid 2009 I read a couple of books by Ian Sanders (LEAP!: Ditch Your Job, Start Your Own Business & Set Yourself Free and Juggle! Rethink Work, Reclaim your Life). Juggle! in particular made me realise that I was already juggling a 9 to 6 job along with a few personal projects, some general dev consultancy as well as helping develop, and owning shares in, an e-commerce website. It made me think about what it was I was passionate about, what I enjoyed in my current job and what I would ultimately like to do and how I would like to achieve it.

Then in December 2009 I picked up a copy of Gary Vaynerchuk’s Crush It! Why Now is the Time to Cash in on your passion as part of going to an event. There were a number of reasons why I liked this book and why it hit home. It wasn’t the title nor was it that Gary signed it “It’s real-time time!” – although that helped. Firstly it said to me that if you are passionate about something and, very importantly, good at it then you can make a living from it. From this point on it was all about building up the courage and belief that I could make something work. That my passion and expertise for the Real-Time Web could really help me achieve true work/life integration. That what I would do when I went home after my job is what I actually did as my job!

My plan was to try and build a reputation as a “Real-Time Web Software and Technology Consultant” (Evangelist) by blogging about real-time web technology, building cool real-time demos, playing with real-time APIs such as the Twitter Streaming API and generally enjoying myself whilst generating loads of really interesting, cool and useful content for others. The belief is that by doing this, and building your “personal brand” (sorry to anybody who doesn’t like this term) that opportunities will present themself and you will also be able to react to opportunties that you see. And these opportunities will be related to the thing that you are passionate about. In my case this is real-time web software and technology and social media (hmmm, The Word Media is wrong). Along with the work I love I was aware that I would have to take on other work that would help pay the bills but hopefully the amount of this type of work would decrease as I became successful.

Just to be very clear: work/life integration is different from work/life balance. Work/life balance tends to be associated with having a defined work time, keeping your weekends free and not thinking about work when you walk out of the office or turn off your computer. I’m not looking for that. I love to work as long as I’m working on the things I love.

After discussing things with my family and friends I finally made the decision to choose work/life integration. This decision meant leaving Caplin Systems, for the second time, after they had taken me back just over two years earlier. It also meant moving back up to Scotland to be closer to both mine and Jo, my girlfriends, family. Since we were getting married in May 2010 and had hopes to start a family this was also a big factor in the decision.

So, it was time to have a chat with my bosses.

February: An unexpected opportunity

I spoke to my bosses and explained what I was planning to do and why. I also started discussing the possibilities of still being able to use Caplin Systems excellent technology of which I had a very good understanding. This is where things got interesting. I remember in one conversation begin told “if you are going to do that [use real-time web technology] we’d like to help you do it properly and support you”. This was fantastic for a number of reasons. Firstly I could still use the technology, secondly it meant I had made a positive impact and was reasonably highly thought of, thirdly I would potentially have some kind of salary and finally it meant I had potentially created an opportunity to work on something I was truly passionate about. I had almost created my own role, exactly the sort of thing Ian Sanders talks about in Juggle!

After a number of meetings, a presentation and quite a bit of research we came up with the idea of putting Caplin System’s software “in the cloud” and offering it as a service for others to use (it took months to eventually come up with the name “Kwwika”). This wasn’t the first time that “Caplin in the cloud” has been discussed but was an interesting way of taking the idea forward. During what would have been my notice period at Caplin I finished off some core work, handed over my development line manager duties and started to investigate putting “Caplin in the cloud”. With a very small amount of effort we could clearly see that Liberator (Caplin’s real-time streaming server) performed excellently running on only a small Amazon Linux instance. We also tested clustering of instances, sharing data, publishing using what would previously be classed as our client APIs and it all worked very well. The unnamed project was given the go-ahead.

I got in touch with Ian and we met up for a coffee.

March & April: Moving back to Scotland and getting started with Kwwika

In March 2010 we moved back to Scotland where I would be working from home in a flat in Broughty Ferry, near Dundee.

The time between March and May is unsurprisingly a bit of a blur. Work on Kwwika continued and lots was achieved including:

• A Kwwika website
• A Kwwika blog
• A Kwwika JavaScript API
• A Kwwika .NET API
• A Kwwika Silverlight API
• Quite a few videos showing off Kwwika
• An impressive Kwwika real-time demo for the UK Election leaders debate

But most importantly for Kwwika it was becoming very clear that there was a market for real-time push services with the general interest that had been shown and the emergence of quite a few possible competitors including Beacon Push, Pusher and Web Sync on-demand. Although Kwwika is potentially a lot more than this it was a really encouraging sign.

Since all that had been done it’s probably clear that the work/life integration had probably been very dominated by work. This had always been expected and Jo and I had accepted this because was a fantastic opportunity for me and I really felt I needed to put in as much time as possible to prove Kwwika as an idea. I’m not complaining. A lot of the work I was doing, building new APIs, building a brand for Kwwika, searching for potential uses for the technology and interacting with people who were interested in this sort of technology is what I wanted to do. It was a bit unfair on Jo though.

May: Let’s get Married!

Almost exactly one year after I proposed to Jo we got married. Because my focus was mainly on work Jo had done almost everything to organise the wedding. You’ll be pleased to hear that I didn’t check my email on my wedding day but unfortunately up until the day before and every day after I would be checking my email and replying to potential clients as well as ensuring the Kwwika service and demos were running as expected. Again, I didn’t mind too much. The work was great but I was very aware my priorities were a bit off centre.

On the wedding day itself I was completely emmersed in the occasion and loved everything about the day from the ceremony to spending time with my beautiful wife, my friends and family. A big thanks to Jo’s parents who hosted the event in their fantastic home, my new brothers and sisters in law, my new niece and nephew, my Mum, Dad and Granny for all their support and a MASSIVE thanks to Jo for saying “Yes”.

Unfortunately on every other day I was in a mindset where I was always thinking about Kwwika, did I have an email I should read and reply to, was the service running okay, were all the demo applications up and running. I don’t regret anything up until this point but I do regret not taking more time off  to spend with my wife and try to relax a bit more around our wedding day.  Jumping ahead a little, this is also something that I regret about Christmas and New Year – although I wasn’t nearly as bad I was always aware that I wasn’t 100% relaxed.

In 2011 this will have to change: my mindset, not the married part!

At the start of May I also delivered a Real-Time Web Workshop at the SPA 2010 conference with Adam Iley from Caplin Systems. It went fantastically and we got some excellent feedback putting our session 3rd for feedback scores in the entire conference. We really must do this workshop again.

June, July and the World Cup

The World Cup was always going to be a big part of the year. Since John Barnes retired (some might say I was obsessed) my interest in league football has waned and my real focus has been England international football. As ever my hopes were high and I was massively let down by the England team along with the rest of a hopeful nation (yes, I’m an Englishman living in Scotland).

Just prior to the World Cup I had an idea for a competition for Kwwika. I’d write an app that recieved all the World Cup Tweets from the Twitter Streaming API and push them through Kwwika. I’d then ask developers to do their best to subscribe to this data in Kwwika and use it to build a cool real-time web application and the best entry would win an Apple iPad (I’d heard they were all the rage). The competition started off a little slow but we eventually got a number of entries, some great publicity, and we decided to award two developers, Aaron Bassett and Gergely Orosz, an iPad. You can read the winners announcment blog post here.

Other Kwwika achievements during this time were:

• Creating a Kwwika real-time push plugin
• Getting Opta Sports to provide some real-time World Cup data for the Kwwika World Cup competition
• A bit of press coverage for Kwwika StartupCafe and Innovation Focus Scotland (actual article now offline)

A big positive in June was going to Poland for Keith and Ewa’s wedding. This was probably the first time that I felt work/life integration really worked. I got up in the morning and did some work and then spent a lot of the time with friends and enjoying Krakow. I was on holiday but I didn’t mind doing a bit of work.

Dexter

Jo and I have always loved the thought of having a dog and it wasn’t until I started working from home that this was possible. We didn’t feel it would be fair to work full time and leave a dog home alone. Jo was still looking for work after the move and the wedding so it gave us the opportunity for her to be able to look after a dog for a while almost fulltime – Jo was absolutely knackered every single day.

On June 29th 2010 we picked up Dexter. Anybody that follows me on Twitter will know about Dexter (@Dexify on Twitter and Dexter’s facebook page). Dexter most definitely deserves a section all to himself.

To say that Dexter was a handful is an understatement. I can’t say things have got much easier with him although he does’t wee all over the place he still requires a lot of attention but he does bring a lot of joy into our lives and guarantees a good few laughs every single day.

August – November: Continued focus on work

Things were strange between August and mid-November. I spent the majority of the time working but had very little visible to show for it. Under the hood of Kwwika we made really big strides with the infrastructure and we also created an admin web application that allowed us to set up users, API Keys, Groups, Topics and Permissions (the need for all this stuff will become clear this year). I also stopped blogging quite so much about Kwwika because it had become very clear that the lack of a user dashboard was a massive barrier to entry for users so my efforts to drive beta users to sign up wasn’t leading to many real opportunities.

The only real visible things during this time were:

• Built a Kwwika-Superfeedr demo showing how to stream real-time updates from Superfeedr into a web browser using Kwwika with the code available in GitHub. Unfortunately I’ve taken the demo down as we’ve ran out of Superfeedr credits but the video that’s embedded in the blog post shows the application in action.
• Aaron Bassett using Kwwika on The Meet 140

The work/life integration focus again was mainly on work with very few social activities going on and a lot of time spent sitting in front of a computer. The main problem here is that the work had moved away from interacting with others, away from evangelising about technology, away from adding value to APIs and over to what I’d class as bog stardard web development. I was struggling.

I did have a fortnightly trip down to London and a couple of trips out to see insitebright and also Matador Digital but other than that I spent a lot of time working home alone.

On top of the Kwwika work I also started doing some freelance writing for Programmable Web and did a small consultancy project with insitebright for a large oil company. This actually provided a nice focus away from Kwwika so when I came back to it I was fresh and I was much more willing and able to tackle the bits I was less keen to.

In August we went to Gigha which is a small Isle off the west coast of Scotland. The weather was fantastic. It was the first time that I felt work really getting to me and I suffered my first ever migraine. That wasn’t fun and acted as a sign that I was working way too hard.

Jo started work in September and although I had Dexter to keep me company I found working home alone very lonely. This is something that I’m still trying to find a solution for but think that social media can be part of the answer along with increased transparency of Kwwika. I’m also very aware that if Kwwika is a success it will mean a lot more client interaction and technology evangelism which I’m really looking forward to.

In October my very good friend Mark married Lynne. I’ve been friends with Mark ever since I moved to Glenrothes in 1993. It was a fantastic day and the couple and the kids were really happy. Unfortunately I don’t have any photos. I do have a very large and empty bottle of Jack Daniels though.

December: The culmination of 9 months effort and relax (almost & temporarily)

When I first started Kwwika I got in touch with Opta Sports. One of the reasons I decided to leave Caplin (although, as you know, I haven’t) was their financial focus. I would have preferred to see the technology applied to something I have more interest in and I’m interested in sport so this was a great first target. We started work on the Real-Time Opta Sports Cricket widget – powered by Kwwika in November and it was ready during December. I’m really excited to see Opta Sport’s data streaming through Kwwika and I’m really hopefully that cricket data is just the start. I was particularly proud when Opta put the Kwwika logo on their homepage. And because Opta then released the Opta Annual 2010 we are also in the brochure. An awesome feeling!

The final part of the year was spent preparing to build the Kwwika User Dashboard. We’ve outsourced some of the development to insitebright which means the work I’ve got to do is design the UX and plumb the delivered HTML, CSS, JS and images into the server-side code (it’ll probably be a bit more complicated than that but you get the idea). Again, this certainly isn’t the sort of thing that I really want to be doing but I absolutely understand that it’s essential to be able to deliver Kwwika as a service and break down the barrier to entry for users. As soon as I’ve finished this post I’ll be writing a post on the Kwwika blog about that.

I managed to relax a bit in the run up to Christmas and probably over compensated by trying to relax too much (i.e. just sat and watched TV). I now know I’d have been better doing something a bit more active as I feel I’ve done very little over the festive period and that just sitting still in front of terrible TV isn’t relaxing at all. I’m actually more relaxed now. All this said I’ve always had the nagging feeling that I’ve a lot to do and prove with Kwwika which hasn’t helped the attempts to relax. One thing that did help me relax was that after declaring that I wanted one I finally got a “What are you looking at Dicknose?” t-shirt on Christmas day. Thanks Jo!

Start, Stop, Continue: The retrospective

Since this is a retrospective I have to state what I now think I really should be doing (Start), what didn’t work and I should Stop doing and what I think worked and I should Continue doing.

Start:

• Spending more time with family and friends and generally socialising. I’ve had my first real experience of trying work/life integration and I need to make sure life gets a bit more of a look-in.
• Increase social media use: During lonely spells try and use social media to ensure I’m getting more interation. In Particular I’m really enjoying using Quora at the moment. Some of my friends already ask me why I share so much online and already think I use social media far too much. It’s a good question really especially when my tweets about Dexter probably don’t mix too well with my real-time web tweets. My answer is I come as a package. I’m going to be true to myself and whilst I do apply some thought to what I publish about myself and to the content of my publishing I’m going to continue this.
• Being more transparent: My friends would again point me to the last bullet point but in this case I mean with Kwwika in order to get feedback as early as possible and to try and get others to help me when I’m struggling to get feedback from elsewhere. My next Kwwika blog post is a move in this direction.
• Reading more: When I worked at Caplin I found I had time to read more. Reading stimulates the mind and really helps ideas grow. Although I’ve had moments like this over the past 9 months I feel my general absolute focus on Kwwika and the time it’s taken up has actually resulted in a bit of a stagnation of ideas. I must find time to read books and more blog posts.
• Attending more tech and entrepreneur events: When I moved to Scotland I had planned to go to a lot of tech events in order to meet people, network and to generally get out and socialise with like-minded geeks. I’ve been to what seems like quite a few events but probably not too many given that this covers 9 months: 1 Scottish Lean Circle, 1 Edinburgh TechMeetup, 1 Linked Data, 1 DeveloperDeveloperDeveloper NoSQL day, the SPA 2010 conference, 1 Making the Most of Cloud event, 1 Edinburgh Web Design and Development Meetup and 1 Edinburgh OpenCoffee Club morning.
• Planning ahead a bit more: I’m not talking about the five year plan. I just mean general planning for the month ahead. I’ve got a roadmap for Kwwika, and plan each day either the night before or first thing in the morning, so I also need a roadmap for social and life activity. This will better help me integrate my work and my life.
• Exercise again: Between mid-2009 and May 2010 I lost 12 kg for my wedding. I did it by using our Water Rower and by running. I’ve put at least half of that back on. This time I’ve got a running companion – Dexter!

Continue:

• My customer focus: I’ve spent a lot of time replying to emails for beta registration over the past 9 months. I want to make sure that each registration is treated as an individual to make sure that they get the most out of Kwwika. I strongly believe in providing exceptional customer service and that ultimately it will make a massive difference. Small town rules! And I really do care.
• Working hard: As well of moments of working exceptionally hard I’ve experienced moments of what can only be described as burn-out. A week or two where I can’t seem to apply the effort required to perform some of the tasks that, in and ideal world, I wouldn’t have to do. I want to continue to work hard but I want to make sure that I do it consitently and the only way to do that is to rein myself in sometimes.
• Finding people and companies to partner with: I’ve established a good relationship with Opta Sports, insitebright and Matador Digital as well as a few individual developers. It’s important that I continue to do that both for Kwwika my own personal development.
• Writing two blog posts a week for Programmable Web: I’m really enjoying writing these blog posts. They get a big audience, it’s good for my “personal brand” (there we go again) and it’s a great way of discovering new technologies and also trying them out.
• To be myself: It’s all part of transparency and customer focus. I’m not the sort of person to play games and wheel n’ deal. Maybe this is a problem in business but I’m going to do my best to make sure I don’t give in to any kind of business rules if it goes against how I feel, who I am and what I believe in. I want people to believe and trust in me because I’ve proven myself to be a nice guy, fair, knowledgeable and unbiased and therefore the when I say or write something it has some weight and proven foundation. That way they will believe that when I say Kwwika is a great product they will give it a go and see that it is.

Stop:

• Spending so much time in front of a computer: It causes mind stagnation. I remember when I first began working at Caplin a senior developer (only link I could find) telling me to take breaks. He smoked and would disappear for a cigarette but quite frequently that break would lead to a fresh way of looking at a problem. I need to take this idea and apply it to my work in general (no, not take up smoking). The family, friends and social activities are all part of a package that can actually help deliver my work. And the other way around, if I’m working on something I enjoy I will be happier with my friends and family. But here I am sitting in front of the computer at 2am – yeah, but I’m “in the zone”!
• Worrying about work when I have dedicated some time to life: Speaks for itself really. I need to have the confidence that things won’t fall over when I’m not there. I actually know things are stable but I still worry. Maybe I just need a few additional processes in place. Actually I know I need and want somebody to work with me fulltime on Kwwika.
• Doing things that I’m either not good at or am simply not interested in: I want to move away from what I’ve already called bog-standard web
  development. I’m not a website builder. I don’t have the passion for it
  and I never will. There are parts of building a website that are interesting but once the problems have been solved I lose interest. This is a bit of an ideal world scenario. But it’s most definitely a goal. As already stated I love evangelising about real-time web technology, I love interacting with clients, sharing knowledge, helping and caring. I also enjoy concieving ideas and helping to see them realised. I love data, considering it’s structure, working with APIs and developing new features.

Conclusion – finally

So, what is it I really want?

Family will always come first but after that I want work/life integration and to feel that work is benefiting from life and life from work. I want to spend time with my family and friends. I want to be working on the things that I’m passionate about and helping others that feel the same way and helping those that don’t understand see the light (technology evangelism). I want to work with others whether it’s colleagues, partners or clients. I want to concerntrate on the things that I’m good at and that I really enjoy: That’s real-time web software and technology evanglism, concieving and realising how ideas can be achieved using this technology, deliverying customer service using social media by giving a damn (caring), sharing knowledge through blogging (and maybe a bit of vlogging), events and social media and working with others to create exciting products, services and solutions. I want to be happy and I believe this will “make it so”.

Related posts:

1. Kwwika World Cup 2010 Real-Time Push Web App – Apple iPad competition
2. Book review: Juggle! Rethink work, reclaim your life
3. Kwwika Powered Real-Time Opta Sports Cricket Widget

The conversation continued:

The main thing here for me is Nick’s point that “you just need a client-side library“. This is an assumption that a lot of services make. Services tend to offer one or a number of server APIs; maybe a REST one and now luckily we are seeing a HTTP Streaming and WebSocket support. That service may then release some example code that can be used to call the web service. I see a number of problems with this mainly when it comes to HTTP Streaming and WebSockets.

• DataSift WebSocket Streaming Docs
• DataSift HTTP Streaming Docs

WebSockets aren’t as yet supported in enough browsers. Therefore it’s not viable to purely rely on WebSocket support in any production application. So, you can’t build a web application unless there is a fallback alternative. HTTP Streaming in a web browser could be a fallback (XmlHttpReqeust, XDomainRequest or Flash connections) if the server provides the necessary Access-Control-Allow-Origin response header, the browser is new enough, or if the user has Flash installed the server serves up a crossdomain.xml file and the developer is willing to also write code to deal with all the browser types and technoloy scenarios. Then you need to deal with authentication from a web browser for some services to let you access their HTTP Streaming API. DataSift will class your connection as a 2nd class client if you don’t authenticate:

Authorization is not required on the streaming API. However, should you chose not to authenticate, you may be subject to stricter rate limiting and other disadvantages over an authorized user. Therefore, authorization is strongly recommended

What this means is that DataSift as yet don’t support real-time updates in web browser clients fully.

Then you need to consider other technologies like .NET, Java, C or C++ desktop applications, mobile applications on iPhone, iPad, Windows Mobile 7 or Palm using webOS. Each of these technologies requires a client API.

• DataSift API Client Libraries

A Scenario

Say I were building a new service which focused on real-time data and delivery of that data in our custom application to desktop app, mobile apps and web apps. My service is focused on curating real-time data from different sources. My first port of call would be DataSift or Superfeedr (I’m going to leave Kwwika out of the equation here). I know these guys can deal with large volumes of data and can provide some filtering. Maybe if filtering was absolutely key I would go for DataSift (maybe Julien would tell me off for that).

Now I’ve made my choice and I’ve got all the data I want streaming through my chosen service and I start to build my client applications. If I choose to build my web application first I’m stumped as I’ve already explained, but let’s say I want to build a Windows desktop application first. Simple, I use the HttpWebRequest object to create a HTTP Streaming connection to DataSift, follow the connection guidelines that DataSift follow (Twitter’s HTTP Streaming guidelines including a backing off process and handle reconnection) and I’ve not got my real-time Windows desktop application.

I now decide to build an iPhone application so that people on the move can access my service. I need to start from scratch by writing iOS (Objective-C) code to connect to DataSift’s HTTP streaming API, following the back off process and deal with reconnection. Once I’ve done that I move on to Windows 7 Mobile and I have to do the same again.

You get my point.

Now, once DataSift becomes popular, and it will, there will be loads of open source client APIs to choose from. However, these APIs are likely to have been built by a number of different developers providing different ways of doing the same things. Some might be DataSift specific but hopefully they’ll be generic HTTP Streaming libraries. At this point building my service applications will become a little easier but, in my opinion, still not as easy as it should be.

Using Kwwika

If I were to build this service, and not exclude Kwwika, I would set up all my real-time data streaming through DataSift and write a single library, probably reusing the Kwwika Tweet Streamer code which handles the HTTP Streaming connection and the Twitter connection guidelines, and push the data from DataSift into Kwwika. For each of my client applications I could simply download/use the relevant Kwwika API (JavaScript, .NET, Silverlight, Flash, Objective-C, Java, C) to subscribe to the DataSift data on any web-enabled device. And because these APIs are are developed and maintained by Kwwika they are consistent across platforms e.g. if you’ve used the .NET API you will be familiar with the JavaScript API.

My point is that the client APIs are as important as the server APIs. A quality real-time client API adds a lot of value to the service such as handling reconnection, failover, subscription management, connection monitoring, error handling, bi-directional messaing and request batching (not relevant in this scenario). As well as real-time data distrubtion/messaging/client push this is a key part of what Kwwika offers.

Related posts:

1. Kwwika-Superfeedr real-time demo available
2. Delivery as a Service and Data as a Service in 2011
3. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub

There’s quite a bit of work gone into this widget in a short space of time and I’m really pleased with the result. I’ll go into exactly how it was built in more detail in a later post but for now i’ll focus on what the widget can do and be used in this post.

Features

Opta Sports have created a an overview page for the widget which details the features at a higher level but I’m going to drill down a little bit deeper into the tech and explain how easily the widget can be embedded on any website.

Real-time ball-by-ball updates

For each event that occurs within a cricket match Opta Sports generate a feed. We get this feed, parse it and push it through Kwwika and into the web page. This means that every event is displayed on the widget.

Show either full scoreboard or simple match status

Ok, we finally get to see how you embed the widget in your page. It’s simple!

opta.cricket.widget("#scorecard-wrapper",
    {
        style:"scorecard",
        gameDirectory:"/OPTA/CRICKET/FIXTURES/AUSTRALIA_V_ENGLAND_ASHES_2010-11/20101226/4TH_TEST"
    }
);

The first parameter you pass to the opta.cricket.widget function is a jQuery selector. This tells the widget where it should be created. If the selector matches multiple elements then you’ll get multiple widgets. You probably wouldn’t want this but it’s a cool trick. The widget will replace the contents of the selected element(s) so until the widget loads you can have some HTML that says the widget is loading or just some static cricket data. If the widget can’t load for any reason then the contents of the element(s) won’t be replaced. This won’t happen though

The second parameter is options for the widget. We have a number of options that we support but the example above shows:

• style: The widget can be created in a number of forms and this option parameter indicates how the widget should be displayed:
  • scorecard: This style is the full scorecard for an innings. This displays an overview section and all batsman and bowler information.
  • mini: This style is very similar to the scorecard. It displays the overview section and just the batsman and bowlers that are active.
  • overview: This style shows just the overview section of the widget.
  • livescores: This style just shows the current score in a smaller widget area.
• gameDirectory: This identifies the Kwwika topic to request for the game and the data for this game will be displayed in the widget. The following topic is used for the 3rd test of the Australia v England Ashes 2010 to 2011 series: /OPTA/CRICKET/FIXTURES/AUSTRALIA_V_ENGLAND_ASHES_2010-11/20101216/3RD_TEST

See and download the full gallery on posterous

We also have a few other option parameters:

• innings: The widget defaults to displaying the latest available information. However, we cache some of the cricket data so it’s possible to display the data for an earlier innings from a match by passing in an innings number.
• flashTime: When an update occurs such as a run being scored or a bowler bowling a batsman out we flash the widget. This allows you to set how long the flash will last for.
• title: This is displayed at the top of the widget if it is in the livescores style.
• topicErrorMsg: A bit techie. If you have entered a gameDirectory that doesn’t exist then the widget displays an error message. This allows you to configure what that error message says. It can contain HTML.

Developed using Kwwika technology for instant data streaming

Yep, it’s the real-time web. Delivered Do you have any data that you want to instantly distribute to any web-enabled device? If so, get in touch.

Easy to customise – apply your own styling and branding

Since the widget is purely HTML and we’ve styled it with CSS you can easily customise the widget. Aaron Basset of Matador Digital did a great job when I asked him to supply a few different stylesheets. He told me that he’d used LESS to generate the CSS for the widget so it was easy to create different styles. Within 15 minutes Aaron had sent 10 different styles my way. You can apply the different styles in our example pages by just changing the drop-down at the top-left of the page. My favourites are the Kill Bill or Comic styles.

See and download the full gallery on posterous

Can be embedded anywhere within the website with just three lines of code

We’ve classed the HTML container element for the widget and the single widget script include as lines of code which bumps this up to 3 lines. I’ve used a number of widgets in the past and I think we’ve done a great job here.

<div id="scorecard"></div>
<script src="http://cdn.kwwika.com/widgets/opta/cricket/opta.cricket.js"></script>
<script>opta.cricket.widget("#scorecard-wrapper", {style:"mini", gameDirectory:"/OPTA/CRICKET/FIXTURES/AUSTRALIA_V_ENGLAND_ASHES_2010-11/20101216/3RD_TEST"});</script>

Okay, that last line is a bit long and maybe counts as 4!

Multiple matches on a single web page

You can call opta.cricket.widget as many times as you like and for each call you could pass a different game directory. That way you could easily have multiple games displayed on the same web page if you liked. You could also mix the styles of widgets that are on a web page.

I want that widget!

Opta Sports specialise in high quality sports data. They have a number of sports products and this real-time cricket widget is the latest such product. Opta Sports are looking for companies who are interested in taking this widget and embedding it on their website or within their web application. If you are interested please get in touch with us or or directly with Opta Sports via their cricket score centre page.

Since the ashes are nearly at an end we will also consider allowing you to embed this widget free of charge on your website for the remaining Ashes matches. Just get in touch.

I want the cricket data!

One of the things that excites us about this widget is that we are now seeing high quality sports data streaming through our servers. We’ve also proven that Kwwika is a fantastic distribution channel for real-time data. And we’ve seen that the delivery speeds between the data being captured and appearing on a web page are fantastic! We hope that this is the start of a big DAAS (Data as a Service) trend and that Kwwika can be at the forefront of it.

If you like the widgets but you’d love to just get hold of the raw data and build your own widget, web app, mobile app or desktop app then please get in touch too.

Examples

You can find a link to all the the live examples on Opta Sports – Cricket Match & Events Centre demo page.

The above examples are hard-coded to show the Ashes 2nd Test but here are the Scorecard widget links to allow you to view the data from the 3rd and 4th tests:

• 3rd Test: http://kwwika.com/Standalone/Demos/opta/cricket/ashes-widget/single-scorecard.html?game_dir=/OPTA/CRICKET/FIXTURES/AUSTRALIA_V_ENGLAND_ASHES_2010-11/20101216/3RD_TEST
• 4th Test: </OPTA/CRICKET/FIXTURES/AUSTRALIA_V_ENGLAND_ASHES_2010-11/20101226/4TH_TEST
• 5th Test: http://kwwika.com/Standalone/Demos/opta/cricket/ashes-widget/single-scorecard.html?game_dir=/OPTA/CRICKET/FIXTURES/AUSTRALIA_V_ENGLAND_ASHES_2010-11/20110103/5TH_TEST

From the format of the above links I’m sure you’ll be able to work out how to view the other examples with different games

Additional: You can also see the South Africa v India test using this topic:

/OPTA/CRICKET/FIXTURES/SOUTH_AFRICA_V_INDIA_TEST_SERIES_2010-11/20101216/1ST_TEST

Thanks

Last but certainly not least, a bit thanks to everybody we worked with on this.

Opta Sports

A big thanks to Opta Sports for working on this product with us and giving us a chance to show off our technology. Working with Alon in particular has been a pleasure.

Matador Digital

We’d also like to thank Matador Digital for the UI work and doing the majority of the widget functionality (I couldn’t help “lending a hand” though). Dave did a great job helping us come up with the UI and Aaron never ceases to amaze me with little pieces of information that make daunting tasks seem so simple. Really appreciate your hard work guys!

Permalink

| Leave a comment »

978d6a1bd7d647028d1d1f29d49cd10d

Related posts:

1. Marriage, Work/life integration, real-time web technology evangelism & Kwwika: A retrospective of my 2010
2. Adding a real-time "Who’s shopping?" widget to an ASP.NET Web App
3. Defining the Kwwika API

What are the differences between XMPP PubSub and PubSubHubbub and in what situations should each be used?

I do think I know some details about this but I’d like what I think I know to be confirmed or indeed corrected.

My knowledge of the real-time web, and specifically real-time client push, comes from working for Caplin Systems who pioneered building Comet servers around 10 years ago. From working with Caplin Systems and with Comet servers I’ve gained an understanding of how connections are made using different technologies and maintained in different scenarios. I also believe that I have a good understanding of what the best connection method is to deliver real-time data whether the delivery be from server to client using real-time client push or server to server using PubSubHubub or XMPP PubSub.

Persistent Connections

Unless the frequency of data updates is very low, and if you want to truly deliver data in real-time, the best type of connection is a persistent one.

As well as a persistent connection being able to deliver data faster it also means that your data subscriber does not have to handle a new HTTP connection for each piece of data it receives. This can actually be a really big thing.

As part of building Kwwika I wrote a demo which integrated the real-time server push capabilities of Superfeedr with the real-time client push capabilities of Kwwika to build a real-time news reader (RTNR) (blog article). In order to receive updates from Superfeedr I had to implement a publisher-subscriber outside of the Kwwika infrastructure. I decided to write a PubSubHubbub implementation in ASP.NET so that I could use a few other .NET Kwwika components to easily integrate with Kwwika.

So, each time Superfeedr has an update for the RTNR it establishes a HTTP connection to the RTNR PubSubhubbub server and sends the update which is then parsed and pushed into Kwwika to be distributed to subscribing web clients. If the RTNR is subscribing to something that is updating really frequently (such as “google” using Superfeedr track) then that can mean the RTNR server has to handle a lot of HTTP requests. Since I am running the demo on a small Amazon Windows instance running IIS it doesn’t take all that long, under a heavy load, for the server to stop responding.

There must be a better way!

Real-time client push

By way of a comparison let’s first take a look at real-time client push.

Before WebSockets provided the simplest way (but as yet not the most reliable since the WebSocket standard is not set in stone and isn’t supported cross-browser) to achieve real-time push to a web browser the best connection method was to maintain a streaming and persistent HTTP connection from the publisher to the client subscriber (using an IFRAME, XMLHttpRequest or XDomainRequest). The persistent connection means that the connection has already been established so as soon as the publisher has any new data for the subscriber it can instantly be pushed to it. This means that the data can be delivered to the web browser as close to real-time as possible.

Real-time Server Push

PubSubHubbub

PubSubHubbub turns things on it’s head in comparison to real-time client push (it actually uses HTTP as it was designed and it’s HTTP streaming that uses things in reverse. See Reverse Ajax.). In this case the publisher still pushes the new data to the subscriber but it does it using a HTTP request. The problem with this is that for each push a new HTTP connection needs to be established and the data then needs to be transferred. Establishing a connection takes time and resources so clearly a single persistent connection is a better solution. This is where I think XMPP PubSub is a better solution.

XMPP PubSub

If you need your data to be delivered in truly real-time (or as close to real-time as web technology will allow) then you have to use a persistent connection between the publisher and the subscriber. It’s my understanding that since XMPP was developed as an instant messaging protocol that it uses a persistent connection.

Conclusion

Here are things as I see them:

• If you want your data in real-time you should use a persistent connection between the publisher and subscriber.
• If you are making a server to server subscription to data that doesn’t update all that often and instant real-time doesn’t matter then PubSubHubbub is fine.
• If you are making a server to server subscription to data that updates very frequently then you need to use a persistent connection and XMPP PubSub is a must.

One thing to also consider is if you are subscribing to multiple sources which update frequently then maintaining a persistent connection to each of those sources, assuming they even allow you to, takes a considerable amount of effort and potentially resource. This is where a service like Superfeedr comes in to its own. They manage the subscriptions and connections to the sources for you which means you only need to maintain one persistent connection to Superfeedr. They do all the heavy lifting so you don’t have to.

Permalink
| Leave a comment  »

Related posts:

1. Real-time Data Delivery: HTTP Streaming Versus PubSubHubbub
2. The benefits of Real-Time Push-Once
3. Chrome/Safari (webkit) + real-time push = always loading indicator