take one onion

So, I'm a good few months post book, nearly a year, I'd expected to have written more on here, but house hunting got in the way and my family are quite lovely too. I've got about a dozen lengthly drafts sitting around in marsedit on the social internet and other ideas inspired by the book, which I'll finish up soon enough. I've also been writing more frequently using tumblr on chopped onions.

However my new excitement (which I carefully avoided while writing the book) is the arduino, They are a huge amount of fun, I saw them first in 2007 via Alex and Matt (thanks) at the Paris XTech. A month or so later and I was drafting a proposal for the book I then started writing in January 2008. I could see just how much of a joyous time sink they would be. I'm quite glad I did in a way, Oscar is now four and really quite excited by the whole idea.

I made my first real own invented thing today, combined the LCD and temperature sensor to make a digital thermometer, which made me happy. the electronics worked first time, which surprised me. My brief exploration of electronics in my teens was not as much fun. Breadboards are excellent, though I have bought a soldering iron and a multimeter for the eventual real making of things.

More things to experiment with planned and I've acquired some of my 1980s space lego and will get the technical lego of the same vintage soon enough. That plus some lego hacking and an arduino mini will be fun. Also quite tempted by garden automation, measuring soil temperature, controlling watering etc seems like a good use of the arduino platform, I've read some of the stuff on growduinos already.

Is there a app store for commodity servers? Or is SaaS the dominant model, as other see it? I've been thinking about the distributed social web and wondering what is required to create a market for social and personal server apps. Ones that you can easily install from the store to your hosting provider of choice. Not everything will be on Google AppEngine or a Facebook app, nor should it require complex make install fiddling. I believe these are coming, in fact this post has been brewing a while and google have announced google apps for your google domain in the meantime, which addresses part of the ideas in this post.

I am thinking of a personal server which has simple consumer facing software which you can effectively drag install, applications like Mint and Fever are a good step in this direction. Certainly Simon Wardley has been arguing in this direction for a while. This thinking has been inspired by looking at the iPad, by reading Charlie Stross' Accelerando and Daniel Suarez's Daemon. Put in a more direct manner, where is the iPad server app store?

What exists at the minute? Server side applications are by and large aimed at developers and sys-admins. WordPress and Movable Type are probably the closest to widespread consumer apps. Other than Mint and Fever apps mentioned earlier, which are aimed at geeks. The vast majority of server installable applications are for other software developers, not a small market, but not mainstream. There are lots of options for these kinds of app Amazon AMIs, virtual MovableType or what Cloudera are doing with Hadoop and the vast array of VMware products. Most if not all of these apps are free and the money is made from consulting and service contracts.

I'm also not really thinking of Google App Server or similar easy development environments, nor just RPM / CPAN / MacPorts "apps" for geeks. Perhaps something like Apple's Remote Desktop with applications packaged as installing DMG files would be a step in the right direction. There are MySQL installs as a DMG, plus many others. These are an improvement over make and make install, but they are not apps in the same light as Mint above, nor as invisible as an iPhone app install. There are tricky issues to solve including the at least four different platforms for which virtual images are created, making these is not trivial.

What is required to make a consumer server side app store possible or even desirable? It needs to be simple to install, eg drag and drop to install. Then quick to configure. Tools like webfinger (protocol) will automate people lookups, these are the kinds of hooks that will simplify setup, I'll come back to this later. I imagine these tools will be personal always on data processing tools. Rather than having a server at the end of your ADSL line you'll have a managed instance of a virtual server in a colo somewhere. They could be the wearable computing of scifi fame, but i suspect that phone as sensor with data processing in the cloud is more likely. The ability to easily move these apps and their data from one service provider to another is vital. Another lock-in is not desirable, I think it is more important with these data rich applications than with iPhone apps. Plus this kind of data portability is widely supported for server apps, eg SQL, imap, commonlog files etc.

At XTech in 2008, Steven Pemberton gave a talk on hosting your own data amongst other ideas, which Jeremy Keith wrote great notes on. The ideas in this talk won't leave me alone, go and read it if you've not come across it. I see identity services as a driver for these tools be you@yourdomain rather than gavin1234@centralizedservice.com, single namespaces get filled up pretty quickly. The web is young still, 16 years old in public awareness. What will new users of Twitter or Facebook or Linkedin have as options for names in even five years? Many people already get a vanity domain with email, but imagine a distributed twitter (fethr?) or linkedin, facebook etc. The laconica / identica project, aka status.net is heading in this direction too, imagine this running on your WRT54G or FON router.

Some other ideas that spring to mind. A web proxy that looks at every thing you read online and finds updates, references etc based on the url stream it receives and processes, there was a mac app that did this locally, can't remember name. This might find prices on things you've looked at; locate good books etc on things you search for or recommended blog posts on other things you've read. There is another set of apps in the pachube mode, processing the increasing array of sensor data that we are generating, location based data is one popular area now, but there are many more coming soon, eg smart meter data.

Another possibility is the delegated you that could exist, we have this to a limited degree already in terms of free/busy statements for calendars. Can we extend this to some sort of basic automated decision making capability on our behalf, this already works well using an multi-agent based model for the estar system, which allocates time on automated telescopes worldwide. I've mentioned brokerage agents before in terms of delegating authority, but now there are proposals for OAuth delegation to support these kinds of ideas. Combined with webfinger and knowledge of who your friends are via an address book a service like this could process some social application invites automatically. Extending this why doesn't my copy of MovableType or Wordpress know who I am? these are generally single user software, so it should be aware of who my friends are, perhaps acting as a whitelist for comments as a small example.

Lastly (and more wildly) are there sufficient personal volumes of data to warrant a hadoop / hive / mahout application for an individual? Trawling the friend of a friend and wider internet for useful information based on a model of your interests? A kind of intelligent RSS reader, perhaps based on the estar multi-agent ideas. That few of us have an ongoing data processing agent watching the web for us bothers me. Where are the intelligent agents of sci-fi fame? Douglas Adams suggested the monk plus which dealt with the world so that you don't have to, a delightful creation. One of those which helped me deal with the world would be very welcome. James Bridle's lovely sxsw book is another good example of the kinds of things these agents might make for us, give it the conference you are going to and the flight your flight plus hotel and a guide pops out tuned to your friend, location etc.

I think these are coming soon, the recent efforts from Google give strength to that, but a single provider lock in is not desirable. How do we make these apps exist and make them easily portable?

I'm at sxsw for the next week or so. I'm doing a core conversation on building good social software, entitled Do The Right Thing: Building Respectful Software with my technical editor for my book, Matthew Rothenberg, let us know you're coming. I'm also doing a book signing on the Sunday at 11:50 at the Southby Book store, come and say hello.

Mainly some notes for myself following a chat on Twitter. Wifi access at SXSW is a bear. Too many people trying to use the 3G if you are on AT&T, too expensive to even try if you are on data roaming. There is some wifi coverage around the convention centre, but it gets pretty swamped. So I went looking for other options. Boingo will give you coverage for about $10 a month, they have 130 hotspots in Austin though coverage is quite widespread. I did find OpenWifiSpots, which have a good map for free wifi in Austin, most of the data is from 2009, so it might be out of date. They even have a pay for iPhone app, which might help. So with those notes hopefully I'll be able to get internet access at sxsw.

Will the iPad work as a standalone computer, the only one that someone owns? Much of the geek led interest in the iPad stems from its simplification of the computing experience. A device for "the rest of us" that focuses on getting things done with the computer, not doing computing. I agree with this viewpoint, but it does raise a troubling issue. How does someone back up an iPad? The easy option is to backup to a local Mac or PC, but if this is meant to simplify computing, then why do you need another computer to act as sync and backup.

I've read that there is a small segment of iPhone owners who bought it for the internet everywhere computing device I mentioned in a previous post. For some people the iPhone was their only computer, it mostly works in that mode too, as long as your needs are not great.

Backup is the aspect that is not covered. The announcement video and details on Apple's website give some information, but I suspect there is more to come. One suggestion which comes to mind involves additional hardware from Apple. They already have the Time Capsule device for automatic backup of a Mac and it works as a wireless hub too. I can see a wireless access point with a 1-200GB drive inside, plus a dock connector as a single device from Apple. It would provide backup when docked, wireless access when not and a place to charge the iPad overnight.

We are not yet at the point where internet backup restore is that feasible, most people still have 512k ADSL at home, rather than ADSL2+ rates of up to 24meg. Also the customers for this machine are not likely to be the purchasers of high end internet access. A simple backup device like this sticks with the simplicity approach which the iPad offers, plus it gives Apple another profit rich accessory for them to sell.

The Time Capsule already has a USB port, I'd expect a cheaper version of with iPad support using this USB port to be released. Potentially the dock version I mention above and software support for the current models as a USB led iPad backup. 1TB is a bit excessive for a 16-64GB device and $299 is two thirds of the cost of the iPad. Apple may decide to push for more MobileMe subs though and an internet based backup system. Restoring 64GB over basic ADSL is going to take a while... So a smaller and cheaper Time Capsule would be a better match.

The blog to accompany my book, Building Social Web Applications is taking shape. Moving house, Christmas and ill children conspired against it being a regular blog until now. Go and have a look. Articles on geo datasets, plus Google Buzz so far and plenty more planned.

I've been meaning to write up something on the iPad since it was announced. Not really about the technology, nor really about it being the computer for the rest of us. Those are interesting, but they've been well covered elsewhere, Fraser Spiers and John Gruber in particular cover it well.

What struck me was instead something about the iPhone. The most significant enabling aspect of the iPhone launch was the unlimited cellular and then later 3G access. It was this ubiquitous access to the internet which drove the continuing demand for the device. This allowed you to check your email, the web, twitter everywhere, plus allowed every app to assume the internet would be there. Imagine an iPhone where you counted every byte you used, I had plenty of phones like that from Nokia on Orange, grim and rarely used.

I believe that this is what Apple are trying with the iPad. I read a lot of science fiction and no where in the plot does someone worry about paying for access to the communal network. It is a given. Just like Apple simplified (forced) the pricing model for 3G phone access I think this is what they are driving at here. No contract, two levels a month for access, compared to 24 month fixed contracts which are the norm from telcos.

Always on, everywhere, unmetered access will change how computers are used, just the same way as the iPhone changed how phones are used, maybe not to the same degree, but it is a shift towards the computer as part of live for everyone, not just geeks with dongles.

We've moved to Wanstead and had Christmas in less than a month, so it has been busy. Wanstead is lovely in the snow, there is a pretty forest behind our house so all is peaceful.

It has been pretty peaceful here too, but as is the way of the new year, I will hopefully change that. I've joined up with the write more project52 which should result in a blog post a week. That would be the first time since 2003, so we'll see. Writing the book has given me a taste for it, but I've not had the time other for much other than the book / job / family, then moving house took over my other spare time.

The other idea I've had is the quarterly project. I hope that three months is enough time to get one of the ideas floating around my head linked to one of the domain names I've bought and pushed out into the wilds. Briefly extending this, I mean to create a simple web app that does something useful for me and hopefully a few other people too.

Last week I had fun making a newspaper with Tom Taylor and Dan Catt. Tom and Dan have already written it up, along with Zach Beauvais at Nodalities. I spent most of my time trawling the data.gov.uk site finding data relevant at a postcode level. It was a great experience and I really enjoyed reconnecting with making things, having spent a long time writing about other people making things.

The intent was to make something that would give you a guide to a new local area, the context was something that a council might send out to you a few times a year. There is a huge amount of interesting information gathered by Government, but what we usually see is aggregated, averaged and sometimes skewed to tell a message. Taking the data and linking it directly to a postcode makes this data more impartial and meaningful, which leads me on to the gist of this post. Freeing up the postcode to census output area data as a freely available API, I'll explain.

Much of the data collected at a Government level is gathered at quite a fine grained level. The lowest level of this data collection is the output area. An output area is quite a small parcel of land, for my work it is essentially the canal basin behind my office. To see this, head over to the Neighbourhood Statistics website and type in N1 9XW on the lefthand panel, click on the More areas link and select the output area radio button. The Neighbourhood Statistics website discourages direct linking, so this direct link to N1 9XW might well fail. The ONS have a helpful guide to UK geography too.

However this Neighbourhood Statistics website is a gold mine of useful information. plus it offers a key conversion from postcode to output area. Land is hard to describe, for the patch of land your chair is sitting on there are many different levels of descriptor. Some of these are point based like latitude and longitude which allow placing on a map. Most of the descriptors for Government data are boundary based. The output area is one of these. It is the smallest parcel of land that the Census collects data for. Once you get the output area though you can then find out the higher area entities like which MP constituency or Health Trust you are inside. The data behind this is all available for free from the ONS, but for internal use only.

The whole process to figure out your MP goes something like this. Enter postcode, map to output area, from there determine which ward and thence which constituency you are living in. It gets more complex if you don't know your postcode or if it has changed recently. The Royal Mail PAF (postcode address file) is the key to managing postcodes, it links postcodes and addresses. The names of the areas of land are managed by a gazetteer which links names to boundaries and will often include older names for areas, eg the old counties of England no longer exist at an authority level, there are all now different types of council.

Coming back to the simple case of knowing your postcode, there are a lot of concerns about access to this data, including campaigns to free the PAF file, which got a pretty lukewarm response. The Royal Mail own the PAF file and the Ordnance Survey own the Boundary line data, so between them they seem to have the entire thing sown up, but the mapping of postcode to output area is the property of the ONS. It is available free of charge from them, go order a copy now. You can use it internally as much as you like.

While that is useful, what I'd like to see is the lovely service on the Neighbourhood Statistics website made available as a REST API. Give it a postcode and get back JSON listing the output areas and upwards it falls into. The data is all on the results pages for the site and as the site states Boundaries for Output Areas (OAs), Lower Layer Super Output Areas (LSOAs), Middle Layer Super Output Areas (MSOAs) and Travel to Work Areas (TTWAs) are available free of charge to all users.

Making it available as a postcode to Output Area API unlocks all the Government data that the data.gov.uk project intends to release. I feel that this should be the responsible attitude of a Government committed to an Open Data initiative. Releasing the Postcode to OA does not free maps from the OS, nor give address details which the PAF file covers. Yet it provides an immense value to online civic projects.

My book Building Social Web Applications is now generally available in print form, as an eBook and even as an iPhone application. Amazon US, Amazon UK, iTunes iPhone or iPod touch app, O'Reilly eBook and online access via Safari from O'Reilly. I'm really pleased to see it out there and I'm starting to get good reactions to it too.