What Data You Got is Not As Important as What you Do With It

First let me start with a story. I have some personal experience in 1990 with this ancient mariners’ rhyme: The Rime of the Ancyent Marinere:

Day after day, day after day,We stuck, nor breath nor motion;As idle as a painted shipUpon a painted ocean.

Water, water, every where, And all the boards did shrink; Water, water, every where, Nor any drop to drink.

I was a 25 year old newly appointed Chief Mate

This was on a ship with a very intransigent bunch of crew from an aggressive Marxist union. What I did not know at this point was that all the crew ratings received a hardship allowance for every day they spent on the ship under water rationing conditions. The ship embarked on a several weeks voyage across a vast ocean with sufficient stock of water, and then some surplus. This was one of my responsibilities.

As usual I was very careful with monitoring the water stock on board

I was also closely monitoring the daily usage. I had the cadet and chippy (carpenter) take sounding morning and evening – and write the daily water consumption on a blackboard in the crew mess, as well as the officers ward rooms. I took daily rounds in the accommodation to monitor the dripping taps to stop wastage of water. Any jump in water consumption was promptly investigated – all for a good reason. I wanted the ships company to arrive with sufficient water stock.

I will never work out where it was a sabotage

Perhaps it was just an accident – but somehow sea water gained entry into every fresh water tank on the ship (but that is a long story about unions and compensations which is best told at some other time). The captain (and I) was left with a dilemma, whether to rely on the fresh water generator on board for a long sea voyage, or not. There was no shortage of water – it just was too salty for most purposes. I will tell you how this voyage panned out at the end of this blog post.

But I want to change course here

to the real purpose of the post – the over-abundance of un-usable data, and the opportunities that it presents for start-ups. In my last blog post I recounted a real life story of a business transformation project where information technology fell woefully short.  This is not the only such situation I encountered. In fact I recount more than 20 similar examples in my book ‘UNCHAIN YOUR CORPORATION’.

Data, Data every where – and not a blot to think

Look at the growth in global data storage capacity below:

Obviously it is not just the capacity – but also the data that is growing. There are many graphics showing growth of data – a few of them are Every Day Big Data Statistics, Data Generated On Social Media Every Minute and 4 V’s of Big Data and all over the cyberspace. The key story they all are telling is just one –

WE ARE FLOATING IN  SEA OF DATA.

Just like the ancient mariner had no dearth of water, we have no dearth of data. And, just as minute quantities of salt (3.5%) that is present in all that seawater, is enough to make raw seawater unusable for most practical purposes, minute quantities of data error is enough to make most of the raw data unusable for practical purposes. On the ships (and in many locations on land) we deploy fresh water desalination plants such as Sydney Desalination Plant. The sole purpose of these massive plants is to parse seawater, take it into a pressure chamber, evaporate it at high temperature (because that requires less energy) and then condensate it into distilled water. Further processing is required to remove other contaminants such as bio-hazards. Check here for infrastructure of the full process in some more detail.

The biggest opportunity

In information technology the most bang for the buck is not in generating or collecting more data, but in making the data more usable. I was asked a question at a recent speech why I was not as bullish as everyone else on big data. I likened the current big data set-ups to an ocean full of seawater. It still takes a huge expense to desalinate the seawater, and to make the data usable.

Start-ups are still woefully short on this

Any start-ups that figure out a better way to collate, parse, access, and make usable the data to create insights would be a tremendous success. If you know of any, please let me know in the comments below.

Coming back to the story

In the voyage above, we had to put into an emergency port to get fresh water rations.  Without doing this none of our sailors would have survived. In the emergency port, we had to fight with a bunch of sea pirates, which is a whole new story .