Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications. The challenges include analysis, capture, curation, search, sharing, storage, transfer, visualization, and privacy violations.

en.wikipedia.org/wiki/Big_data

In my world it seems like everyone is talking about big data but when I move out of my specialist world and into the more ‘normal’ NHS front-line, and rub shoulders with nurses and other health care professionals, it doesn’t seem to have entered their world at all. But in truth big data is everything about their world – in future it could have a profound effect on care and everyone will have a role to play.

What is it?

Big data is really just lots and lots of data, from different places, that is mashed together and then analysed. It has become increasingly possible to understand data as more sophisticated computing power has come along. Modern computing power allows us to analyse what would have seemed impossible in the past. Now we can also store volumes of data that would have seemed impossible not so many years ago. We can now analyse data that is less well structured and still make it meaningful, especially by spotting patterns and trends that can then lead us to more detailed analysis.

I liken it to those fancy scanners they use on ‘time team’. The scanners give you clues what might have been underground but actually until you do the digging you may not be able to make real sense of it. Like the scanners big data can help you to see interesting patterns but often it needs much closer scrutiny – it takes a bit of digging to really understand. But if you couldn’t do the scanning you would never know there was anything interesting underground. Big data allows you to create new hypotheses and spot new relationships in care and treatments.

Of course big data isn’t just used in healthcare, it can be used in so many areas of life. Commercial companies are keen to tap into it to give them an edge to understand, for example, our purchasing behaviours; sports men and women can use it to improve performance; and we can use it in education to better understand how we develop and learn new skills. In all these areas it has the potential to transform and make a real difference.  In fact it has potential in so many areas of our lives.

Why does it matter?

In healthcare it matters because the data may have the clues to many disease processes that in the past have eluded our understanding. I have had type 1 Diabetes for nearly 35 years and in truth it feels like there has been very little progress in our understanding of the ‘why’ of Diabetes. Yes, treatments have improved but it often feels like a crude guessing game – and I apologise for that statement to all the wonderful scientists working in the field but I think big data might help them to get to the point more quickly.

The very precious nature of healthcare data

Of course any debate about access and storage of healthcare data is rightly heated and contested. Data about your health is one of the most personal aspects of your life and most people have a view about what it can be used for and who should have access to it. I agree that I should have some control but I really do want someone to find a cure for Diabetes. If I thought gifting my data, with some controls for privacy, would help to stop another young person at 16, as I was, finding out they had to face a lifetime of Diabetes I would do it gladly and willingly. Yet the debate about privacy and confidentiality continues to rage in the public domain.  We need to get this right – no excuses and no easy options; protecting the rights of individuals goes without saying.

If you are interested in what people who have chronic conditions want to use their data for then ‘Patients Like Me’ is a great case study to look at. I know that the data belongs to those individuals and they have the right to do with it what they will.  I do not want this post to be hi-jacked by the issue of privacy or confidentiality, nor am I saying it doesn’t matter – I just believe there are also other considerations to think about.

http://www.technologyreview.com/view/526266/patientslikeme-gives-genentech-full-access/

Data quality and responsibility

For practitioners big data does have an impact. Not only has it got the potential to transform how we deliver care in the future but practitioners have a responsibility to ensure the data they collect is high quality. In the past many records were rarely reviewed and languished for decades in medical records libraries in the bowels of hospitals. Now, and in the future, information we record will have a different visibility and transparency and we would do well to remember this.

Skills we will need

So the brave new world demands that we also have new skills. Being data savvy will, I believe, become a basic skill expected of people who work in the system and will go beyond simple statistics and the ability to use spread-sheets. We need skilled specialists too, people who can really help us to get to the nub of the data meaning.

Moving from knowledge to wisdom

But the most important addition we will all make to the big data debate is that of providing the context. Moving from knowing facts to a possessing wisdom requires us to throw upon the debate the light of truth and add our tacit knowledge and experience. It is people who provide this context, the insights and the meaning, turning facts into knowledge and then applying this to achieve greater wisdom; an endeavour we should all be contributing to. Here I mean ‘everyone’ – I don’t mean just people who work in systems, I mean just that: ‘everyone’. It is only if we have this whole context will we really be able to take the meaning from the data and take the steps we need to real wisdom.

Watch this TedTalk by Susan Etlinger to understand why big data is a journey we should all be engaged with. The title of my blog relates directly to her brilliant talk: