Is big data heading for its 'horsemeat moment'?

There have been so many leaks, hacks and scares based on misuse or misappropriation of personal data that any thought that "big data" could provide benefits rather than only opportunities for harm may be fading in the public imagination.

Beyond the humiliation of those involved, the main effect has been to deepen the widespread erosion of trust in the information infrastructure that records and holds this disparate data – from phones and cloud storage to companies and government services. The growing concern is that the wrong data will be used against consumers, by the wrong organisations, and in the wrong way.

Whether it's an A-list celebrity hacking scandal or an everyday portion of data-driven spam, each time consumers' faith in the idea that personal data can be used for good takes a knock.

And there are parallels with other sectors. Recall the consequences of the revelation in January 2013 that products advertised as containing beef were in fact partly or mostly made of horsemeat. Consumer confidence in the food supply chain, including some of the country's leading supermarkets, plummeted almost overnight. With every celebrity exposé, every little act of misappropriation, the concept of "big data" edges ever closer to its own horsemeat moment.

Benefits undersold

This would be a catastrophe, because of the extraordinary benefits that big data could bring if used correctly. We are all acutely aware that the expansion of personal data collection and the new uses it's being put to are transforming how organisations work with the customers and with each other. But what is less appreciated is that even the very latest techniques really only scratch the surface surface of what might be achieved.

Of course, "data" means different things to different people. "Data is the new oil" is one metaphor to describe the importance of its new role. "Big data is the minute-to-minute personal diary of everyone and everything," is another. But in the end the real meaning of big data, particularly in times of austerity, is to uncover the unmet needs of customers and citizens.

We do more and more online, and everything we do there generates useful data, as does each stage of every supply chain. At present it is our smartphones, not our banks or GPs or anything else, that hold our most personal data. The more organisations that can use that data, irrespective of how uncomfortable the idea might seem, the more useful that information becomes to us.

As explained in a paper published by the Information Commissioner's Office earlier this year, gaining the maximum value from big data will come from much great precision and personalisation. Given the vast scope for directing resources and services efficiently and accurately, it's no exaggeration to state that the future success of our society and our economy may well rest on these two pillars. The challenge we face is how to get the most out of precision and personalisation while still preserving consumer trust.

Big data needs big regulator

One reason why meeting that challenge will not be easy is the absence of a meaningful and effective third party in the data ecosystem. There is some consensus among policymakers, regulators, commercial data firms and consumer privacy experts that what is needed is some sort of recognised, accountable, trustworthy body to bridge the gaps between consumers, suppliers and the market.

This organisation would serve as a broker, managing a person's data and dealing with firms on their behalf. It would aim to prevent or at the very least fix any harm that occurs due to the misuse of data. It could uphold the public interest, but also educate.

For example, it could help organisations share and use data in new and useful ways, but only if the data was used appropriately – staying true to the spirit of the right people using the right data for the right reasons. It could even award trust marks – such as the one that the Information Commissioner's Office are consulting on – to firms that can prove they use their customer's data responsibly.

Of course, assembling one or more organisations capable of sitting at the difficult point where consumers, companies, regulators and privacy organisations meet is no mean undertaking. For a start the organisation would itself have to demonstrate its own trustworthyness and independence.

But the alternative is infinitely less attractive: if we allow consumers' already shaky trust in very notion of big data to be damaged beyond repair – if that "horsemeat moment" is reached – the full potential it holds for society might never be realised. And in the final reckoning, that would be a bitter pill to swallow.

This story is published courtesy of The Conversation (under Creative Commons-Attribution/No derivatives).