There is gold in big data, but there are not enough gold miners

March 21, 2018, University of Twente

The promise of big data has been on the horizon of the field of social sciences for years, but so far nobody has been able to deliver on this promise. In his inaugural lecture on 22 March, Professor and research methodologist Bernard Veldkamp explains why and offers solutions. His main point: we have to find different ways of dealing with this type of data. "The why-question should be replaced by pattern interpretation."

There is currently a revolution underway in terms of the amount of . Due to the rise of the internet, social media, mobile phones and all kinds of sensors, the amount of data rises by as much as forty percent annually. For , who deal with the behaviour of individuals and groups, these enormous amounts of data can be a goldmine. However, even though the promise of big data has been on the horizon for the field of social sciences for many years, the analysis has not yielded the results anyone expected. Consequently, enthusiasm is slowly turning to scepticism. In the meantime, computer scientists who develop methods to analyse the data are making off with the goods. An important disadvantage, however, is that computer scientists, given their background, are unable to sufficiently take account of the context and the information they are studying. As a result, the usefulness of their findings is limited.

Closing the gap

According to Professor Veldkamp, it is time for a paradigm shift. As one of the main reasons why social scientists lost the first battle, he mentions the fact that they tried to process and analyse the data using traditional methods for far too long. For traditional methods, the amount of data is simply too large. Moreover—and perhaps more importantly—the type of data involved is completely different compared to the data that was previously available. Because the data—unlike data from traditional samples, observations or questionnaires—has not been specifically collected for scientific analysis, the origin and quality are not always clear. Or, as Veldkamp simply puts it: "There is a lot of in it." This is why he believes it is very important to choose a different approach. The field of research methodology, Veldkamp's field, is ideally suited for closing the gap between and the of social sciences. "The data is worth its proverbial weight in gold, but there are currently not enough qualified gold miners for the job."


Professor Veldkamp believes that the fact that the origin of the data is sometimes unknown and that there may be a lot of white noise does not necessarily have to be a problem, but it is something you need to factor into your analysis. Big data, precisely due to the large quantity, offers the possibility to statistically correct uncertainties. In addition, Professor Veldkamp believes it's essential to "model in the white noise". This means that you have to be more careful with assumptions and that any conclusions you draw must be less firm. Professor Veldkamp: "For this reason, it is becoming more important to interpret connections than to look at causality. Or, to put it more simply, pattern interpretation should replace the why-question."

Explore further: Big data hype hasn't led to tan­gi­ble re­sults in the so­cial sci­ences, expert says

Related Stories

Using data mining to make sense of climate change

January 18, 2018

Big data and data mining have provided several breakthroughs in fields such as health informatics, smart cities and marketing. The same techniques, however, have not delivered consistent key findings for climate change.

What 15 years of mobile data can say about us

August 5, 2015

Large-scale anonymised datasets from mobile phones can give a better picture of society than ever before available. Mobile phone use helps us understand social networks, mobility and human behaviour. A review article recently ...

Recommended for you

What rising seas mean for local economies

February 15, 2019

Impacts from climate change are not always easy to see. But for many local businesses in coastal communities across the United States, the evidence is right outside their doors—or in their parking lots.

A river of stars in the solar neighborhood

February 15, 2019

Astronomy & Astrophysics publishes the work of researchers from the University of Vienna, who have found a river of stars, a stellar stream in astronomical parlance, covering most of the southern sky. The stream is relatively ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Mar 21, 2018
Very insightful article! We are seeing an increase in businesses seeking specialized skills to help address challenges that arose with the era of big data. The open source HPCC Systems platform from LexisNexis helps to fill this gap by allowing data analysts themselves to own the complete data life cycle. Designed by data scientists, the programming language called ECL is declarative and expresses data algorithms across the entire HPCC platform. Their built-in analytics libraries for Machine Learning and BI integration provide a complete integrated solution from data ingestion and data processing to data delivery. HPCC Systems provides proven solutions to handle what are now called Big Data problems, and have been doing so for more than a decade. Their free online introductory courses allow for students, academia and other developers to quickly get started. For more info visit:

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.