How to find the knowns and unknowns in any research

Have you ever felt overloaded by information? Ever wondered how to make sense of claims and counter-claims about a topic? With so much information out there on many different issues, how is a person new to a subject supposed to find out what is known about a subject?

We often face the challenge of mastering new topics. This is the same for anyone be they students, journalists, activists or even researchers starting or changing research topics.

In a world of increasing information availability, finding ways to make sense of the literature on a topic is important.

We have developed a direct way for people to approach a chosen topic and to drill down through published literature to find out what is known, but also where there are knowledge gaps.

We call it a Systematic Quantitative Literature Review. This fancy term just means counting how many times papers on a chosen topic have been published, where, when and by who, and then thinking about results – what does this mean?

We have produced online support for those wanting to do these Systematic Quantitative Literature Reviews. This includes YouTube videos and papers outlining the method and a growing list of examples of many papers published using this technique.

What's most exciting about this method is that almost anyone can use it - not just students or academic researchers. No specialist equipment is required - just spreadsheet software such as Microsoft's Excel.

And almost any topic can be investigated: from climate change, to vaccination and even searching for better schools. Users can quickly master the technique to wade through claims and counter claims on any subject, discovering what's known and finding the gaps.

So how do you do it?

For a start you'll need to review the published literature on your chosen issue. But which literature and how much do we need to review?

Back in 2002, the then US Defence Secretary Donald Rumsfeld made the now famous quote about known and unknown information when he said:

[…] as we know, there are known knowns; there are things that we know that we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know.

Traditional narrative reviews conducted by experts tend to focus on Rumsfeld's "known knowns". These reviews tell a story about how knowledge has developed, where research has been concentrated and what it has found.

Other methods such as the famous Cochrane Reviews in medical sciences focus on what is known, but also create new knowledge: Rumsfeld's "unknown knowns". They collate data from published studies, reanalyse the data to provide more robust answers to important research questions, and identify knowledge gaps.

Both approaches are quite complex; they require technical knowledge and considerable familiarity with the literature before you can start, and definitely are not for the novice researcher or citizen activist.

There must be an easier way?

We use a third approach, which is potentially more useful for ordinary people wanting to navigate a seemingly endless mass of literature.

Our approach enables users to systematically identify what is known and then using quantitative methods they can assess what is unknown.

Our systematic quantitative reviews involve four steps:

1. First you systematically search the literature for original research (published reports and papers) on a specific topic. This can be done by anyone with an internet connection. Google Scholar has been proven to be a reliable tool to search for published literature. Just enter your keywords on the chosen topic and you're on your way.
2. Next, enter the different reports and studies that you find into a your spreadsheet (such as author, location, date of study, purpose, research questions etc).
3. Then you summarise and evaluate the papers by using tables and maps - showing what is known, but also what is not known. Google Maps can be used to display the countries where most research has been conducted. You can even create tables or graphs from your spreadsheet, showing who is being funded, or what findings are being reported about particular places and topics.
4. Finally, you can report the findings - in a newspaper, newsletter, chatroom, blog or even a scholarly journal.

What to look for

A Systematic Quantitative Literature Review can help answer important questions, such as:

1. who has published on the topic?
2. where has research been conducted?
3. when was most of the research undertaken?
4. what key research questions were addressed, and are any missing?
5. what methods have been used and have any been neglected?
6. what have researchers discovered?
7. are there any limitations (such as problems with methods, analyses and results)?
8. what is still unexplored?

There are some limits to this method. It may not be as appropriate for some areas in humanities, architecture, music, literature or dance-studies. It is important to be consistent and objective in how you search and select documents, and how you classify the data about each document in your database too.

This method can work for a wide range of subjects: from identifying the benefits of community gardens to selecting street trees, from assessing the impacts of tourism on birds and other animals to assessing performance in architecture, to evaluating the risk of cars carrying weeds, to finding out how your European holiday may threaten rare plants.

Using this method, any one from academic researchers to teachers, lawyers, activists and parents can become more knowledgeable on a subject, contribute to knowledge-creation and identify limits to current knowledge.

They can also prioritise where we need to go to gain new knowledge, turning more of Rumsfeld's unknown unknowns into known unknowns.

This story is published courtesy of The Conversation (under Creative Commons-Attribution/No derivatives).