Study highlights flaw in common approach of public opinion surveys about science

January 13, 2011

A new study from North Carolina State University highlights a major flaw in attempting to use a single survey question to assess public opinion on science issues. Researchers found that people who say that risks posed by new science fields outweigh benefits often actually perceive more benefits than risks when asked more detailed questions.

"We set out to determine whether we can accurately assess on complex science issues with one question, or if we need to break the issue down into questions on each of the issue's constituent parts," says Dr. Andrew Binder, an assistant professor of communication at NC State and lead author of the study. "We found that, to varying degrees, accuracy really depends on breaking it down into multiple questions for people."

To assess the problematic nature of a single-question surveys, the researchers developed two surveys; one focused on and the other on biofuels. In each , respondents were asked an overarching question: do the risks associated with nanotechnology/biofuels outweigh the benefits; do the benefits outweigh the risks; or are the risks and benefits approximately the same? The researchers then asked survey participants a series of questions about specific risks and benefits associated with nanotechnology or biofuels.

The researchers then compared a participant's response to the overarching question with his or her responses to the specific questions in order to see whether the overarching question accurately captured the opinion of the individual respondent.

They found a problem.

"There was a significant discrepancy among people who responded to the overarching question that the risks of emerging science outweighed the benefits when compared to their responses to the questions about the specific risks and benefits," says Binder. "Namely, those same people really perceived more benefits than risks when given the opportunity to evaluate these attributes separately.

"For example, in the nanotechnology survey, 50 percent of respondents who said risks outweighed benefits actually evaluated nanotechnology positively in the other portion of the survey," Binder says. "In fact, only 35.4 percent of respondents who thought risks outweighed benefits actually calculated more risks than benefits in the specific section of the survey." The researchers found similar, though less pronounced, results in the biofuels survey.

The study also showed that people who said that benefits outweighed risks in response to the overarching question consistently perceived more benefits than risks in the specific question section of the surveys.

"The bottom line is that social scientists and journalists need to be very careful when relying on data from a single, overarching survey question," Binder says. "These oversimplified questions can result in misleading poll data and create problems for policymakers who base their decisions on those findings. They can also be problematic because they may contribute to different polls showing widely different results, which weakens the public's faith in surveys generally."

Explore further: More talk, less agreement: Risk discussion can hurt consensus-building on science/technology

More information: The paper, "Measuring risk/benefit perceptions of emerging technologies and their potential impact on communication of public opinion toward science," was published online Jan. 12 by Public Understanding of Science.

Related Stories

Poll reveals public awareness of nanotech stuck at low level

September 25, 2007

National survey findings released today indicate that Americans’ awareness of nanotechnology remains low. Popular awareness is nearly as small as the tiny nanoscale materials and nano-enabled devices and products now flowing ...

Nanotechnology: What's that?

September 4, 2007

Nanoscience and nanotechnology are two of the hottest fields in research, investment, and manufacturing. Some hail nanotechnology as enabling "The Next Industrial Revolution."

Public attitudes to nanotechnology: Lessons for regulators

September 21, 2009

New technologies may change our lives for the better, but sometimes they have risks. Communicating those benefits and risks to the public, and developing regulations to deal with them, can be difficult -- particularly if ...

Recommended for you

Fossils of early tetrapods unearthed in Scotland

December 7, 2016

(—A team of researchers working at a dig site in Scotland has found tetrapod fossils dated to approximately 15 million years after the Devonian mass extinction—a time period experts in the field have referred ...


Adjust slider to filter visible comments by rank

Display comments: newest first

5 / 5 (1) Jan 13, 2011

So: Asking more specific questions produces more accurate answers? (insert sarcastic remark of your choice)

In other news, crime is causally linked to people who don't adhere to laws. More details at 11.
5 / 5 (1) Jan 13, 2011
i wonder if in the biofuels section they mentioned the risk that taking food from the mouths of the hungry to feed our cars will lead to worldwide food shortages?
3 / 5 (2) Jan 13, 2011
So why is it common sense didn't let people know that this was true? Half the time I am on the CNN web site I don't respond to their poll even if I want to because the answers to the question are always yes or no. Neither answer is correct much of the time. If you have fools asking the questions you will have foolish answers.
not rated yet Jan 13, 2011
They should ask specific questions but there will aleays be biased opinions... sadley...
not rated yet Jan 13, 2011
Yes, you can make a poll say anything you like, but with verifiable accuracy, people can counter or uphold the opinion with facts.
not rated yet Jan 13, 2011
As Einstein once said "Simplify as much as possible, but not more!" Administrators all want that "sound-byte" that sums up all known knowledge, answers all their problems and overcomes all objections... hence we have the public poll. The other unfortunate fact is administrators aided by sociologists formulate the questions in polls according to what they want to hear. Truth through polling is the dumbest way to seek information because scientific knowledge is complex and there is no way to get any "complete" knowledge on any subject due to Godel's incompleteness theorems forcing a fundamental limitation in all our wisdom.

As Professor Andrew Stirling recently stated ("Keep it Complex" Nature 23-30 Dec 2010 V468 p1031) "When knowledge is uncertain, experts should avoid pressures to simplify their advice. Render decision-makers accountable for decisions".

IMHO these decision-makes should study science in depth in order that their decisions are relevant and understood.
not rated yet Jan 13, 2011
The value of breaking complex questions down - not any different than breaking problems down to sub- components for more efficient solution - has been remarkably well understood for several millennia now. However, no matter how you break down a question, if the person being asked has little or no knowledge on the subject of the related question - he or she will still not be able to produce an answer of any value - because it will be baseless. Why would you want the average person making decisions regarding complex problems without the necessary expertise? This is the great and fatal weakness of democracy - the average person is never going to have expertise necessary to make decisions on anything that is more complex than average. We have the incompetent politicians that prove this - daily.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.