Study highlights flaw in common approach of public opinion surveys about science

Jan 13, 2011

A new study from North Carolina State University highlights a major flaw in attempting to use a single survey question to assess public opinion on science issues. Researchers found that people who say that risks posed by new science fields outweigh benefits often actually perceive more benefits than risks when asked more detailed questions.

"We set out to determine whether we can accurately assess on complex science issues with one question, or if we need to break the issue down into questions on each of the issue's constituent parts," says Dr. Andrew Binder, an assistant professor of communication at NC State and lead author of the study. "We found that, to varying degrees, accuracy really depends on breaking it down into multiple questions for people."

To assess the problematic nature of a single-question surveys, the researchers developed two surveys; one focused on and the other on biofuels. In each , respondents were asked an overarching question: do the risks associated with nanotechnology/biofuels outweigh the benefits; do the benefits outweigh the risks; or are the risks and benefits approximately the same? The researchers then asked survey participants a series of questions about specific risks and benefits associated with nanotechnology or biofuels.

The researchers then compared a participant's response to the overarching question with his or her responses to the specific questions in order to see whether the overarching question accurately captured the opinion of the individual respondent.

They found a problem.

"There was a significant discrepancy among people who responded to the overarching question that the risks of emerging science outweighed the benefits when compared to their responses to the questions about the specific risks and benefits," says Binder. "Namely, those same people really perceived more benefits than risks when given the opportunity to evaluate these attributes separately.

"For example, in the nanotechnology survey, 50 percent of respondents who said risks outweighed benefits actually evaluated nanotechnology positively in the other portion of the survey," Binder says. "In fact, only 35.4 percent of respondents who thought risks outweighed benefits actually calculated more risks than benefits in the specific section of the survey." The researchers found similar, though less pronounced, results in the biofuels survey.

The study also showed that people who said that benefits outweighed risks in response to the overarching question consistently perceived more benefits than risks in the specific question section of the surveys.

"The bottom line is that social scientists and journalists need to be very careful when relying on data from a single, overarching survey question," Binder says. "These oversimplified questions can result in misleading poll data and create problems for policymakers who base their decisions on those findings. They can also be problematic because they may contribute to different polls showing widely different results, which weakens the public's faith in surveys generally."

Explore further: Study finds law dramatically curbing need for speed

More information: The paper, "Measuring risk/benefit perceptions of emerging technologies and their potential impact on communication of public opinion toward science," was published online Jan. 12 by Public Understanding of Science.

Related Stories

Poll reveals public awareness of nanotech stuck at low level

Sep 25, 2007

National survey findings released today indicate that Americans’ awareness of nanotechnology remains low. Popular awareness is nearly as small as the tiny nanoscale materials and nano-enabled devices and products now flowing ...

Nanotechnology: What's that?

Sep 04, 2007

Nanoscience and nanotechnology are two of the hottest fields in research, investment, and manufacturing. Some hail nanotechnology as enabling "The Next Industrial Revolution."

Public attitudes to nanotechnology: Lessons for regulators

Sep 21, 2009

New technologies may change our lives for the better, but sometimes they have risks. Communicating those benefits and risks to the public, and developing regulations to deal with them, can be difficult -- particularly if ...

Recommended for you

Study finds law dramatically curbing need for speed

Apr 18, 2014

Almost seven years have passed since Ontario's street-racing legislation hit the books and, according to one Western researcher, it has succeeded in putting the brakes on the number of convictions and, more importantly, injuries ...

Newlyweds, be careful what you wish for

Apr 17, 2014

A statistical analysis of the gift "fulfillments" at several hundred online wedding gift registries suggests that wedding guests are caught between a rock and a hard place when it comes to buying an appropriate gift for the ...

Can new understanding avert tragedy?

Apr 17, 2014

As a boy growing up in Syracuse, NY, Sol Hsiang ran an experiment for a school project testing whether plants grow better sprinkled with water vs orange juice. Today, 20 years later, he applies complex statistical ...

Creative activities outside work can improve job performance

Apr 16, 2014

Employees who pursue creative activities outside of work may find that these activities boost their performance on the job, according to a new study by San Francisco State University organizational psychologist Kevin Eschleman ...

User comments : 7

Adjust slider to filter visible comments by rank

Display comments: newest first

LKD
5 / 5 (1) Jan 13, 2011
...

So: Asking more specific questions produces more accurate answers? (insert sarcastic remark of your choice)

In other news, crime is causally linked to people who don't adhere to laws. More details at 11.
Smoulder
5 / 5 (1) Jan 13, 2011
i wonder if in the biofuels section they mentioned the risk that taking food from the mouths of the hungry to feed our cars will lead to worldwide food shortages?
Moebius
3 / 5 (2) Jan 13, 2011
So why is it common sense didn't let people know that this was true? Half the time I am on the CNN web site I don't respond to their poll even if I want to because the answers to the question are always yes or no. Neither answer is correct much of the time. If you have fools asking the questions you will have foolish answers.
Paljor
not rated yet Jan 13, 2011
They should ask specific questions but there will aleays be biased opinions... sadley...
LKD
not rated yet Jan 13, 2011
Yes, you can make a poll say anything you like, but with verifiable accuracy, people can counter or uphold the opinion with facts.
GoodElf
not rated yet Jan 13, 2011
As Einstein once said "Simplify as much as possible, but not more!" Administrators all want that "sound-byte" that sums up all known knowledge, answers all their problems and overcomes all objections... hence we have the public poll. The other unfortunate fact is administrators aided by sociologists formulate the questions in polls according to what they want to hear. Truth through polling is the dumbest way to seek information because scientific knowledge is complex and there is no way to get any "complete" knowledge on any subject due to Godel's incompleteness theorems forcing a fundamental limitation in all our wisdom.

As Professor Andrew Stirling recently stated ("Keep it Complex" Nature 23-30 Dec 2010 V468 p1031) "When knowledge is uncertain, experts should avoid pressures to simplify their advice. Render decision-makers accountable for decisions".

IMHO these decision-makes should study science in depth in order that their decisions are relevant and understood.
Dug
not rated yet Jan 13, 2011
The value of breaking complex questions down - not any different than breaking problems down to sub- components for more efficient solution - has been remarkably well understood for several millennia now. However, no matter how you break down a question, if the person being asked has little or no knowledge on the subject of the related question - he or she will still not be able to produce an answer of any value - because it will be baseless. Why would you want the average person making decisions regarding complex problems without the necessary expertise? This is the great and fatal weakness of democracy - the average person is never going to have expertise necessary to make decisions on anything that is more complex than average. We have the incompetent politicians that prove this - daily.

More news stories

Egypt archaeologists find ancient writer's tomb

Egypt's minister of antiquities says a team of Spanish archaeologists has discovered two tombs in the southern part of the country, one of them belonging to a writer and containing a trove of artifacts including reed pens ...

NASA's space station Robonaut finally getting legs

Robonaut, the first out-of-this-world humanoid, is finally getting its space legs. For three years, Robonaut has had to manage from the waist up. This new pair of legs means the experimental robot—now stuck ...

Ex-Apple chief plans mobile phone for India

Former Apple chief executive John Sculley, whose marketing skills helped bring the personal computer to desktops worldwide, says he plans to launch a mobile phone in India to exploit its still largely untapped ...

Filipino tests negative for Middle East virus

A Filipino nurse who tested positive for the Middle East virus has been found free of infection in a subsequent examination after he returned home, Philippine health officials said Saturday.

Airbnb rental site raises $450 mn

Online lodging listings website Airbnb inked a $450 million funding deal with investors led by TPG, a source close to the matter said Friday.