Online survey researchers should be cautious with trick questions

May 27, 2015 by Jared Wadley

New studies suggest that researchers should be careful with setting "trap" questions for respondents for the sake of accurate research.

Researchers use trap questions (also known as attention checks and instructional manipulation checks) to assess whether participants are paying attention to the instructions. However, most participants see these trick questions and become cautious about their answers, potentially altering a study's results.

A pair of University of Michigan studies show that these instructional manipulation checks, or IMCs, which are popular measures used by researchers for online surveys, have unforeseen effects.

David Hauser, a U-M doctoral candidate, and colleague Norbert Schwarz of the University of Southern California, found that answering a trap question changes the way people respond with later questions.

"IMCs cause participants to think harder when answering survey questions than they normally would in order to avoid potentially being tricked again," Hauser said.

IMCs in surveys look like normal questions. However, hidden in a large block of instructions are specific commands that tell participants to ignore the specific question and to submit a non-intuitive response instead. Participants who miss those special instructions and answer the question as normal are considered not being attentive.

For instance, an IMC might involve this scenario: Under the topic "sports participation," respondents are asked, "Which of these activities do you engage in regularly?" followed by a list of sports to select. However, above the question, embedded within a block of instructions a command indicates that, in order to demonstrate attention, respondents should click the "other" option and enter "I read the instructions" in the corresponding text box. Following these instructions is scored as passing the trap.

In the first U-M study, participants received a trap question and the Cognitive Reflection Test, a math test assessing how the analytical abilities of participants. Half of the participants completed the trap question before the math test, whereas the other half completed the math test first.

Hauser and Schwarz found that completing a trap question first increased participants' analytical thinking scores on the .

For the second study, participants received a trap question and a reasoning task assessing biased thinking. Again, half of the participants completed the trap question before the reasoning task, whereas the other half completed the reasoning task first.

The researchers found that completing the trap question first decreased biased thinking and led to more correct answers. Thus, completing a trap question made participants think more systematically about later questions.

Hauser believes many social scientists have used trick in their work, which have possibly affected their results. He says that while sometimes deeper thinking may be desirable for ensuring that participants are paying attention, it also could lead to study results that might not otherwise occur when are thinking as they normally would in everyday life.

Explore further: Internet searches create illusion of personal knowledge, research finds

More information: "It's a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on 'Tricky' Tasks." DOI: 10.1177/2158244015584617

Related Stories

Challenging work tasks may have an upside for the brain

April 29, 2015

Professionals whose jobs require more speaking, developing strategies, conflict resolution and managerial tasks may experience better protection against memory and thinking decline in old age than their co-workers, according ...

Oncologist recs plus motivation package increases exercise

May 20, 2015

(HealthDay)—For breast and colorectal cancer survivors, the level of exercise participation is significantly increased for those receiving an oncologist's exercise recommendations with an exercise motivation package, according ...

Artful dodgers: Responding but not answering often undetected

May 5, 2011

How can some people respond to a question without answering the question, yet satisfy their listeners? This skill of "artful dodging" and how to better detect it are explored in an article published by the American Psychological ...

Intuitive thinking may influence belief in God

September 20, 2011

Intuition may lead people toward a belief in the divine and help explain why some people have more faith in God than others, according to research published by the American Psychological Association.

Recommended for you

Identifying major transitions in human cultural evolution

July 26, 2017

Over the past 10,000 years human cultures have expanded from small groups of hunter-gatherers to colossal and complexly organized societies. The secrets to how and why this major cultural transition occurred have largely ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.