Research eyes optimized design for web, mail surveys

Singular-mode mail and telephone surveys are going the way of the landline—slowly but surely being phased out in favor of more technology-assisted communication.

Once considered the gold standard for surveys, mail and phone questionnaires now largely include a web-based component. These mixed modalities have given survey researchers new freedom in questionnaire design and lessened costs, but how do they affect data quality? University of Nebraska–Lincoln scientists in the Department of Sociology and the Bureau of Sociological Research are examining that question, and a new study illustrates that the design of mixed modality surveys can increase data quality.

The novel study, published in the International Journal of Market Research, suggests that using item-by-item displays, which ask single questions with possible answers, rather than grid displays, or groupings of questions and answers in a block, in both mail and web surveys decreases the "straightlining" effect, or respondents choosing the same answer repetitively.

The study also found that using wide bar designs, which are more typical in web surveys, in both mail and web surveys made little difference in response rates and data quality compared to the radio buttons—small fill-in circles—that are much more common in mail-based surveys. The findings also suggest that questionnaire designers can combine radio buttons on mail surveys and wide bars on web surveys with little impact on data quality.

"The prevailing wisdom for questionnaire design with multiple modes is to use what's called a unified mode design, which is to make things look as similar as possible across the modes," said Kristen Olson, lead author of the study and director of the Bureau for Sociological Research. "This shows a different approach, or an optimized mode design, where you optimize the design for each mode, can get better quality answers within each mode and therefore minimize measurement error."

Olson said the findings are good news for survey researchers. More and more large sociological and political surveys have had to include more web components as technology changes, and a mixed modality survey is likely to include more varied demographic groups.

"The population that does web surveys are typically younger, more educated and have internet access," Olson said. "With mail, you get those people, too, but you also tend to pick up older people who don't have internet access or a smartphone. If you mix mail and web together, you get a more representative group of respondents. That's well established."

Olson, who served on a task force for the American Association for Public Opinion Research examining these data quality questions, said most of the previous research on how battery questions in self-administered surveys are designed has almost exclusively looked at web-based surveys, since mail surveys have historically been formatted in grid styles to lessen the cost per page. With the use of web-based components in addition to mail surveys, this new research is among the first to look at both web and mail surveys together. And it suggests using item-by-item questions and answers in both modalities would improve data quality.

"Item-by-item was better for web surveys and for mail surveys," Olson said. "Even though it took up more space on a mail survey, it didn't seem to increase the perceived burden for the respondent. Taking those questions out of grids will increase the quality of data we get."

Olson acknowledged that item-by-item formatted surveys may cost more to produce but said the upshot is having better quality data.

"In this world that we're moving into, where more and more of our surveys are having multiple modes of data collection, it's really important to know that it can be worth the extra money to have an additional page in the mail questionnaire. It costs more money, but research is showing that it will likely give you a better data."