Fewer willing to participate in surveys

The percentage of respondents to many important research surveys is dropping sharply. Social science researchers are concerned that survey samples can be skewed, which can compromise the validity of results.

For 40 years the Norwegian Institute for Alcohol and Drug Research (SIRUS) conducted an annual survey of young people aged 15–20. The responses provided researchers and the authorities with insight into drug and alcohol use in this age group in Norway. But in 2009 the surveys came to a halt.

"At that point the response rate had fallen from roughly 70 per cent to just over 30 per cent," says Ingeborg Rossow, a senior researcher at SIRUS.

According to the researchers, such a low response rate could cast doubt on the comparability of the results over time. The alcohol and drug use survey was showing that young people in Norway in recent years were drinking significantly less alcohol compared to young people in the 1980s and 1990s. But what if these figures reflected a situation in which only the "good" pupils were answering the survey, making results unreliable? The drug and alcohol researchers did not believe this to be the case, but they could not be certain.

"We now face a similar problem when it comes to surveys about alcohol use amongst the adult population," says Dr Rossow. "There, too, we must ask ourselves whether the responses we get are representative of the population at large."

Eight percent responding

Ottar Hellevik, a professor of political science at the University of Oslo, is a nestor in social science-related methodology. He heads up the Norsk Monitor survey, which measures changing values in the Norwegian population. Professor Hellevik says that social scientists from years past would be shocked at today's response rates.

"In the Norsk Monitor survey, roughly one-fourth of people we ring up will actually participate," explains the professor. "When we follow up those telephone surveys with a written survey, we are left with a full response rate of around eight per cent. This is far lower than in the 1980s."

"We have to ask ourselves whether the sample of individuals who agree to take part in the survey is adequately representative. Actually, there is much that suggests that the refusals are quite random, which gives us reason to believe we can still use the results we get."

Professor Hellevik is certain that serious surveys suffer from the public's exposure to too many marketing surveys.

"Surveys are an important instrument for social science research in general. If this instrument starts to become compromised, we may need to introduce some form of regulation for who is allowed to conduct surveys," he proposes.

Young males respond least, older women most

"Response rates to surveys began declining suddenly here in Norway around five to six years ago," says Kirstine Kolsrud, a section head at Norwegian Social Science Data Services AS (NSD) in Bergen. NSD assists the research sector with a range of services including collecting data.

In general, NSD sees that young, single men living in urban areas are the least likely to respond, while older women are the most willing. Patterns vary, however, depending on the survey topic. "When we ask people about health issues," says Ms Kolsrud, "we can still get answers from quite a large proportion of those we contact."

NSD contacts people by sending them a survey form in the post, which they may also answer using the Internet. Whereas NSD could once count on a response rate of 60 to 70 per cent of those asked to participate, that figure has now dropped to 40 to 50 per cent. "Officially our target rate is still 70 per cent, but today we know this is unrealistic." Nonetheless, NSD is confident its survey responses are from a representative sample.

Worrisome for health research

"We see a downward trend in response in health-related research, too," notes Camilla Stoltenberg, Director General of the Norwegian Institute of Public Health in Oslo. "But in no way do we get the impression that people are less positive about the actual research being done."

"The decline is most likely due to people reaching a saturation point with too many surveys. Everyone is asking for market feedback, from the hotel you last stayed at to the online bank you use."

Dr Stoltenberg is concerned that the lower participation in health surveys can negatively affect health care services, since health researchers and the health care system often have to depend on people's feedback to further develop these services.

"At the Norwegian Institute of Public Health, we see a steady rise in the need for updated knowledge. One example involves surveys on the quality of health care services, which is a topic currently under close political scrutiny. To be able to assess quality, we are dependent on high participation in our surveys. If we need feedback on, say, whether a new surgery method is better or worse than the current method, we need responses from as many of the patients who have undergone the operations as possible – and preferably from all of them."

International trend

The problem of falling response rates to surveys is not specific to Norway. NSD assists in conducting the European Social Survey (ESS), an ongoing face-to-face interview survey to quantify socio-economic issues, values, attitudes and policies in most European countries. There have been major changes in response rates in many countries.

"Traditionally, Norway and the other Nordic countries have had fairly high response rates," says Kirstine Kolsrud. "But now the Eastern European countries have the highest participation, with 70 per cent or higher in countries such as Poland and the Czech Republic. At the top is Bulgaria with 81 per cent. Norway is in the middle ranks at 58 per cent. The lowest rate is in Germany, at 30.5 per cent."

A good deal of international research is being carried out on response rates in social science research and how this can affect research results. What characterises those who do not respond? Are the samples of respondents distinctly skewed? Researchers largely agree that while high participation does not guarantee a representative sample group, it does raise the likelihood that the sample represents the intended target group.