Social marketers catch cheating on online surveys, casting doubt on compensated internet research

A warning from researchers at the University of South Florida: Scam artists are taking advantage of online surveys that pay for participation—a method of market research that has become more common practice since the start of the COVID-19 pandemic.

This conclusion stems from a study that began just before the start of the pandemic in 2020. The Florida Department of Health Bureau of Tobacco Free Florida funded the USF College of Public Health and Muma College of Business to evaluate anti-tobacco public service announcements using neuromarketing measures. The researchers planned to monitor participants as they watched the PSAs using electroencephalograms, sensors that measure hand sweat and heart rate and computer software that tracks eye movements and facial expressions.

But like many aspects of research, the COVID-19 pandemic forced the USF team to pivot its approach to an "at home" setting. They and the lab's technology partner, iMotions, reconfigured the study and the software, limiting it to facial expression and attention analysis via webcam. Participants were also asked about the effectiveness of the PSAs, producing what's known as a perceived effectiveness (PE) score. Many agencies and institutions rely on PE scores to identify which PSAs would be most persuasive in specific media markets.

In a study published in the journal Social Marketing Quarterly, the researchers report that the facial expression data showed a large percentage of participants may not be who they claimed to be and that their data could have dramatically shifted the results.

Participants were required to be a current or recent tobacco user, Florida resident, and have access to a web camera. They were recruited using two methods: Through community partners sharing digital flyers and through a professional panel recruiter. The participants were offered a cash or other incentive to watch three of 12 PSAs in random order.

Despite signing a consent form that explained their faces would be recorded, 42 of the 92 responses from the "community" group were fake, either using stock photos in place of their faces or taking the survey multiple times with different IP addresses. Even the 409 responses from the vetted panel recruiter included 45 that deployed deceitful tactics, such as uploading a video of a person watching something on a screen or they demonstrated lack of engagement, which was calculated by the participant's head position.

"This research presents a cautionary tale about compensated online surveys," said co-author Rob Hammond, marketing instructor and director of the Center for Marketing and Sales Innovation at USF. "In addition to deliberate deceitful behavior in both the community and vetted panel groups, the lack of attention suggests individuals may approach online surveys 'as a job' resulting in less effort and attention."

These findings show the impact fraudulent data can have on a study's results. When rank-ordering the PSAs by PE score, six of the 12 would have been misjudged. More critically, two of the three top PSAs selected by the valid respondents had Black lead characters. Those same PSAs were not ranked in the top three based on the sample that included deceitful responses.

"Public health agencies strive to develop culturally attuned health messages. It's essential to test these messages with representatives of the intended audience," said principal investigator Claudia Parvanta, professor and social marketing concentration lead. "We've learned the hard way that we need to confirm that study participants reflect the desired cultural background but also have sufficient interest in the subject under study."

With marketing surveys increasingly moving online, the researchers suggest that compensation should only be provided to respondents after the data have been verified.