Views you can use? How online ratings affect your judgment

Aug 08, 2013 by Peter Dizikes
Views you can use? How online ratings affect your judgment
Credit: Christine Daniloff

Are you influenced by the opinions of other people—say, in the comments sections of websites? If your answer is no, here's another question: Are you sure?

A new study co-authored by an MIT professor suggests that many people are, in fact, heavily influenced by the positive opinions other people express online—but are much less swayed by negative opinions posted in the same venues. Certain topics, including politics, see much more of this "herding" effect than others.

The results, published today in the journal Science, detail a five-month experiment conducted on a major news-aggregation web site. The research group systematically altered the favorability ratings given to certain on the site, to see how perceptions of favorability affected people's judgment about those comments. They found that comments whose ratings were manipulated in a favorable direction saw their popularity , receiving a 25 percent higher average rating from other site users.

"This herding behavior happens systematically on positive signals of quality and ratings," says Sinan Aral, an associate professor at the MIT Sloan School of Management, and one of three authors of the study. At the same time, Aral notes, the results "were asymmetric between positive and negative herding." Comments given negative ratings attracted more negative , but that increase was drowned out by what the researchers call a "correction effect" of additional positive responses.

"People are more skeptical of negative ," Aral says. "They're more likely to 'correct' a negative vote and give it a positive vote."

While this phenomenon of social positivity sounds pleasant enough on the surface, Aral warns that there are pitfalls to it, such as the manipulation of online ratings by some political operatives, marketers or anyone who stands to profit by creating an exaggerated appearance of popularity.

"These positive ratings also represent bias and inflation," Aral says. "The was a spread of positivity, but when it burst, some people lost their savings and their houses went underwater. Stock bubbles represent a positive herding, and they can be dramatically bad in the wrong context."

Still, the experiment also revealed topical limitations in herding: Stories under the rubrics of "politics," "culture and society" and "business" generated positive herding, but stories posted under the topics of "economics," "IT," "fun" and "general news" did not.

More wisdom about crowds

In turn, Aral suggests, we should be as analytical as possible when it comes to harnessing collective judgments.

"We have to be careful about the design and analysis of systems that try to aggregate the wisdom of crowds," Aral says.

The research was conducted by Lev Muchnik of the Hebrew University of Jerusalem; Sean Taylor of New York University; and Aral, who joined MIT this summer.

The experiment was conducted on a news-aggregation site whose identity the researchers cannot disclose for legal reasons, although Aral allows that it operates along the lines of popular sites such as Reddit. Over the five-month period, the researchers randomly manipulated the ratings given to 101,281 comments to the site. In this way, they could see how readers evaluated the same comments when those comments were given different ratings.

This approach was necessary, Aral points out, because in most circumstances, "It's hard to distinguish the effect of high quality from the effect of social influence bias. It could be that past positive ratings have snowballed to create a high score, or it could just be that those items likely to get high scores are just of high quality."

The researchers also found that comments manipulated to have positive ratings were 32 percent more likely than untreated comments to receive a favorable rating from the very next viewer of those comments, and 30 percent more likely than untreated comments to obtain a very high favorable rating.

Positive ratings for the research—but more is needed

For his part, Aral agrees that the experiment "opens up as many questions as it answers." He suggests that it would also be valuable to have more work "explaining the psychology of the correction effect on the negative side," as a way of understanding how collective judgments are formed.

"Our message is not that we should do away with crowd-based opinion ," Aral says. "Our point is that you need solid science under the hood trying to understand exactly how these mechanisms work in a broad population, what that means for the diffusion of opinion, and how can we design the systems to be fair, to have less incentives for manipulation and fraud, and be safe in aggregating opinions."

Explore further: Less privileged kids shine at university, according to study

More information: "Social Influence Bias: A Randomized Experiment," by L. Muchnik et al Science, 2013.

Related Stories

Physician review websites rely on few patient reviews

Jan 02, 2013

Millions of Americans read physician ratings on websites such as Healthgrades.com, but such ratings are based on scores from an average of only 2.4 patients, a Loyola University Medical Center study has found.

Physician-rating websites are biased, study says

Jun 14, 2011

Patients posting their opinions about doctors on online ratings websites are much less likely to discuss physicians with low perceived quality and are more prone than offline populations to exaggerate their opinions, according ...

Recommended for you

Why are UK teenagers skipping school?

Dec 18, 2014

Analysis of the results of a large-scale survey reveals the extent of truancy in English secondary schools and sheds light on the mental health of the country's teens.

Fewer lectures, more group work

Dec 18, 2014

Professor Cees van der Vleuten from Maastricht University is a Visiting Professor at Wits University who believes that learning should be student centred.

How to teach all students to think critically

Dec 18, 2014

All first year students at the University of Technology Sydney could soon be required to take a compulsory maths course in an attempt to give them some numerical thinking skills. ...

User comments : 10

Adjust slider to filter visible comments by rank

Display comments: newest first

Guy_Underbridge
5 / 5 (2) Aug 09, 2013
We ignore down-vote bots... Who knew?
DarkHorse66
3.4 / 5 (5) Aug 09, 2013
Ahem!

"The experiment was conducted on a NEWS-AGGREGATION SITE whose identity the researchers cannot disclose for legal reasons, although Aral allows that it operates along the lines of popular sites such as Reddit."

Apart from the 'proper' Trolls, there have been a couple (usually with four letter names) that seem to carpet bomb certain people with inexplicable rankings fly-overs
I'm not asserting anything, but does anybody smell a RAT???

Food for Thought and Regards to all, DH66
Q-Star
3.2 / 5 (9) Aug 09, 2013
Ahem!

"The experiment was conducted on a NEWS-AGGREGATION SITE whose identity the researchers cannot disclose for legal reasons, although Aral allows that it operates along the lines of popular sites such as Reddit."

Apart from the 'proper' Trolls, there have been a couple (usually with four letter names) that seem to carpet bomb certain people with inexplicable rankings fly-overs
I'm not asserting anything, but does anybody smell a RAT???

Food for Thought and Regards to all, DH66


It sure would explain the tolerant moderation, eh?
TheGhostofOtto1923
4 / 5 (4) Aug 09, 2013
Ahem!

"The experiment was conducted on a NEWS-AGGREGATION SITE whose identity the researchers cannot disclose for legal reasons, although Aral allows that it operates along the lines of popular sites such as Reddit."

Apart from the 'proper' Trolls, there have been a couple (usually with four letter names) that seem to carpet bomb certain people with inexplicable rankings fly-overs
I'm not asserting anything, but does anybody smell a RAT???

Food for Thought and Regards to all, DH66
I for one am very happy to be a direct participant in the scientific process. For posterity. Besides ingesting prescription drugs and eating engineered foods.
Frilla_Poo
3.5 / 5 (11) Aug 12, 2013
I for one am very happy to be a direct participant in the scientific process.

Much more so than you might think.

As a sockpuppet who really doesn't care (decode my anagram origins) I have observed the described behaviors. My original identity initially garnered and maintained very high ratings here, mostly due to carefully considered initial comments. A good reputation having been established, I began to notice some of my lesser thought-out remarks drew unusually good ratings also.

Curious, I created a second, more fragile identity here for which I gathered lower ratings. Once that alter-ego was was well established, I posited similar opinions at times using both identities. My initial identity was universally ranked higher than my second even as the remarks and opinions espoused remained same for both.

You, Otto, repeatedly trashed my second identity many times in comments while NEVER attacking my original ID. The opinions I expressed were the same.

Interesting research.
cantdrive85
2.2 / 5 (10) Aug 12, 2013
I have at least four shadows, seemingly every comment. The first initials of those four are B, O, T..., and K. The KBOT down voters.
DarkHorse66
3.4 / 5 (5) Aug 12, 2013
How predictable! My earlier comment was promptly downrated by 'toot'!
@Frilla Poo: I haven't created a second account, but I've noticed the same trend that you have. I used to have a good ranking (4.2 at least), until the four letter crank rankers started in on me. I rather suspect that they have even dug out old posts just to downrate them. When I reread some old, subscribed posts a while back, I noticed that many of my old post ratings suddenly appeared mysteriously lower than they had been at the time. This was at the same time as my overal ranking equally mysteriously dropped quite literally overnight by several subpoints. I just wish that phys.org would actually follow through with dealing properly with the trolls. Otherwise all those fancy rules that they have for this site are just so many fancy words. No wonder trolls have such a field day here.
Best Regards, DH66
Frilla_Poo
3.5 / 5 (11) Aug 12, 2013
I might add, this mirthful personality is NOT the second experimental identity I was referring to above, and no alter ego of mine has ever been used as a down-vote mechanism. I will say, however, I was initially motivated by the annoyingly incessant appearance of the immature Lite/Toot/Open cabal. They have obviously fixations that point to deficiencies of purpose.

In the end, criticism says more about the criticizer than it does you. There is more to life and we must get over it.

Sincerely... Polar_Foil
or maybe... Fra_polloi
hmmm....... Floor_Pail
could be.. Profal_Oil

I'm just a fool.
DarkHorse66
3 / 5 (4) Aug 12, 2013
Incidentally, my ranking has gone down again, by 0.1, just since Friday (it's now Monday here), which was the last time I was at a computer. Yet the amount of downvoting required to counter the 5's that I've be getting, is not actually visible. So I can only assume that there is more systematic downvoting of old posts going on. And I would also assume that it is pretty safe to say that I am not the only one that this happening to. Unfortunately there does not appear to be a point in emailing the moderators to 'moderate' on this issue. You just get ignored.
Cheers, DH66
Solidproof_Layman
3.5 / 5 (11) Aug 12, 2013
I see proof, ere I spoof.

FP - You are a wise anagram of a man. Obviously the weight of your wisdom was the bait that drew out the down-vote troll currently residing in this spate of comments.

Surely proof of residence in a sad and lonely continuum.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.