BES reveals longstanding flaw in opinion polls

January 25, 2016

Flaws which made the opinion polls wrong in 2015 also explain errors in 2010, new research from the British Election Study (BES) reveals.

BES researchers Dr Chris Prosser from The University of Manchester and Dr Jon Mellon from The University of Oxford last year revealed that sampling was the reason for the 2015 polling miss, confirmed in Tuesday's Independent Inquiry preliminary report.

Now they show the same mechanism of over-representing groups such as young people who were less likely to vote, is also likely to have been at least partially responsible for the 2010 over-estimation of the Liberal Democrats' vote share.

The flaw, they say, was also likely to have been present in the 2001 and 2005 general elections, but the electoral maths meant the polls correctly forecast which party would form the Government, despite overestimating Labour's share.

Dr Mellon said, "We have shown the 2015 polls were fundamentally unrepresentative of the electorate and that Labour leaning respondents were the most over-represented as voters in the sample.

"Our new research shows that this problem is a longstanding issue: in 2010 these over-represented respondents were Liberal Democrat voters, which could potentially explain the 2010 polls' over-estimation."

In 2010, the polls over-estimated the Liberal Democrats by around 4.0 percentage points, but no reason was found for the error.

The BES findings now suggest the error was down to sampling voters from demographic groups with low turnout demographics, who tended to support the Liberal Democrats.

Dr Chris Prosser said: "The surprising surge of support for the Liberal Democrats among low-turnout demographic groups, such as students, may in fact have saved the pollsters from a 2015 polling miss in 2010."

The academics have devised a way of correcting the flaw, so pollsters can in future create more accurate forecasts by factoring in additional weighting to account for expected proportions of non-voters.

By adding an additional weighting factor of the expected proportion of non-voters, respondents from demographics which vote less would be more represented in the non-voting section of the general population sample.

That means they will be less represented as voters – improving accuracy.

"While we do not know exactly what the turnout will be in a given election, we do know that it is likely to be reasonably close to the previous election's turnout," said Dr Mellon.

"But further work is needed to determine whether this method will be sufficient."

The BES face-to-face survey, carried out by GfK between May 2015 and September 2015, was a random probability sample of 2,987 people.

It showed a closer match to the 2015 general election Conservative lead over Labour than the pre-election polls, which failed to forecast the scale of the Conservative victory.

Analysis of the data by the BES team suggests that this closer match was made possible by using probability-based sampling methods and successful targeting of hard-to-reach groups, including non-voters.

Dr Mellon said: "We believe these findings have serious implications for pollsters both in the UK and in other countries.

"Weighting to population targets and then removing non-voters is by far the most dominant election polling approach, and any pollster that uses it risks the same kind of polling miss that Britain saw in 2015.

"Given the increasing number of polling misses throughout the world, we may already be seeing the results of this methodological choice."

Explore further: Epic fails in 2015 elections: Does polling jeopardize fairness?

Related Stories

Polls show deep partisan divide over Affordable Care Act

October 29, 2014

A comprehensive analysis of data from 27 public opinion polls conducted by 14 organizations, including a poll in September of those most likely to vote, shows an electorate polarized by political party when it comes to the ...

Recommended for you

3 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

Squirrel
not rated yet Jan 25, 2016
So it was not pollsters "Putting A Thumb On The Scale" as suggested by Nate Silver http://fivethirty...e-scale/ http://www.telegr...ied.html
antialias_physorg
not rated yet Jan 25, 2016
The academics have devised a way of correcting the flaw, so pollsters can in future create more accurate forecasts by factoring in additional weighting to account for expected proportions of non-voters.

I'm a bit in two minds about this.
Pro: Making polls more accurate is good
Con: Introducing a fudge factor can render any result one wishes since the fudge factor is not based on the current poll but previous polls. (Also the one-person-one-vote principle gets negated)

While I realize that polls are always heavily biased towards people who like to participate in polls I feel that doing such weighting might undermine the entire point of taking polls in the first place.
SamB
not rated yet Jan 25, 2016
If many people like myself either refuse the do a survey or lie on key questions, then their polls may be almost useless anyway. I do not believe that accurate forecasts are in our best interests since nefarious leaders can use this data to undermine voter wishes.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.