Team finds new method to improve predictions

November 30, 2016
Credit: Public Domain

Researchers at Princeton, Columbia and Harvard have created a new method to analyze big data that better predicts outcomes in health care, politics and other fields.

The study appears this week in the journal Proceedings of the National Academy of Sciences.

In previous studies, the researchers showed that significant variables might not be predictive and that good predictors might not appear statistically significant. This posed an important question: how can we find highly predictive variables if not through a guideline of statistical significance? Common approaches to prediction include using a significance-based criterion for evaluating variables to use in models and evaluating variables and models simultaneously for prediction using cross-validation or independent test data.

In an effort to reduce the error rate with those methods, the researchers proposed a new measure called the influence score, or I-score, to better measure a variable's ability to predict. They found that the I-score is effective in differentiating between noisy and predictive variables in and can significantly improve the prediction rate. For example, the I-score improved the rate in breast cancer data from 70 percent to 92 percent. The I-score can be applied in a variety of fields, including terrorism, civil war, elections and financial markets.

"The practical implications are what drove the project, so they're quite broad," says lead author Adeline Lo, a postdoctoral researcher in Princeton's Department of Politics. "Essentially anytime you might be interested in predicting and identifying highly predictive variables, you might have something to gain by conducting variable selection through a statistic like the I-score, which is related to variable predictivity. That the I-score fares especially well in high dimensional data and with many complex interactions between variables is an extra boon for the researcher or policy expert interested in predicting something with large dimensional data."

Explore further: Scientists devise new method to solve significant variables conundrum

More information: Adeline Lo et al, Framework for making better predictions by directly estimating variables' predictivity, Proceedings of the National Academy of Sciences (2016). DOI: 10.1073/pnas.1616647113

Related Stories

New tool may predict survival after gunshot wounds

October 26, 2016

Researchers have developed a first-of-its-kind tool to help predict a person's chance of survival after a gunshot wound in the head or other penetrating injury, according to a new study published in the October 26, 2016, ...

Predicting avoidable 30-day readmissions

March 7, 2016

In an international, multicenter study researchers have found that the HOSPITAL score, an acronym created to identify the variables associated with hospital readmissions, successfully predicted patients at high risk of a ...

Recommended for you

New method analyzes corn kernel characteristics

November 17, 2017

An ear of corn averages about 800 kernels. A traditional field method to estimate the number of kernels on the ear is to manually count the number of rows and multiply by the number of kernels in one length of the ear. With ...

Optically tunable microwave antennas for 5G applications

November 16, 2017

Multiband tunable antennas are a critical part of many communication and radar systems. New research by engineers at the University of Bristol has shown significant advances in antennas by using optically induced plasmas ...

2 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

burrelleb
not rated yet Dec 01, 2016
So, the question is, would this technique have accurately predicted Trump winning the republican nomination and the US Presidential election?
gkam
Dec 01, 2016
This comment has been removed by a moderator.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.