Analyzing almost 10 million tweets, research finds public mood can predict Dow days in advance

Oct 19, 2010
A graph of Dow Jones Industrial Average values (center, blue) and tweets identified with a "calm" mood during a time series (bottom, red) running three days prior are overlaid in the top graph to show gray areas of significant overlap.

Measurements of the collective public mood derived from millions of tweets can predict the rise and fall of the Dow Jones Industrial Average up to a week in advance with an accuracy approaching 90 percent, Indiana University information scientists have found.

Researchers at IU Bloomington's School of Informatics and Computing found the correlation between the value of the Dow Jones Industrial Average (DJIA) and public sentiment after analyzing more than 9.8 million tweets from 2.7 million users during 10 months in 2008.

Using two mood-tracking tools to analyze the text content of the large-scale collection of feeds, Associate Professor Johan Bollen and Ph.D. candidate Huina Mao were able to measure variations in public mood and then compare them to closing stock market values.

One tool, OpinionFinder, analyzed the tweets to provide a positive or negative daily time series of public mood. The second tool, Google-Profile of Mood States (GPOMS), measured the mood of tweets in six dimensions: calm, alert, sure, vital, kind, and happy. Together, the two tools provided the researchers with seven public mood time series that could then be set against a similar daily time series of Dow Jones closing values.

The researchers then correlated the two sets of values -- Dow Jones and public mood -- and used a self-organizing network model to test a hypothesis that predicting stock market closing values could be improved by including public mood measurements.

"We were not interested in proposing an optimal Dow Jones prediction model, but rather to assess the effects of including public mood information on the accuracy of the baseline prediction model," Bollen said. "What we found was an accuracy of 87.6 percent in predicting the daily up and down changes in the closing values of the Dow Jones Industrial Average."

By implementing a called a Self-Organizing Fuzzy Neural Network (SOFFNN) similar to one already used to successfully forecast electrical load needs, the researchers were able to demonstrate that public mood had the ability to significantly improve the accuracy of the most basic models currently in use to predict Dow Jones closing values. Bollen described this particular SOFFNN as a five-layer hybrid neural network with the ability to self-organize its own neurons during a learning process that included information of past Dow Jones and public mood time series values.

"Given the performance increase for a relatively basic model such as the SOFNN, we are hopeful to find equal or better improvements for more sophisticated market models that may in fact include other information derived from news sources and a variety of relevant economic indicators," he said.

The researchers found the OpinionFinder positive/negative sentiment input had no effect on prediction accuracy, while the Calm and the Calm-Happy combination of the GPOMS had the highest prediction accuracy.

"In fact, the calmness index appears to be a good predictor of whether the goes up or down between two and six days later," Bollen said.

The odds of the prediction accuracy rate of 87.6 percent being sheer chance were then calculated for a random period of 20 days and determined to be just 3.4 percent.

Explore further: Hyperbolic homogeneous polynomials, oh my!

More information: The research paper is available for download here.

Related Stories

SAP posts better-than-expected net profit

Jul 29, 2009

SAP, the German professional software giant, posted a better than expected second quarter profit on Wednesday, and said it would maintain tight cost controls for the remainder of the year.

Happy extraverts are more creative: study

Aug 03, 2010

(PhysOrg.com) -- Outgoing people who are in a good mood are significantly more creative than people who keep themselves to themselves, according to a new study.

Bad mood, better recall, researchers find

Apr 11, 2009

People grumbling their way through the grimness of winter have better recall than those enjoying a carefree, sunny day, Australian researchers have found.

Recommended for you

Not just the poor live hand-to-mouth

4 hours ago

When the economy hits the skids, government stimulus checks to the poor sometimes follow. Stimulus programs—such as those in 2001, 2008 and 2009—are designed to boost the economy quickly by getting cash ...

Math modeling handbook now available

7 hours ago

Math comes in handy for answering questions about a variety of topics, from calculating the cost-effectiveness of fuel sources and determining the best regions to build high-speed rail to predicting the spread ...

Archaeologists, tribe clash over Native remains

7 hours ago

Archaeologists and Native Americans are clashing over Indian remains and artifacts that were excavated during a construction project in the San Francisco Bay Area, but then reburied at an undisclosed location.

Male-biased tweeting

9 hours ago

Today women take an active part in public life. Without a doubt, they also converse with other women. In fact, they even talk to each other about other things besides men. As banal as it sounds, this is far ...

Developing nations ride a motorcycle boom

11 hours ago

Asia's rapidly developing economies should prepare for a full-throttle increase in motorcycle numbers as average incomes increase, a new study from The Australian National University has found.

User comments : 8

Adjust slider to filter visible comments by rank

Display comments: newest first

SincerelyTwo
not rated yet Oct 19, 2010
Ah, of course, the solution to the recession is making people happy!

... but how do we make them happy? By giving people more money. But on the road through a recession there are fewer jobs available... less money to give. Crap taco.
ErickS
4 / 5 (1) Oct 19, 2010
Ah, this is a prediction algorithm system based upon a variant of the wisdom of the crowds concept. Usually systems like this only work during a specific time interval since all kinds of feedback loops kick in after a certain amount of time.
CarolinaScotsman
not rated yet Oct 19, 2010
I still like the one that says the Dow rises and falls with hem lines. Gives me an excuse to look.
droid001
not rated yet Oct 19, 2010
"can predict the Dow Jones Industrial Average with an accuracy approaching 90 percent"
So Indiana University scientists already millionaires?
Quantum_Conundrum
not rated yet Oct 19, 2010
"can predict the Dow Jones Industrial Average with an accuracy approaching 90 percent"
So Indiana University scientists already millionaires?


Being able to predict the average wouldn't necessarily allow you to make money, except possibly by buying and selling Gold repeatedly.

If the DOW average is expected to be up, then normally Gold would be expected to be down. If the average is expected to be down, then Gold would normally be expected to be up.

Else, they might profit marginally by investing equally in all DOW companies if they think it is supposed to rise.

You can predicte the average based on mob mentality, but you can't necessarily predict changes in indiviual companies.
madrespek
not rated yet Nov 14, 2010
@conundrum

you can just buy an ETF for the Dow Jones, or S&P 500, whichever index you prefer. An ETF is similar to an ordinary share, but represents holding all the companies in a given index with the weights prescribed by that index. So in effect, you can buy the Dow very easily.
pokerdice1
1 / 5 (1) Nov 15, 2010
and what about Doe futures?
pokerdice1
1 / 5 (1) Nov 15, 2010
Having looked at the charts above, I think that, the chief difficulty would be in sizing and timing exits... as usual.

More news stories

Not just the poor live hand-to-mouth

When the economy hits the skids, government stimulus checks to the poor sometimes follow. Stimulus programs—such as those in 2001, 2008 and 2009—are designed to boost the economy quickly by getting cash ...

Male-biased tweeting

Today women take an active part in public life. Without a doubt, they also converse with other women. In fact, they even talk to each other about other things besides men. As banal as it sounds, this is far ...

Archaeologists, tribe clash over Native remains

Archaeologists and Native Americans are clashing over Indian remains and artifacts that were excavated during a construction project in the San Francisco Bay Area, but then reburied at an undisclosed location.

Math modeling handbook now available

Math comes in handy for answering questions about a variety of topics, from calculating the cost-effectiveness of fuel sources and determining the best regions to build high-speed rail to predicting the spread ...