A good read: AI evaluates quality of short stories

August 21, 2017, Disney Research

The idea that artificial intelligence will someday be able to understand and even generate narratives has inspired and motivated researchers for years. A question inextricably bound to both lines of research remains unresolved, however: Can AI recognize a good story if it sees one?

In the first large-scale attempt to tackle this challenging problem, scientists at Disney Research and the University of Massachusetts Boston have developed neural networks that can evaluate short narratives. These AIs don't try to mimic the professional judgment of literary critics, but attempt to predict which stories will appeal to a large population.

"Our neural networks had some success in predicting the popularity of stories," said Boyang "Albert" Li, a research scientist at Disney Research. "You can't yet use them to pick out winners for your local writing competition, but they can be used to guide future research."

The team will present its findings Aug. 23 at the International Joint Conference on Artificial Intelligence in Melbourne, Australia.

"The ability to predict narrative quality impacts on both story creation and story understanding," said Markus Gross, vice president at Disney Research. "To evaluate quality, the AI needs some level of understanding of the text. And if AIs are to create narratives, they need to be able to judge the quality of what they are producing."

One of the challenges in developing automated evaluation of story quality is the lack of large databases of stories that have been evaluated by humans and can be used to train AIs, noted Tong Wang, a computer science Ph.D. student at UMass Boston and a lab associate at Disney Research. The scientists found the question and answering website Quora, to be a good data source, as many of its answers are in the form of stories, he added. Reader upvotes, likewise, measure a story's popularity and serve as a proxy for narrative quality.

After gathering almost 55,000 answers, the researchers developed an algorithm to classify them as either stories or non-stories, yielding more than 28,000 stories with an average of 369 words.

To understand the complex semantics of the stories, the research team looked for ways to represent the influence of story structures in because a sequence of events can interact to reveal character intentions.

The researchers developed an AI that evaluated separate regions of each story—including the question that prompted each. They created a network that looked at the regions interdependently and another network that took a holistic view, looking for how the meaning of the events and story regions emerged from the entire story. In each case, the AIs made predictions of which texts would prove most popular with readers.

In experiments, the neural nets showed improvement over a baseline text evaluation system, with the holistic neural network registering an 18 percent improvement.

In addition to Li and Wang, Ping Chen, Wang's PhD adviser and associate professor of computer science and engineering at UMass Boston, contributed to this research.

Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow.

Explore further: More than animation: Software supports animated storytelling

Related Stories

More than animation: Software supports animated storytelling

November 17, 2016

Disney Research has developed new tools to help people use animation to tell stories by eliminating distracting details that hamper creativity, suggesting ways to fill holes in plots and assisting in the creation of virtual ...

Fiction books narratives down to six emotional story lines

November 21, 2016

Our most beloved works of fiction hide well-trodden narratives. And most fictions is based on far fewer storylines than many have imagined. To come to this conclusion, big data scientists have worked with colleagues from ...

Computers using linguistic clues to deduce photo content

July 21, 2017

Scientists at Disney Research and the University of California, Davis have found that the way a person describes the content of a photo can provide important clues for computer vision programs to determine where various things ...

Recommended for you

Google braces for huge EU fine over Android

July 18, 2018

Google prepared Wednesday to be hit with huge EU fine for freezing out rivals of its Android mobile phone system in a ruling that could spark new tensions between Brussels and Washington.

EU set to fine Google billions over Android: sources

July 17, 2018

The EU is set to fine US internet giant Google several billion euros this week for freezing out rivals of its Android mobile phone system, sources said, in a ruling that risks fresh tensions with Washington.

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Aug 21, 2017
These AIs don't try to mimic the professional judgment of literary critics, but attempt to predict which stories will appeal to a large population.

In which case they're mimicking the "professional judgement" of Amazon's recommendation AI that dictates what stories the public are exposed to.

It's the same deal as with popular music: people overall and on average buy and listen to what they know exists, which is down to how much money you spend on advertising, buying your way up the charts, buying up radio station playlists, buying spots on movies and television - or buying the radio/tv station and the movie studio itself...

So popularity is a poor metric of merit, because most people don't know of anything better.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.