Automated method beats critics in picking great movies

January 19, 2015
Credit: Cristie Guevara/public domain

Don't rely on the Academy Awards next month if you are seeking to know whether the movies deemed great today will survive the test of time.

According to a new Northwestern University study, the best predictor of a movie's significance is how often a movie is referenced by other . In other words, a movie's significance is decided by today's and tomorrow's film directors—not the critics.

"Movie critics can be overconfident in spotting important works, and they have bias," said Luís Amaral, the leader of the study and co-director the Northwestern Institute on Complex Systems. "Our method is as objective as it gets."

Amaral also is a professor of chemical and biological engineering in Northwestern's McCormick School of Engineering and Applied Science.

He and his colleagues are the first to systematically compare different approaches for estimating a film's significance. They considered metrics for measures both subjective (critical reviews, awards, public opinion) and objective (citations, box office sales).

The researchers found their automated method of movie citations is better at predicting greatness, especially in movies 25 years old or older, than these runners-up: the expertise of movie critics (a group of critics or a single critic), the wisdom of the crowd, the numbers of awards won and the amount of box office sales, among others.

Based on this analysis, Amaral plans to also assess the true importance of scientific papers, paintings and music.

The study will be published the week of Jan. 19 by the Proceedings of the National Academy of Sciences (PNAS).

The research team conducted a big data study of 15,425 U.S.-produced films listed in the Internet Movie Database (IMDb). Specifically, they looked to see how well an approach predicted a movie's inclusion in the National Film Registry of the U.S. Library of Congress, which is akin to the National Baseball Hall of Fame.

In their analysis, the researchers found the number of times a movie 25 years or older is referenced by other movies best predicts inclusion in this registry of American films deemed "culturally, historically or aesthetically significant."

Films with the most long-gap citations that also are in the National Film Registry are "The Wizard of Oz," "Star Wars," "Psycho," "Casablanca" and "Gone With the Wind."

"Directors keep coming back to movies that are significant," Amaral said. "If you show a little bit from 'Pscyho,' such as referencing the shower scene, you are putting that whole movie in front of the viewer of the new movie."

"There is something about a movie that is hidden to us, but there are measurable things, such as critic ratings, awards and referencing by other filmmakers, that hint at this hidden element—a movie's significance," he said. "We find that ultimately it is the creators, the filmmakers themselves, who will determine which movies are important, not the expert critics."

Also important, the researchers write, is that the automated method can easily be applied to older films for which no other rating may be available.

Amaral enjoys movies, but what he really wants to do is develop a method for identifying the most significant scientific papers. Given the varied sizes of scientific fields, the sheer number of citations is not sufficient for determining greatness, he said.

"More than 1 million are published each year worldwide," Amaral said. "It can be difficult to distinguish a good scientific paper from an average one, much like the movies. My next goal is to develop a good measure of scientific citations to get inside what is going on in the scientific literature."

Explore further: Valuable movies and valued movies may be two different things

More information: "Cross-evaluation of metrics to estimate the significance of creative works" PNAS, www.pnas.org/cgi/doi/10.1073/pnas.1412198112

Related Stories

Recommended for you

Facts, beliefs, and identity: The seeds of science skepticism

January 22, 2017

Psychological researchers are working to understand the cognitive processes, ideologies, cultural demands, and conspiracy beliefs that cause smart people to resist scientific messages. Using surveys, experiments, observational ...

Major Viking Age manor discovered at Birka, Sweden

January 19, 2017

During spring of 2016 a number of large presumed house terraces were identified by the authors at Korshamn. As a consequence high resolution geophysical surveys using ground-penetrating radar were carried out in September ...

3 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

KBK
not rated yet Jan 19, 2015
Regardless, the academy awards are a component of a propaganda machine, a publicity directed rubber stamp for a small clique of irrelevant er ..people. (I say people very reluctantly)(in the potentially nominal sense.. arms, legs, and so on) (ie, only vaguely resembling humanity, like the whole idea of the trained dancing bear making herky-jerky motions that parts of most people try to label as dancing)
Rustybolts
not rated yet Jan 20, 2015
People really use movie critics? Got the chills just thinking about listening to them.
Squirrel
not rated yet Jan 20, 2015
Surely whether a movie gets included in the National Film Registry of the U.S. Library of Congress links to how much it is referenced by other movies making them dependent not independent measures. And how do you measure whether a movie is "referenced" or not by another one--will not references to films not in the Registry and so not so well known have "references" to them missed and left out of the authors statistical analysis? I been lazy and not checked the PNAS and no doubt its authors took account of these problems.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.