Software automatically transforms movie clips into comic strips

March 14, 2012 by Lisa Zyga feature
A comic page generated from a movie clip of “Titanic.” Image credit: Wang, et al. ©2012 IEEE

( -- While some comics today are still drawn by hand, many modern cartoonists use a variety of digital tools to create comics. But even with the help of these tools, creating comics is a time-consuming task that requires many human hours of work. In a new study, a team of researchers has designed a program that can automatically transform movie scenes into comic strips, without the need for any human intervention.

Meng Wang, a professor at the Hefei University of Technology, , and coauthors have published their study on the new software, which they call “Movie2Comics,” in a recent issue of IEEE Transactions on Multimedia.

“The software can be useful for both professionals and hobbyists,” Wang told “Professionals can directly use the software to generate (or integrate their interaction to achieve more impressive results); hobbyists may have interests to try it to see what will be generated from different movie clips.”

As the researchers explain in their study, previous programs have been developed to assist cartoonists in converting movies into comics, but the new method is the first fully automated approach. Without the need for any manual intervention, the method has the potential to significantly cut the time and expense associated with creating comics.

The new cartoonization process involves several steps, including an automatic script-face mapping algorithm that identifies the speaking character in scenes with multiple characters, automatic generation of comic panels of different sizes, positioning word balloons, and rendering movie frames in a cartoon style.

The researchers used the new method to transform 15 movie clips into . The clips came from three movies – “Titanic,” “Sherlock Holmes,” and “The Message” – and varied in length from 2 to 7 minutes. Although the method performed the transformation well for the most part, it sometimes put word bubbles next to the faces of incorrect characters. The script-face mapping algorithm had an accuracy of 85%, which the researchers hope to improve.

A comic page generated from a movie clip of “The Message.” Image credit: Wang, et al. ©2012 IEEE

The researchers then conducted a user evaluation to see how well users understood and enjoyed the comics. They found that users had a slightly lower content comprehension of the comics compared with the original movie clips. Although some loss in comprehension is inevitable when converting to a different format, the script-face mapping errors also played a role. As for enjoyment, the user evaluations showed that enjoyment was highest when the comics included word balloons and effective layout and stylization.

Although the technique is capable of performing all steps automatically, the researchers noted that involving some human effort could lead to even better results. In such a scenario, the software would provide recommendations for each step of the transformation process, and humans could manually adjust the results much more quickly and efficiently than in pure manual methods. But the researchers also hope to improve the accuracy of the automated method.

“We have two future plans,” Wang said. “One, improve the performance of each component, such as script-face mapping, and hope we can generate perfect clips without user interaction. Two, integrate speech recognition technology to generalize the software, such that we can generate comics without movie scripts (currently we need the script file of the movie clip, but by using speech recognition we can automatically transform speech to texts).”

Explore further: For improving early literacy, reading comics is no child's play

More information: Meng Wang, et al. “Movie2Comics: Towards a Lively Video Content Presentation.” IEEE Transactions on Multimedia. DOI: 10.1109/TMM.2012.2187181


Related Stories

PHD Comics hits the big screen

September 16, 2011

( -- If you are a graduate student, you are more than likely aware of the popular Piled Higher and Deeper, or PHD, Comics created by Jorge Cham. These comics cover the everyday struggles that scientists face while ...

Researchers hope to build universal human age estimator

December 5, 2011

( -- As humans, we have a knack for estimating another person’s age quite accurately just by glancing at their face. Although age estimation may seem relatively simple to us, computers have a much more difficult ...

It's a bird, it's a plane, it's super hero sexism

December 21, 2011

As parents do their final holiday shopping, comic books, and their related superhero-themed toys and children's gear, continue to be popular. From Batman rain boots and Legos, to paperback books about Wonder Woman, many stores ...

Comics are serious teaching tools for USC linguist

January 10, 2012

The study of linguistics is not a laughing matter -- unless you happen to have Stan Dubinsky as your professor. The University of South Carolina linguist has been sharing jokes and puns and cartoons with students for more ...

Recommended for you

Internet giants race to faster mobile news apps

October 4, 2015

US tech giants are turning to the news in their competition for mobile users, developing new, faster ways to deliver content, but the benefits for struggling media outlets remain unclear.

Radio frequency 'harvesting' tech unveiled in UK

September 30, 2015

An energy harvesting technology that its developers say will be able to turn ambient radio frequency waves into usable electricity to charge low power devices was unveiled in London on Wednesday.

Professors say US has fallen behind on offshore wind power

September 29, 2015

University of Delaware faculty from the College of Earth, Ocean, and Environment (CEOE), the College of Engineering and the Alfred Lerner School of Business and Economics say that the U.S. has fallen behind in offshore wind ...


Adjust slider to filter visible comments by rank

Display comments: newest first

5 / 5 (7) Mar 14, 2012
Hand drawn cartoons are art created by artists. This looks like crap. Degenerated photos are not cartoons. The technology may have its uses, but as comics it's a FAIL.
not rated yet Mar 14, 2012
Somebody do the State of the Union speech,... please.
not rated yet Mar 17, 2012
Somebody do the State of the Union speech,... please.

The software has to replicate the speech patterns like "aaaannnnd". That would be funny.
1 / 5 (1) Mar 17, 2012
I found the watching of many movies a waste of time and reading of subtitle files counterintuitive to the actual content of movies. Background image is worth of thousands words after then. This technology could become useful for automatic generation of excerpts from movies.
not rated yet Mar 19, 2012
The better not mess with the fantastic artwork of modern anime.
not rated yet Mar 20, 2012
The reverse would be much more interesting.
not rated yet Mar 21, 2012
Hi i think the invention is very useful, and think that the guys do great job, don't critique instantly (to others), i see here a huge business opportunity. Please tell how i can contact with the creator of this software and is there any prototype put online?
Thanks in advance

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.