Researchers use augmented reality to turn coloring books into 3-D experience

October 2, 2015
Credit: Disney Research

A coloring book and a box of crayons may give kids an early opportunity for creative expression but, next to TV and video games, coloring can sometimes seem unexciting. A coloring book app devised by Disney Research, however, can cause characters to leap from the page in 3-D glory with the help of augmented reality.

A child colors a character, such as an elephant, on the book page normally, while a tablet or smartphone running the monitors the drawing. Based on the child's coloring, the app fills in colors in real-time on an animated 3-D version of the elephant that is visible on the device's screen and integrated into the video.

The app keeps the core focus on the traditional activity of coloring while offering a magical digital overlay that enhances engagement.

In user testing - performed with adults rather than children in this early study - the researchers found that most users said the app increased their motivation to draw in coloring books and 80 percent said the app increased their feeling of connection to a character.

Researchers from Disney, ETH Zurich and the Swiss university EPFL presented the augmented reality app at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2015) in Fukuoka, Japan. Although the research work is just now being presented to scientific audiences, it has already gone through the tech transfer process, inspiring the commercial product called "Disney Color and Play" launched earlier this year by Disney Publishing Worldwide and Bendon.

This work fits into a larger initiative at Disney Research called Augmented Creativity, which focuses on using to enhance creative play.

"Augmented reality holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity," said Robert W. Sumner, a principal research scientist who leads the group on animation and interactive graphics at Disney Research. "We are thrilled to have the opportunity to present the scientific advances behind this technology," continued Sumner, "and are especially happy that it is available to consumers, thanks to our cooperation with Disney Publishing."

To create this new experience, the researchers first created animated 3-D virtual characters and then use custom software to generate 2-D line-art representations of the characters for a coloring book. The app, operating on a device with a camera viewing the user and the coloring book, automatically detects the character the user is coloring and displays the 3-D version.

As the child applies color to the 2-D drawing, the app applies the same color to the 3-D character - both to the areas visible in the 2-D drawing and to the remainder of the 3-D form not visible in the book. Because the coloring occurs in real-time, the illusion is created that the user is also coloring the occluded areas, with similar texturing of the color.

Determining how to apply color to the occluded areas was one of the tougher problems to solve, Sumner said. Simply mirroring the user's colored strokes doesn't work because the pattern of colors used for, say, a character's face will not be appropriate for the back of the character's head. The color also has to be continuous, so no seams can be seen between the visible areas and the occluded areas or where disparate portions of the textures meet.

Their approach was to create a "lookup map" for each character, which matches pixels in the occluded parts with corresponding pixels in the portion visible to the user. User testing showed that this method provides better results than "naïve" approaches such as mirroring. Most importantly, the lookup map method enables the coloring to be performed instantly.

Because paper books don't lie perfectly flat and can flex as a user colors, the team developed a deformable surface tracking method to monitor surface changes and to keep the virtual character displayed on the device screen correctly oriented with the book's page.

Explore further: Using mobile devices to augment reality can enhance creative play and exploration

More information: Live Texturing of Augmented Reality Characters from Colored Drawings-Paper,

Related Stories

Game apps for creative kids

March 8, 2016

Augmented creativity can encourage children towards a new-found sense of creativity, cooperation and interaction with their environment. ETH Zurich's Game Technology Center will present its latest apps in the field at the ...

App-makers draw a bead on adult coloring books

May 20, 2016

Since she got in on the adult coloring book craze two years ago, Cheri Brown has spent more than $400 on 50 books holding intricate sketches that she embellishes with Sharpies, colored pencils and gel pens.

3D animation with the stroke of a pen

July 7, 2016

Mosketch, a software developed by Moka Studio and EPFL, lets you generate 3D animation without sophisticated training. It's professional-grade software that can be used by anyone, from independent artists to animation studios.

Recommended for you

Making it easier to collaborate on code

October 26, 2016

Git is an open-source system with a polarizing reputation among programmers. It's a powerful tool to help developers track changes to code, but many view it as prohibitively difficult to use.

Dutch unveil giant vacuum to clean outside air

October 25, 2016

Dutch inventors Tuesday unveiled what they called the world's first giant outside air vacuum cleaner—a large purifying system intended to filter out toxic tiny particles from the atmosphere surrounding the machine.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.