Researchers use augmented reality to turn coloring books into 3-D experience

October 2, 2015
Credit: Disney Research

A coloring book and a box of crayons may give kids an early opportunity for creative expression but, next to TV and video games, coloring can sometimes seem unexciting. A coloring book app devised by Disney Research, however, can cause characters to leap from the page in 3-D glory with the help of augmented reality.

A child colors a character, such as an elephant, on the book page normally, while a tablet or smartphone running the monitors the drawing. Based on the child's coloring, the app fills in colors in real-time on an animated 3-D version of the elephant that is visible on the device's screen and integrated into the video.

The app keeps the core focus on the traditional activity of coloring while offering a magical digital overlay that enhances engagement.

In user testing - performed with adults rather than children in this early study - the researchers found that most users said the app increased their motivation to draw in coloring books and 80 percent said the app increased their feeling of connection to a character.

Researchers from Disney, ETH Zurich and the Swiss university EPFL presented the augmented reality app at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2015) in Fukuoka, Japan. Although the research work is just now being presented to scientific audiences, it has already gone through the tech transfer process, inspiring the commercial product called "Disney Color and Play" launched earlier this year by Disney Publishing Worldwide and Bendon.

This work fits into a larger initiative at Disney Research called Augmented Creativity, which focuses on using to enhance creative play.

"Augmented reality holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity," said Robert W. Sumner, a principal research scientist who leads the group on animation and interactive graphics at Disney Research. "We are thrilled to have the opportunity to present the scientific advances behind this technology," continued Sumner, "and are especially happy that it is available to consumers, thanks to our cooperation with Disney Publishing."

To create this new experience, the researchers first created animated 3-D virtual characters and then use custom software to generate 2-D line-art representations of the characters for a coloring book. The app, operating on a device with a camera viewing the user and the coloring book, automatically detects the character the user is coloring and displays the 3-D version.

As the child applies color to the 2-D drawing, the app applies the same color to the 3-D character - both to the areas visible in the 2-D drawing and to the remainder of the 3-D form not visible in the book. Because the coloring occurs in real-time, the illusion is created that the user is also coloring the occluded areas, with similar texturing of the color.

Determining how to apply color to the occluded areas was one of the tougher problems to solve, Sumner said. Simply mirroring the user's colored strokes doesn't work because the pattern of colors used for, say, a character's face will not be appropriate for the back of the character's head. The color also has to be continuous, so no seams can be seen between the visible areas and the occluded areas or where disparate portions of the textures meet.

Their approach was to create a "lookup map" for each character, which matches pixels in the occluded parts with corresponding pixels in the portion visible to the user. User testing showed that this method provides better results than "naïve" approaches such as mirroring. Most importantly, the lookup map method enables the coloring to be performed instantly.

Because paper books don't lie perfectly flat and can flex as a user colors, the team developed a deformable surface tracking method to monitor surface changes and to keep the virtual character displayed on the device screen correctly oriented with the book's page.

Explore further: Breaking the fourth wall in human-computer interaction: Really talking to each other

More information: Live Texturing of Augmented Reality Characters from Colored Drawings-Paper, www.disneyresearch.com/wp-content/uploads/Live-Texturing-of-Augmented-Reality-Characters-from-Colored-Drawings-Paper.pdf

Related Stories

Game apps for creative kids

March 8, 2016

Augmented creativity can encourage children towards a new-found sense of creativity, cooperation and interaction with their environment. ETH Zurich's Game Technology Center will present its latest apps in the field at the ...

If the tech fits, wear it

October 16, 2013

The digital domain is creeping off our desktops and onto our bodies, from music players that match your tunes to your heart beat to mood sweaters that change color depending on your emotional state.

Google creates a spectacle with project

April 6, 2012

(AP) -- If you think texting while walking is dangerous, just wait until everyone starts wearing Google's futuristic, Internet-connected glasses.

Recommended for you

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.