Teaching machines to see: Professor reverse-engineers human vision for computers

Mar 01, 2013

(Phys.org)—How do we know if we're looking at the three-dimensional world or at a kind of trompe l'oeil image painted on the inside of a huge glass sphere? More to the point, how would a robot know?

Blessed with brains and the power of biological computation, humans can compute the most likely explanation for what we see. Our neural networks turn the fizz of photons, hitting a curved screen, into perception.

That's awfully difficult to translate into code, says David Cox, who holds a joint appointment as Assistant Professor of Molecular and Cellular Biology and of Computer Science at Harvard.

"Vision is the process of figuring out what's out there in a 3D world, from a set of 2D images cast onto our retinas," Cox explains. "It's actually really hard, and the only reason it seems easy is that we're seeing the world through the solution to the problem."

After all, evolution over hundreds of millions of years has given us a system that works rather well. When we look out at the world, Cox marvels, "we sort of just transparently see."

"That's one of the challenges for ," he says: "Our intuitions about what's easy and what's difficult are usually wrong, because all of our intuitions are coming by way of this . When you sit down and try to write a computer program that does the same thing, you discover just how hard it is."

Teaching machines to see
“If we had computer vision systems that worked as well as our own visual systems do, there’s a much richer set of interactions we could have with machines," says David Cox. Credit: Eliza Grinnell, SEAS Communications

Working at the Harvard School of Engineering and Applied Sciences, the Department of , and the Center for , Cox aims to create artificial systems that can both see and understand what they're looking at. It's a task that requires an in-depth knowledge of neuroscience, but also a fair amount of blue-sky thinking about what might be possible in the realm of artificial .

Cox thinks of his research as reverse engineering—or, more whimsically, committing on nature.

"There's only one set of systems in the known universe that can do what we're looking for, and they happen to be biological systems," he says, "so the motivation for the reverse engineering side of the work is to get the competing product, as it were, open up the box, and figure out how it works so that we can turn around and build artificial systems that work the same way."

Of course, reproducing a brain is easier said than done. Cox's research group employs massively parallel, high-performance computers to try to reproduce the level of computation that happens within the brain—for example, to study facial recognition techniques, which he's been pursuing with members of Todd Zickler's group at SEAS.

There's a huge difference between recognizing a face in a mugshot and recognizing a face that's embedded in a complicated and cluttered real-world scene.

"If you move an object around your visual field, it's appearing on different parts of your retina, it may be lit differently, and you're seeing it from different angles," Cox explains. "There's effectively an infinite number of ways the same object can appear. At the same time, there are infinitely many valid interpretations for any one image that falls onto your retina."

Many of the potential payoffs for designing such an intelligent system are so bizarre that they seem only to belong in the realm of science fiction. Your laptop could notice whether you look tired, happy, or sad, and interact with you appropriately. Your self-driving car could spot you on the sidewalk and offer you a ride.

"If we had computer vision systems that worked as well as our own visual systems do, there's a much richer set of interactions we could have with machines," Cox says.

It's not just about the applications for Cox, though; the basic science of the retina and neurons is wondrously complex and mysterious, and it's on the bridge between biology and computer science that he finds himself at home.

"I'm a 'have your cake and eat it too' kind of person," says. "I think there's great potential to advance our knowledge of how the brain works, but one of the things that's most exciting for me is this idea that if we really understand that, we should be able to build machines that work the same way.

"With that, there's a huge range of really world-changing applications that we could bring to bear."

Explore further: California quake points to research advancements in retrofitting older buildings

Provided by Harvard School of Engineering and Applied Sciences

5 /5 (2 votes)
add to favorites email to friend print save as pdf

Related Stories

Imaging inflammation in the living brain

Sep 30, 2011

Inflammation occurs in the human brain during illnesses such as Alzheimer's disease, Parkinson’s disease, stroke and traumatic brain injury. Now, a research team in Japan has developed a probe that can ...

Cox to offer video-on-demand content through TiVo

Aug 12, 2010

(AP) -- Digital video recording pioneer TiVo Inc. and cable television provider Cox Communications Inc. are making it easier for Cox subscribers who use TiVo's DVR boxes to watch Cox's on-demand video content.

Cox kills Sprint-based cellphone service

Nov 16, 2011

(AP) -- Cox Communications, the country's third-largest cable company, stopped offering cellphone service Wednesday, saying it's too small to compete with the big phone companies.

Recommended for you

Greater safety and security at Europe's train stations

Sep 01, 2014

When a suspicious individual fleas on a bus or by train, then things usually get tough for the police. This is because the security systems of the various transportation companies and security services are ...

Fingerprints for freight items

Sep 01, 2014

Security is a top priority in air freight logistics but screening procedures can be very time consuming and costly. Fraunhofer researchers intend to boost efficiency with a new approach to digital logistics, ...

On the way to a safe and secure smart home

Sep 01, 2014

A growing number of household operations can be managed via the Internet. Today's "Smart Home" promises efficient building management. But often the systems are not secure and can only be retrofitted at great ...

DIY glove-based tutor indicates muscle-memory potential

Aug 31, 2014

A senior editor at IEEE Spectrum worked on a DIY project that enabled his 11-year-old son to improve his touch typing by use of a vibrating glove. His son was already "pretty quick on the keyboard," said ...

User comments : 0