Teaching machines to see: Professor reverse-engineers human vision for computers

March 1, 2013

(Phys.org)—How do we know if we're looking at the three-dimensional world or at a kind of trompe l'oeil image painted on the inside of a huge glass sphere? More to the point, how would a robot know?

Blessed with brains and the power of biological computation, humans can compute the most likely explanation for what we see. Our neural networks turn the fizz of photons, hitting a curved screen, into perception.

That's awfully difficult to translate into code, says David Cox, who holds a joint appointment as Assistant Professor of Molecular and Cellular Biology and of Computer Science at Harvard.

"Vision is the process of figuring out what's out there in a 3D world, from a set of 2D images cast onto our retinas," Cox explains. "It's actually really hard, and the only reason it seems easy is that we're seeing the world through the solution to the problem."

After all, evolution over hundreds of millions of years has given us a system that works rather well. When we look out at the world, Cox marvels, "we sort of just transparently see."

"That's one of the challenges for ," he says: "Our intuitions about what's easy and what's difficult are usually wrong, because all of our intuitions are coming by way of this . When you sit down and try to write a computer program that does the same thing, you discover just how hard it is."

Teaching machines to see
“If we had computer vision systems that worked as well as our own visual systems do, there’s a much richer set of interactions we could have with machines," says David Cox. Credit: Eliza Grinnell, SEAS Communications

Working at the Harvard School of Engineering and Applied Sciences, the Department of , and the Center for , Cox aims to create artificial systems that can both see and understand what they're looking at. It's a task that requires an in-depth knowledge of neuroscience, but also a fair amount of blue-sky thinking about what might be possible in the realm of artificial .

Cox thinks of his research as reverse engineering—or, more whimsically, committing on nature.

"There's only one set of systems in the known universe that can do what we're looking for, and they happen to be biological systems," he says, "so the motivation for the reverse engineering side of the work is to get the competing product, as it were, open up the box, and figure out how it works so that we can turn around and build artificial systems that work the same way."

Of course, reproducing a brain is easier said than done. Cox's research group employs massively parallel, high-performance computers to try to reproduce the level of computation that happens within the brain—for example, to study facial recognition techniques, which he's been pursuing with members of Todd Zickler's group at SEAS.

There's a huge difference between recognizing a face in a mugshot and recognizing a face that's embedded in a complicated and cluttered real-world scene.

"If you move an object around your visual field, it's appearing on different parts of your retina, it may be lit differently, and you're seeing it from different angles," Cox explains. "There's effectively an infinite number of ways the same object can appear. At the same time, there are infinitely many valid interpretations for any one image that falls onto your retina."

Many of the potential payoffs for designing such an intelligent system are so bizarre that they seem only to belong in the realm of science fiction. Your laptop could notice whether you look tired, happy, or sad, and interact with you appropriately. Your self-driving car could spot you on the sidewalk and offer you a ride.

"If we had computer vision systems that worked as well as our own visual systems do, there's a much richer set of interactions we could have with machines," Cox says.

It's not just about the applications for Cox, though; the basic science of the retina and neurons is wondrously complex and mysterious, and it's on the bridge between biology and computer science that he finds himself at home.

"I'm a 'have your cake and eat it too' kind of person," says. "I think there's great potential to advance our knowledge of how the brain works, but one of the things that's most exciting for me is this idea that if we really understand that, we should be able to build machines that work the same way.

"With that, there's a huge range of really world-changing applications that we could bring to bear."

Explore further: Researchers demonstrate a better way for computers to 'see' (w/ Video)

Related Stories

Imaging inflammation in the living brain

September 30, 2011

Inflammation occurs in the human brain during illnesses such as Alzheimer's disease, Parkinson’s disease, stroke and traumatic brain injury. Now, a research team in Japan has developed a probe that can bind to the pro-inflammatory ...

Cox to offer video-on-demand content through TiVo

August 12, 2010

(AP) -- Digital video recording pioneer TiVo Inc. and cable television provider Cox Communications Inc. are making it easier for Cox subscribers who use TiVo's DVR boxes to watch Cox's on-demand video content.

Cox kills Sprint-based cellphone service

November 16, 2011

(AP) -- Cox Communications, the country's third-largest cable company, stopped offering cellphone service Wednesday, saying it's too small to compete with the big phone companies.

Recommended for you

Forget oil, Russia goes crazy for cryptocurrency

August 16, 2017

Standing in a warehouse in a Moscow suburb, Dmitry Marinichev tries to speak over the deafening hum of hundreds of computers stacked on shelves hard at work mining for crypto money.

Researchers clarify mystery about proposed battery material

August 15, 2017

Battery researchers agree that one of the most promising possibilities for future battery technology is the lithium-air (or lithium-oxygen) battery, which could provide three times as much power for a given weight as today's ...

Signs of distracted driving—pounding heart, sweaty nose

August 15, 2017

Distracted driving—texting or absent-mindedness—claims thousands of lives a year. Researchers from the University of Houston and the Texas A&M Transportation Institute have produced an extensive dataset examining how ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.