Teaching machines to see: Professor reverse-engineers human vision for computers

Mar 01, 2013

(Phys.org)—How do we know if we're looking at the three-dimensional world or at a kind of trompe l'oeil image painted on the inside of a huge glass sphere? More to the point, how would a robot know?

Blessed with brains and the power of biological computation, humans can compute the most likely explanation for what we see. Our neural networks turn the fizz of photons, hitting a curved screen, into perception.

That's awfully difficult to translate into code, says David Cox, who holds a joint appointment as Assistant Professor of Molecular and Cellular Biology and of Computer Science at Harvard.

"Vision is the process of figuring out what's out there in a 3D world, from a set of 2D images cast onto our retinas," Cox explains. "It's actually really hard, and the only reason it seems easy is that we're seeing the world through the solution to the problem."

After all, evolution over hundreds of millions of years has given us a system that works rather well. When we look out at the world, Cox marvels, "we sort of just transparently see."

"That's one of the challenges for ," he says: "Our intuitions about what's easy and what's difficult are usually wrong, because all of our intuitions are coming by way of this . When you sit down and try to write a computer program that does the same thing, you discover just how hard it is."

Teaching machines to see
“If we had computer vision systems that worked as well as our own visual systems do, there’s a much richer set of interactions we could have with machines," says David Cox. Credit: Eliza Grinnell, SEAS Communications

Working at the Harvard School of Engineering and Applied Sciences, the Department of , and the Center for , Cox aims to create artificial systems that can both see and understand what they're looking at. It's a task that requires an in-depth knowledge of neuroscience, but also a fair amount of blue-sky thinking about what might be possible in the realm of artificial .

Cox thinks of his research as reverse engineering—or, more whimsically, committing on nature.

"There's only one set of systems in the known universe that can do what we're looking for, and they happen to be biological systems," he says, "so the motivation for the reverse engineering side of the work is to get the competing product, as it were, open up the box, and figure out how it works so that we can turn around and build artificial systems that work the same way."

Of course, reproducing a brain is easier said than done. Cox's research group employs massively parallel, high-performance computers to try to reproduce the level of computation that happens within the brain—for example, to study facial recognition techniques, which he's been pursuing with members of Todd Zickler's group at SEAS.

There's a huge difference between recognizing a face in a mugshot and recognizing a face that's embedded in a complicated and cluttered real-world scene.

"If you move an object around your visual field, it's appearing on different parts of your retina, it may be lit differently, and you're seeing it from different angles," Cox explains. "There's effectively an infinite number of ways the same object can appear. At the same time, there are infinitely many valid interpretations for any one image that falls onto your retina."

Many of the potential payoffs for designing such an intelligent system are so bizarre that they seem only to belong in the realm of science fiction. Your laptop could notice whether you look tired, happy, or sad, and interact with you appropriately. Your self-driving car could spot you on the sidewalk and offer you a ride.

"If we had computer vision systems that worked as well as our own visual systems do, there's a much richer set of interactions we could have with machines," Cox says.

It's not just about the applications for Cox, though; the basic science of the retina and neurons is wondrously complex and mysterious, and it's on the bridge between biology and computer science that he finds himself at home.

"I'm a 'have your cake and eat it too' kind of person," says. "I think there's great potential to advance our knowledge of how the brain works, but one of the things that's most exciting for me is this idea that if we really understand that, we should be able to build machines that work the same way.

"With that, there's a huge range of really world-changing applications that we could bring to bear."

Explore further: Off-world manufacturing is a go with space printer

Provided by Harvard School of Engineering and Applied Sciences

5 /5 (2 votes)
add to favorites email to friend print save as pdf

Related Stories

Imaging inflammation in the living brain

Sep 30, 2011

Inflammation occurs in the human brain during illnesses such as Alzheimer's disease, Parkinson’s disease, stroke and traumatic brain injury. Now, a research team in Japan has developed a probe that can ...

Cox to offer video-on-demand content through TiVo

Aug 12, 2010

(AP) -- Digital video recording pioneer TiVo Inc. and cable television provider Cox Communications Inc. are making it easier for Cox subscribers who use TiVo's DVR boxes to watch Cox's on-demand video content.

Cox kills Sprint-based cellphone service

Nov 16, 2011

(AP) -- Cox Communications, the country's third-largest cable company, stopped offering cellphone service Wednesday, saying it's too small to compete with the big phone companies.

Recommended for you

Off-world manufacturing is a go with space printer

20 hours ago

On Friday, the BBC reported on a NASA email exchange with a space station which involved astronauts on the International Space Station using their 3-D printer to make a wrench from instructions sent up in ...

First drone in Nevada test program crashes in demo

Dec 19, 2014

A drone testing program in Nevada is off to a bumpy start after the first unmanned aircraft authorized to fly without Federal Aviation Administration supervision crashed during a ceremony in Boulder City.

Fully automated: Thousands of blood samples every hour

Dec 19, 2014

Siemens is supplying automation technology for the longest and one of the most cutting-edge sample processing lines in any clinical laboratory. The line, or automation track, 200 meters long, in Marlborough, ...

Explainer: What is 4-D printing?

Dec 19, 2014

Additive manufacturing – or 3D printing – is 30 years old this year. Today, it's found not just in industry but in households, as the price of 3D printers has fallen below US$1,000. Knowing you can p ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.