Decoding multiple frames from a single, scattered exposure

September 27, 2018 by Ken Kingery, Duke University
Engineers at Duke University have developed a way to extract a sequence of images from light scattered through a mostly opaque material -- or even off a wall -- from one long photographic exposure. Credit: Michael Gehm, Duke University

Engineers at Duke University have developed a way to extract a sequence of images from light scattered through a mostly opaque material—or even off a wall—from one long photographic exposure. The technique has applications in a wide range of fields from security to healthcare to astronomy.

The study appeared online on September 10 in the journal Scientific Reports.

"When I explain to people what this algorithm can do, it sounds like magic," said Michael Gehm, associate professor of electrical and computer engineering at Duke. "But it's really just statistics and a ton of data."

When gets scattered as it passes through a translucent material, the emerging pattern of "speckle" looks as random as static on a television screen with no signal. But it isn't random. Because the light coming from one point of an object travels a path very similar to that of the light coming from an adjacent point, the speckle pattern from each looks very much the same, just shifted slightly.

With enough , astronomers used to use this "memory effect" phenomenon to create clearer images of the heavens through a turbulent atmosphere, as long as the object being imaged is sufficiently compact.

The technique fell out of favor with the development of adaptive optics, which do the same job by using adjustable mirrors to compensate for the scattering.

A few years ago, however, the memory effect technique became popular with scientists again. Because modern cameras can record hundreds of millions of pixels at a time, only a single exposure is needed to make the statistics work.

While this approach can reconstruct a scattered image, it has its limitations. The object has to remain motionless and the scattering medium has to be constant.

Gehm's new approach to imaging breaks through these limitations by extracting a sequence of images from a single, long exposure.

Duke's Michael Gehm has figured out a way to gather scattered light from multiple images on a single exposure, separate them, and then reconstruct each individual image. Credit: Ken Kingery, Duke University

The trick is to use a coded aperture. Think of this as a set of filters that allow light to pass through some areas but not others in a specific pattern. As long as this pattern is known, scientists can computationally extract what the original image looked like. (See Video)

Gehm's new technique uses a sequence of coded apertures to stamp which light is coming from which moment in time. But because each image is collected on a single, long photographic exposure, the resulting speckle ends up even more of a jumbled mess than usual.

"People thought that the resulting would be too random to separate out the individual frames," said Gehm. "But it turns out that today's cameras have such amazing resolution that if you look closely, there's still enough of a pattern to computationally get a toehold and tease them apart."

In their experiment, a simple sequence of four backlit letters appeared one after the other behind a coded aperture and a scattering material. The shutter of a 5.5-megapixel CCD camera was left open for more than a minute during the sequence to gather the images.

While the best results were achieved with a 100-second exposure time, good results could still be obtained with much shorter exposure times. After only a few seconds of processing, the computer successfully returned the individual images of a D, U, K and E from the sequence. The researchers then showed the approach also works when the scattering medium is changed, and even when both the images and scattering mediums are changing.

The best results were achieved when the letters appeared for 25 seconds each because the intensity of the backlight was not very high to begin with, and was even further diminished by the coded aperture and scattering material. But with a more sensitive camera or a brighter source, there's no reason the approach couldn't be used to capture live-action images, Gehm said.

The technique has many potential applications. Not only does it work for light scattering through a material, it would also work for off of a surface—say the paint on a wall. This could allow security cameras to work around corners or even through frosted glass.

In the medical arena, many light-based devices look to gather data through skin and other tissues—such as a Fitbit capturing a person's pulse through their wrist. Light scattering as it travels through the skin and flowing blood cells, however, poses a challenge to more advanced measurements. This technique may provide a path forward.

"We're also looking to see if this approach can be used to separate different aspects of light, particularly color," said Gehm. "One could imagine using coded apertures to gain more information about a single image rather than using it to obtain a sequence of images."

Explore further: Random light scattering enhances the resolution of wide-field optical microscope images

More information: Xiaohan Li et al, Single-shot memory-effect video, Scientific Reports (2018). DOI: 10.1038/s41598-018-31697-8

Related Stories

Human eye capable of seeing 'ghosted' images

September 4, 2018

A team of researchers in the U.K. has found that the human eye and brain are together capable of seeing "ghosted" images. The researchers have published a paper describing their work on the arXiv preprint server.

Algorithm helps turn smartphones into 3-D scanners

December 22, 2015

While 3-D printers have become relatively cheap and available, 3-D scanners have lagged well behind. But now, an algorithm developed by Brown University researchers my help bring high-quality 3-D scanning capability to off-the-shelf ...

The beam of invisibility

September 13, 2017

How do we make an object invisible? Researchers from TU Wien (Vienna), together with colleagues from Greece and the USA, have now developed a new idea for a cloaking technology. A completely opaque material is irradiated ...

Recommended for you

CMS gets first result using largest-ever LHC data sample

February 15, 2019

Just under three months after the final proton–proton collisions from the Large Hadron Collider (LHC)'s second run (Run 2), the CMS collaboration has submitted its first paper based on the full LHC dataset collected in ...

Gravitational waves will settle cosmic conundrum

February 14, 2019

Measurements of gravitational waves from approximately 50 binary neutron stars over the next decade will definitively resolve an intense debate about how quickly our universe is expanding, according to findings from an international ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.