Picture-perfect: Quick, efficient chip cleans up common flaws in amateur photographs

Feb 19, 2013 by Helen Knight
Picture-perfect: Quick, efficient chip cleans up common flaws in amateur photographs
Die photo of the processor chip. Credit: Researchers

Your smartphone snapshots could be instantly converted into professional-looking photographs with just the touch of a button, thanks to a processor chip developed at MIT.

The chip, built by a team at MIT's Microsystems Technology Laboratory, can perform tasks such as creating more realistic or enhanced lighting in a shot without destroying the scene's ambience, in just a fraction of a second. The technology could be integrated with any smartphone, or .

Existing computational photography systems tend to be software applications that are installed onto cameras and smartphones. However, such systems consume substantial power, take a considerable amount of time to run, and require a fair amount of knowledge on the part of the user, says the paper's lead author, Rahul Rithe, a graduate student in MIT's Department of Electrical Engineering and Computer Science.

"We wanted to build a single chip that could perform multiple operations, consume significantly less power compared to doing the same job in software, and do it all in real time," Rithe says. He developed the chip with Anantha Chandrakasan, the Joseph F. and Nancy P. Keithley Professor of Electrical Engineering, fellow graduate student Priyanka Raina, research scientist Nathan Ickes and undergraduate Srikanth Tenneti.

One such task, known as High Dynamic Range (HDR) imaging, is designed to compensate for limitations on the range of brightness that can be recorded by existing digital cameras, to capture pictures that more accurately reflect the way we perceive the same scenes with our own eyes. 

To do this, the chip's processor automatically takes three separate "low dynamic range" images with the camera: a normally exposed image, an overexposed image capturing details in the dark areas of the scene, and an underexposed image capturing details in the bright areas. It then merges them to create one image capturing the entire range of brightness in the scene, Rithe says.

Software-based systems typically take several seconds to perform this operation, while the chip can do it in a few hundred milliseconds on a 10-megapixel image. This means it is even fast enough to apply to video, Ickes says. The chip consumes dramatically less power than existing CPUs and GPUs while performing the operation, he adds.

Another task the chip can carry out is to enhance the lighting in a darkened scene more realistically than conventional flash photography. "Typically when taking pictures in a low-light situation, if we don't use flash on the camera we get images that are pretty dark and noisy, and if we do use the flash we get bright images but with harsh lighting, and the ambience created by the natural lighting in the room is lost," Rithe says.

So in this instance the processor takes two images, one with a flash and one without. It then splits both into a base layer, containing just the large-scale features within the shot, and a detailed layer. Finally, it merges the two images, preserving the natural ambience from the base layer of the nonflash shot, while extracting the details from the picture taken with the flash.

To remove unwanted features from the image, such as noise—the unexpected variations in color or brightness created by digital cameras—the system blurs any undesired pixel with its surrounding neighbors, so that it matches those around it. In conventional filtering, however, this means even those pixels at the edges of objects are also blurred, which results in a less detailed image.

But by using what is called a bilateral filter, the researchers are able to preserve these outlines, Rithe says. That is because bilateral filters will only blur pixels with their neighbors if they have been assigned a similar brightness value. Since any objects within the image are likely to have a very different level of brightness than that of their background, this prevents the system from blurring across any edges, he says.

To perform each of these tasks, the chip's processing unit uses a method of organizing and storing data called a bilateral grid. The image is first divided into smaller blocks. For each block, a histogram is then created. This results in a 3-D representation of the image, with the x and y axes representing the position of the block, and the brightness histogram representing the third dimension.

This makes it easy for the filter to avoid blurring across edges, since pixels with different brightness levels are separated in this third axis in the grid structure, no matter how close together they are in the image itself.

The algorithms implemented on the chip are inspired by the computational photography work of associate professor of computer science and engineering Fredo Durand and Bill Freeman, a professor of computer science and engineering in MIT's Computer Science and Artificial Intelligence Laboratory. With the aid of Taiwanese semiconductor manufacturer TSMC's University Shuttle Program, the researchers have already built a working prototype of the chip using 40-nanometer CMOS technology, and integrated it into a camera and display. They will be presenting their chip at the International Solid-State Circuits Conference in San Francisco in February.

The chip offers a hardware solution to some important problems in computational photography, says Michael Cohen at Microsoft Research in Redmond, Wash. "As algorithms such as bilateral filtering become more accepted as required processing for imaging, this kind of hardware specialization becomes more keenly needed," he says.

The power savings offered by the are particularly impressive, says Matt Uyttendaele, also of Microsoft Research. "All in all [it is] a nicely crafted component that can bring applications onto more energy-starved devices," he says.

Explore further: UT Dallas professor to develop framework to protect computers' cores

Related Stories

3-D, after-the-fact focus image sensors invented

Apr 03, 2012

(PhysOrg.com) -- At the heart of digital photography is a chip called an image sensor that captures a map of the intensity of the light as it comes through the lens and converts it to an electronic signal.

Ultrasensitive photon hunter

Nov 07, 2012

When it comes to imaging, every single photon counts if there is barely any available light. This is the point where the latest technologies often reach their limits. Researchers have now developed a diode ...

Powerful LED flash for smartphones

Nov 26, 2010

Researchers from Osram Opto Semiconductors have developed an LED chip that makes it possible to offer smaller pocket projectors, vehicle headlights, and cell phone camera flashes. In combination with a new ...

High-resolution microscopy without a lens

Sep 05, 2012

(Phys.org)—Over the past several years, major advances have been made at UCLA in the field of lens-less computational imaging technology, particularly in the design of lens-free holographic microscopes, which, because of ...

Recommended for you

User comments : 4

Adjust slider to filter visible comments by rank

Display comments: newest first

EyeNStein
1 / 5 (4) Feb 19, 2013
Hope this die image is from one of their reject chips.
Also dont see a 'few hunderd miliseconds' as fast enough for video processing at 20ms per frame.
Recovering detail from dark under-exposed unlit areas will have to be seen to be believeable.
AlejoHausner
5 / 5 (1) Feb 19, 2013
EyeNStein said:

"Recovering detail from dark under-exposed unlit areas will have to be seen to be believeable"

The technique was published about 10 years ago, and is described here:

http://research.m...noflash/

The idea is to take two pictures, a grainy one without flash, which gives you the chroma information, and a sharp one with flash, which shows spatial detail but has less color information. Then the two are combined.
Eikka
1 / 5 (1) Feb 19, 2013
The idea is to take two pictures


Which necessarily halves your framerate.

The problem is that when you take separate pictures separated by several milliseconds of time and combine them, the features don't necessairly match and you get processing artifacts. The rolling shutter distortion is already an annoying feature of cheap CMOS sensors, and now you require the camera to take two or three pictures in succession, which leads to even weirder effects on anything that moves.

If you want more dynamic range, it would be better to build a sensor with more dynamic range instead of trying to cheat your way into it by computational tricks, because there's no free lunch.

baudrunner
2.3 / 5 (3) Feb 19, 2013
The chip will perform some of the same functions that a half-decent image editing application can do, and over which the user has greater control. Nevertheless, this is a good feature to have in a camera for people who take only snapshots.

More news stories

Ex-Apple chief plans mobile phone for India

Former Apple chief executive John Sculley, whose marketing skills helped bring the personal computer to desktops worldwide, says he plans to launch a mobile phone in India to exploit its still largely untapped ...

Airbnb rental site raises $450 mn

Online lodging listings website Airbnb inked a $450 million funding deal with investors led by TPG, a source close to the matter said Friday.

Health care site flagged in Heartbleed review

People with accounts on the enrollment website for President Barack Obama's signature health care law are being told to change their passwords following an administration-wide review of the government's vulnerability to the ...

A homemade solar lamp for developing countries

(Phys.org) —The solar lamp developed by the start-up LEDsafari is a more effective, safer, and less expensive form of illumination than the traditional oil lamp currently used by more than one billion people ...

NASA's space station Robonaut finally getting legs

Robonaut, the first out-of-this-world humanoid, is finally getting its space legs. For three years, Robonaut has had to manage from the waist up. This new pair of legs means the experimental robot—now stuck ...

Filipino tests negative for Middle East virus

A Filipino nurse who tested positive for the Middle East virus has been found free of infection in a subsequent examination after he returned home, Philippine health officials said Saturday.

Egypt archaeologists find ancient writer's tomb

Egypt's minister of antiquities says a team of Spanish archaeologists has discovered two tombs in the southern part of the country, one of them belonging to a writer and containing a trove of artifacts including reed pens ...