Researchers amplify variations in video, making the invisible visible

Jun 22, 2012
In these frames of video, a new algorithm amplifies the almost imperceptible change in skin color caused by the pumping of the blood. Photo: Michael Rubinstein

At this summer's Siggraph — the premier computer-graphics conference — researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) will present new software that amplifies variations in successive frames of video that are imperceptible to the naked eye. So, for instance, the software makes it possible to actually "see" someone's pulse, as the skin reddens and pales with the flow of blood, and it can exaggerate tiny motions, making visible the vibrations of individual guitar strings or the breathing of a swaddled infant in a neonatal intensive care unit.

The system is somewhat akin to the equalizer in a stereo sound system, which boosts some frequencies and cuts others, except that the pertinent is the frequency of color changes in a sequence of frames, not the frequency of an audio signal. The prototype of the allows the user to specify the frequency range of interest and the degree of amplification. The software works in real time and displays both the original video and the altered version of the video, with changes magnified.

This video is not supported by your browser at this time.

Although the technique lends itself most naturally to phenomena that recur at regular intervals — such as the beating of a heart, the movement of a vibrating string or the inflation of the lungs — if the range of frequencies is wide enough, the system can amplify changes that occur only once. So, for instance, it could be used to compare different images of the same scene, allowing the user to easily pick out changes that might otherwise go unnoticed. In one set of experiments, the system was able to dramatically amplify the movement of shadows in a street scene photographed only twice, at an interval of about 15 seconds.

Happy accident

The MIT researchers — graduate student Michael Rubinstein, recent alumni Hao-Yu Wu '12, MNG '12 and Eugene Shih SM '01, PhD '10, and professors William Freeman, Fredo Durand and John Guttag — intended the system to amplify color changes, but in their initial experiments, they found that it amplified motion as well. "We started from amplifying color, and we noticed that we'd get this nice effect, that motion is also amplified," Rubinstein says. "So we went back, figured out exactly why that happens, studied it well, and saw how we can incorporate that to do better motion amplification."

Using the system to amplify motion rather than color requires a different kind of filtration, and it works well only if the motions are relatively small. But of course, those are exactly the motions whose amplification would be of interest.

Rubinstein envisions that, among other applications, the system could be used for "contactless monitoring" of hospital patients' vital signs. Boosting one set of frequencies would allow measurement of pulse rates, via subtle changes in skin coloration; boosting another set of frequencies would allow monitoring of breathing. The approach could be particularly useful with infants who are born prematurely or otherwise require early medical attention. "Their bodies are so fragile, you want to attach as few sensors as possible," Rubinstein says.

Similarly, Rubinstein says, the system could be used to augment video baby monitors for the home, so that the respiration of sleeping infants would be clearly visible. A father himself, Rubinstein says that he and his wife equipped their daughter's crib with commercial pressure sensors intended to gauge motion and reassure anxious parents that their children are still breathing. "Those are kind of expensive," Rubinstein says, "and some people really complain about getting false positives with them. So I can really see how this type of technique will be able to work better."

In their paper, the researchers describe experiments in which they began investigating both of these applications. But since they've begun giving talks on the work, Rubinstein says, colleagues have proposed a range of other possible uses, from laparoscopic imaging of internal organs, to long-range-surveillance systems that magnify subtle motions, to contactless lie detection based on pulse rate.

"It's a fantastic result," says Maneesh Agrawala, an associate professor in the electrical engineering and computer science department at the University of California at Berkeley, and director of the department's Visualization Lab. Agrawala points out that Freeman and Durand were part of a team of MIT researchers who made a splash at the 2005 Siggraph with a paper on motion magnification in video. "This approach is both simpler and allows you to see some things that you couldn't see with that old approach," Agrawala says. "The simplicity of the approach makes it something that has the possibility for application in a number of places. I think we'll see a lot of people implementing it because it's fairly straightforward."

Explore further: Researcher develops method for monitoring whether private information is sufficiently protected

Related Stories

Faster computer graphics

Jun 13, 2011

Photographs of moving objects are almost always a little blurry — or a lot blurry, if the objects are moving rapidly enough. To make their work look as much like conventional film as possible, game and ...

Body-mounted cameras turn motion capture inside out

Aug 08, 2011

Traditional motion capture techniques use cameras to meticulously record the movements of actors inside studios, enabling those movements to be translated into digital models. But by turning the cameras around — mounting ...

System improves automated monitoring of security cameras

Jun 04, 2012

Police and security teams guarding airports, docks and border crossings from terrorist attack or illegal entry need to know immediately when someone enters a prohibited area, and who they are. A network of ...

Recommended for you

Tackling urban problems with Big Data

20 hours ago

Paul Waddell, a city planning professor at the University of California, Berkeley, with a penchant for conducting research with what he calls his "big urban data," is putting his work to a real-world test ...

Computer-assisted accelerator design

Apr 22, 2014

Stephen Brooks uses his own custom software tool to fire electron beams into a virtual model of proposed accelerator designs for eRHIC. The goal: Keep the cost down and be sure the beams will circulate in ...

User comments : 22

Adjust slider to filter visible comments by rank

Display comments: newest first

mrtea
not rated yet Jun 22, 2012
This is great for specialised use, but how about using surrounding frames to detect variations in lighting, colour, noise and blur patterns etc, in handy-cam, webcam, and security camera type video? Couldn't we improve the quality using similar technologies? Obviously the cost would be prohibitive for normal usage at present, but in the future who knows? There's a great scene in Blade Runner where Deckard keeps ramping up the video quality to zoom in on a reflection of an object hidden from view.
Bowler_4007
1.7 / 5 (6) Jun 22, 2012
This is great for specialised use, but how about using surrounding frames to detect variations in lighting, colour, noise and blur patterns etc, in handy-cam, webcam, and security camera type video? Couldn't we improve the quality using similar technologies? Obviously the cost would be prohibitive for normal usage at present, but in the future who knows? There's a great scene in Blade Runner where Deckard keeps ramping up the video quality to zoom in on a reflection of an object hidden from view.

lighting and color changes essentially amount to the same thing, noise and blur are a whole other level. btw if you had read the article you would have noted how detecting color changes lead to the detection of small imperceptible movements
antialias_physorg
5 / 5 (1) Jun 22, 2012
Couldn't we improve the quality using similar technologies?

Quality is a measure of information. Noise is 'missing' information and no amount of fiddling with filters will restore that information (amplification of selected color/spatial frequency ranges will only amplify the noise with it)

But when you think about actual quality control of manufactured products this could be very interesting (and of course the medical monitoring possibilities demonstrated in the article are fascinating)
Sean_W
1.8 / 5 (5) Jun 22, 2012
While correlating physiological changes with specific mental and emotional states is fraught with peril (curent polygraphs being not much better than dowsing), having more forms of data (being able to monitor a person's pulse, pallor/blush and facial movements without them knowing it, for instance) could help guide investigations, interviews and investigations. In addition to police, journalists might be interested in this technology--or interested in evading it themselves if they broadcast on video. Psychiatrists and psychologists may be able to gain insights into what their patients are holding back and help prod around the topic more tactfully than they might otherwise do.
Maat
1.8 / 5 (5) Jun 22, 2012
Quality is a measure of information. Noise is 'missing' information and no amount of fiddling with filters will restore that information


Of course you are correct that you cannot get missing information, but you can approximate it. For example, in a video of a projectile taken at 15 frames per second the projectile might appear to move in jerky, discrete steps from one frame to the next... analyzing this pattern of motion across frames you could inject intermediate frames with the projectile positioned between it's position in the prior frame and the next frame. This is an obvious example but you can do this with movement of less than 1 pixel between frames as well by monitoring the color change of the leading edge pixels as the object advances... I use extrapolation techniques like this in my line of work to improve the apparent resolution of the data we collect by a factor of 4 and I've never heard anyone question the accuracy of that data.
antialias_physorg
3 / 5 (2) Jun 22, 2012
analyzing this pattern of motion across frames you could inject intermediate frames with the projectile positioned between it's position in the prior frame and the next frame

You have just described antialiasing (which all your displays at home use in one form or another).

antialias_physorg is aware of what antialiasing is ;-)

I use extrapolation techniques like this in my line of work to improve the apparent resolution of the data we collect by a factor of 4 and I've never heard anyone question the accuracy of that data.

In my line of work (medical imaging) that is a big no-no. It can be deadly to a patient to automatically add such 'inferred' information to a CT or MR image - be it smoothing or edge enhancing.
If you do this (and the doctor isn't aware of it) it can mean the difference between being diagnosed with a benign or malign tumor.
Eikka
3 / 5 (4) Jun 22, 2012
Of course you are correct that you cannot get missing information, but you can approximate it. For example, in a video of a projectile taken at 15 frames per second the projectile might appear to move in jerky, discrete steps from one frame to the next... analyzing this pattern of motion across frames you could inject intermediate frames with the projectile positioned between it's position in the prior frame and the next frame.


Here you have to consider what is "noise". The bullet appearing at a certain part of the frame is the information, and noise would be camera shake and distortion by the pixel grid that places the object at an indetermined location.

So at any time you don't know exactly where the bullet is relative to the camera, or its surroundings. You can fill in more frames, but that won't represent real information because the data you calculated them from is imprecise.
Eikka
2.3 / 5 (3) Jun 22, 2012
You have just described antialiasing (which all your displays at home use in one form or another).

antialias_physorg is aware of what antialiasing is ;-)


I do hope so, because what was actually described was motion interpolation - which is being employed by modern TV sets - not anti-aliasing, which in some senses is the opposite process to interpolation.

There's also the problem in the case of a bullet, that it spins. You'd have to somehow make a system that figures out what the backside of the bullet looks like in order to render it correctly in the frames between.
ArisKatsaris
5 / 5 (2) Jun 22, 2012
"Quality is a measure of information. Noise is 'missing' information and no amount of fiddling with filters will restore that information "

That is a too simplistic view. Imagine you have two sequential frames of a video showing the same person. If you take the frames individually there's lost information in the sense of noise. But taking them together, you could hypothetically use the information present in one frame to fill in absent information of the other. A zit is shown clearly in one frame, it's only a blur in the other,

If you have three or four or five sequential frames of the same person -- and you know that person didn't magically grow or lose zits in the space of a fraction of a second, that it is actually the same object/person being videoed, that the frames weren't photoshopped, etc... There's lots of extra information that could be found in merely knowing that a given frame is a continuation of a previous one.

TkClick
3.7 / 5 (6) Jun 22, 2012
The whole algorithm is is quite simple and it fits to one page in Matlab.
antialias_physorg
not rated yet Jun 22, 2012
If you have three or four or five sequential frames of the same person -- and you know that person didn't magically grow or lose zits in the space of a fraction of a second

That's how some video compression algorithms work. However if you use such an algorithm you lose the ability to respond to sharp/sudden changes in pictures (compression algorithms counter this by detecting, during the compression, when this happens and adaptively saving such additional information - at a cost of slightly increasing filesize). For most photo/video applications that's OK.

But there are some applications where putting extra information in that just isn't there is not allowed (medical) or downright detrimental (astronomy, videos of experiments in physics, biology, chemistry, ...). Sometimes it is those sudden changes where the information is. If you gloss that over you lose it.
Eikka
2.3 / 5 (3) Jun 22, 2012
But taking them together, you could hypothetically use the information present in one frame to fill in absent information of the other.


But you still aren't gaining any information. You get a clearer still image at the loss of information about how the person has moved between the frames.

Of course stacking sequential frames gives you a better picture, but that's not information that you didn't have before. You can emphasize some part of it by filtering the image, but nothing gets you more than what is already in your data.

compression algorithms counter this by detecting, during the compression, when this happens


At higher compression ratios, you get interesting effects. DVB digital television sometimes suffers from a sort of "anime effect" where parts of the picture get detached and move independently for a second because the algorithm didn't catch some gradual movement of the background.
Tangent2
2.3 / 5 (6) Jun 22, 2012
Perhaps we can now finally get some useful information from those shaky and blurred videos of UFOs since seemingly no one is able to get a steady clear video of one.
xen_uno
2 / 5 (4) Jun 22, 2012
Anti Phys - "You have just described antialiasing (which all your displays at home use in one form or another)"

No .. he (Maat) described key framing for the bullet sequence (assuming a computer can fill in the intermediate frames).

http://en.wikiped...ey_frame
antialias_physorg
not rated yet Jun 22, 2012
Same thing. One is just in the spatial domain and the other is in the temporal domain. The math behind it is identical.

(For a computer a number is a number - whether it denotes a distance step or a timestep).

For both you have an initial and a target space and then generate inbetweens. Keyframing does it from an initial dataset to a target dataset over a series of timesteps. Antialiasing in a spatial domain does it from an initial dataset (e.g. a black line on a pixellated background) to a target dataset (e.g. the surrounding white pixels)

Tomato - tomahto.
chromosome2
5 / 5 (1) Jun 23, 2012
If you're wondering what OS it is in that video that they're running the software on, it's Ubuntu.
kuntur2k
5 / 5 (1) Jun 23, 2012
Wow, beam me up Scotty. The medical Tricoder has arrived.
xen_uno
2 / 5 (4) Jun 23, 2012
I don't know how AA works on video, but with still images it edge detects with some kind of blending/blurring algorithm. Since a vid is a series of still's, maybe it'd doing same thing on a frame by frame basis. That is not the same as key framing, which generates complete whole frames between target frames.

Rock on Ubuntu
trekgeek1
not rated yet Jun 23, 2012
Wow, beam me up Scotty. The medical Tricoder has arrived.


Ha! I was just saying that to my wife and then I saw your comment. That's great.
Valentiinro
5 / 5 (1) Jun 23, 2012
This is great for specialised use, but how about using surrounding frames to detect variations in lighting, colour, noise and blur patterns etc, in handy-cam, webcam, and security camera type video? Couldn't we improve the quality using similar technologies? Obviously the cost would be prohibitive for normal usage at present, but in the future who knows? There's a great scene in Blade Runner where Deckard keeps ramping up the video quality to zoom in on a reflection of an object hidden from view.


Why would cost be prohibitive? This is just a software function run on a video, which is easy enough to do to be done in real time. For something like this, it seems the trip to market readiness is as quick as getting a company that makes something (such as video baby monitors) to add the software with frequency checks for something (such as heart rate or breathing rate) to their system. Or a cheap webcam connected to a laptop could do the whole job.
Isaacsname
5 / 5 (1) Jun 24, 2012
I wonder how it'd work with plants/fungi, etc ?

..very interesting
Eric_B
1 / 5 (1) Jun 25, 2012
Tangent2 "Perhaps we can now finally get some useful information from those shaky and blurred videos of UFOs since seemingly no one is able to get a steady clear video of one."

search Youtube for LunaCognita and you will find some interesting post-processing of anomalous video imagery.

More news stories

Google+ boss leaving the company

The executive credited with bringing the Google+ social network to life is leaving the Internet colossus after playing a key role there for nearly eight years.

Facebook woos journalists with 'FB Newswire'

Facebook launched Thursday FB Newswire, billed as an online trove of real-time information for journalists and newsrooms to mine while reporting on events or crafting stories.

Genetic legacy of rare dwarf trees is widespread

Researchers from Queen Mary University of London have found genetic evidence that one of Britain's native tree species, the dwarf birch found in the Scottish Highlands, was once common in England.