New surveillance camera system provides text feed

June 3, 2010 by Lin Edwards, report

Two major tasks of the I2T framework: (a) image parsing and (b) text description. Image credit: Benjamin Yao.
( -- Scientists at the University of California in Los Angeles (UCLA) have developed a prototype surveillance camera and computer system to analyze the camera images and deliver a text feed describing what the camera is seeing. The new system aims to make searching vast amounts of video much more efficient.

The system was developed by Professor Song-Chun Zhu and colleagues Haifeng Gong and Benjamin Yao, in collaboration with the company ObjectVideo, of Reston Virginia in the US. Dubbed I2T for Image to , the system runs frames through a series of vision algorithms to produce a textual summary of the contents of the frames. The text can then be indexed and stored in a database that can be searched using a simple text search. The system has been successfully demonstrated on surveillance footage.

The I2T system draws on a database of over two million images containing identified objects in over 500 classifications. The database was collected by Zhu starting in 2005 in Ezhou, China, with support from the Chinese government, but is still not large enough to allow the system to assess a dynamic situation correctly.

The first process in I2T is an image parser that analyzes an image and removes the background and identifies the shapes in the picture. The second part of the process determines the meanings of the shapes by referring to the image database. Zhu said that once the image is parsed transcribing the results into natural language “is not too hard.”

Diagram of the I2T framework. Image credit: Benjamin Yao.

The system also uses algorithms describing the movement of objects from one frame to another and can generate text describing motions, such as “boat 3 approaches maritime marker at 40:01.” It can also sometimes match objects that have left and then re-entered a scene, and can describe events such as a car running a stop sign.

Professor Zhu said at the moment almost all searches for images within video is done using surrounding text, but the new system directly generates text from the images. He also added that the existence of YouTube and other video collections, and the expanding use of surveillance cameras everywhere show that being unable to efficiently search video is a major problem.

The I2T system is not yet advanced enough to recognize a large number of images instantly and is not ready yet for commercialization, but the researchers say it is close and needs only “minor tweaks.” The scientists also say they may be able to feed the text into a vocal synthesizer to increase its usefulness.

You can now listen to all podcasts at

Explore further: Seeing things: Researchers teach computers to recognize objects

More information: -- Technical description of I2T: Image Parsing to Text Generation -
-- Research paper: Benjamin Yao, et al. I2T: Image Parsing to Text Description, Proceedings of IEEE [pdf].

Related Stories

CeBIT 2005: Sound Added to Text Messages for Talking Images

March 12, 2005

Soon it may be possible to send short messages as audio documents with animations created by the sender. As a supplement to text messaging and MMS, Siemens has developed the Animated Instant Voice Message, which is being ...

Picture this - automatic image categorisation

May 3, 2005

Creating, storing and transmitting visual images has become increasingly easy. Yet the same problem always arises – how to categorise or classify visual images automatically without using external metadata or image thumbnails? ...

P2P comes to the aid of audiovisual search (w/ Video)

November 18, 2009

( -- Current methods of searching audiovisual content can be a hit-and-miss affair. Manually tagging online media content is time consuming, and costly. But new 'query by example' methods, built on peer-to-peer ...

Recommended for you

Pushing lithium ion batteries to the next performance level

December 13, 2018

Conventional lithium ion batteries, such as those widely used in smartphones and notebooks, have reached performance limits. Materials chemist Freddy Kleitz from the Faculty of Chemistry of the University of Vienna and international ...

Uber filed paperwork for IPO: report

December 8, 2018

Ride-share company Uber quietly filed paperwork this week for its initial public offering, the Wall Street Journal reported late Friday.


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Jun 03, 2010
yield and stop sign violation cameras?
not rated yet Jun 03, 2010
Would be nice to have murder and rape prevention cameras.

More people are murdered in America each year by other Americans than the total number of all American "war on terror" casualties combined.

Our troops are in the wrong place. They should be here hunting down murderers, gang members, and drug dealers.
not rated yet Jun 03, 2010
Actually a subject, finally, on this site I know a lot about. This system will rely entirely on the database; other than the database, there is nothing new about this system. Consistency will be key in the database, and really, all the objects need to be defined by the same person in the same fashion. For example, you can't call one angle of the objects above backpack, and another angle called napsack. This will be very difficult to maintain and is the primary reason it hasn't been implemented effectively yet. You run into a similar problem with voice recognition and dictation software. The image processing techniques are nothing new, cookie cutter. Good luck guys, you'll need it.
not rated yet Jun 03, 2010
I can't help but wonder, is that first picture a human-generated or a computer-generated example of the algorithm? I slightly suspect that it's computer-generated, as the water is described as being green. I mean sure it's mostly green, but that's just because it's reflecting the trees. Wouldn't most humans ignore that and describe the water as blue?

Or, perhaps, that's what the computer would say if it worked yet but it's really just a human imitating the expected computer behavior?
not rated yet Jul 17, 2010
Infrared cameras are devices that forms an image using infrared radiation would be great.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.