Computer Scientists Build Pedestrian Remover

Aug 05, 2010
The dog stands alone (bottom image) after a UC San Diego pedestrian remover automatically removed the man walking the dog (top image) and filled in the hole with building, grass, curb and sidewalk.

(PhysOrg.com) -- Imagine encountering leashed dogs without dog walkers, or shoes filled just with ankles - when scoping out potential apartments using Google Street View. These are the sorts of visual hiccups that an experimental computer vision system occasionally generates when it automatically removes individual pedestrians from images that populate Google Street View.

Computer science graduate student Arturo Flores from the University of California, San Diego developed this proof-of-concept system. Flores and UC San Diego computer science professor Serge Belongie presented the work in June 2010 at the IEEE International Workshop on Mobile Vision. Their paper:“Removing pedestrians from Google Street View images.

The as-yet unnamed system removes pedestrians from urban scenes pulled from Google Street View - which provides panoramic views of cities, towns and rural areas across the world. Street views are constructed by stitching together overlapping images taken from a moving vehicle.

Removing Pedestrians

The UC San Diego project explores one way that could be used to preserve privacy in public environments in our digital age.

The system removes pedestrians and replaces the holes in the images with an approximation of the actual background behind each pedestrian. These corresponding background pixels are pulled from the image taken right before or right after the image in question.

When the automatic pedestrian remover replaced the woman (top image), the umbrella remained (bottom image).

One next step, according to Flores, is to remove groups of pedestrians from single images.

Street View currently blurs faces and license plates from its images. Nevertheless, clothes, body shape, and height combined with geographical location can be enough to make some pedestrians personally identifiable even if the face is blurred out, say Flores and Belongie in their paper.

The pedestrian removal is relatively “ghost free” - meaning that the artifacts caused by the pixel swapping are usually not distracting. But the pedestrian remover does occasionally produce strange results - like dogs on leashes with no owners, and shoes with feet but nothing else.

In addition, the system struggles to generate background pixels when the pedestrian happens to be walking in the same direction as the vehicle at just the right speed. In these cases, the pedestrian may cover up the same spot in multiple frames, foiling the computer scientists’ pixel-swapping approach to removing pedestrians.

The pedestrian remover only works in urban settings - where the pixels blocked by people are often “on a dominant planar surface” - which makes them simpler to replace.

The system, for example, can replace the pixels blocked by a person walking by a mural of horses grazing in a pasture. But the system cannot replace the pixels behind a person on a country road walking by actual horses grazing in a pasture, because this background is not predominately flat.

The man walking past the glass door (top image) is automatically removed and replaced with the actual glass door (bottom image)

It All Started in Class

Flores developed the project during CSE 190A, a project-based computer vision and machine learning class taught by Serge Belongie, a professor in the Department of Computer Science and Engineering (CSE) at the UC San Diego Jacobs School of Engineering.

Belongie encourages his students to take on computer vision projects that tap freely available tools and datasets. Flores, for example, leveraged the pedestrian detector for created by professor Bastian Leibe from RWTH Aachen University. From this technological base, Flores developed his automated system that replaces pedestrians with the actual urban scene the people are blocking.

“This is a cute idea that, as far as we know, has not been explored,” said Belongie.

While students are free to choose their own CSE 190A projects, Belongie keeps a running list of project ideas, such as analyzing coral reef videos, and finding swimming pools in neighborhoods with aerial photos. The project blogs for Flores and his Winter 2010 CSE 190A classmates are here. Check out some of the"dancer detector" videos here.

“I’m always trying to get the students to think about applying computer vision to real-world data,” said Belongie. “CSE 190A is a perfect opportunity for students to do so.”

Explore further: Computerized emotion detector

More information: Paper: Removing pedestrians from Google Street View images presented in June 2010 at the IEEE International Workshop on Mobile Vision.

Related Stories

Deciphering the movement of pedestrians in a crowd

Apr 13, 2010

(PhysOrg.com) -- How do pedestrians move in the street? How do they interact? French researchers from the Université Toulouse, in partnership with the Swiss Federal Institute of Technology, Zurich, ...

Google removes street images over privacy complaints

Mar 20, 2009

US software giant Google said Friday it had removed several images from its Street View software, which allows web surfers to view parts of 25 British cities, after users raised privacy concerns.

Pedestrian crossings could be monitored

Sep 18, 2009

A team of researchers from the University of Castilla-La Mancha (Spain) has developed an intelligent surveillance system able to detect aberrant behaviour by drivers and people on foot crossing pedestrian ...

Recommended for you

Computerized emotion detector

14 hours ago

Face recognition software measures various parameters in a mug shot, such as the distance between the person's eyes, the height from lip to top of their nose and various other metrics and then compares it with photos of people ...

Cutting the cloud computing carbon cost

Sep 12, 2014

Cloud computing involves displacing data storage and processing from the user's computer on to remote servers. It can provide users with more storage space and computing power that they can then access from anywhere in the ...

Teaching computers the nuances of human conversation

Sep 12, 2014

Computer scientists have successfully developed programs to recognize spoken language, as in automated phone systems that respond to voice prompts and voice-activated assistants like Apple's Siri.

Mapping the connections between diverse sets of data

Sep 12, 2014

What is a map? Most often, it's a visual tool used to demonstrate the relationship between multiple places in geographic space. They're useful because you can look at one and very quickly pick up on the general ...

User comments : 6

Adjust slider to filter visible comments by rank

Display comments: newest first

jnagyjr
Aug 05, 2010
This comment has been removed by a moderator.
DaveGee
5 / 5 (1) Aug 05, 2010
Most impressive indeed... I guess all those lawyers queuing umpteen thousand invasion of privacy law suits are gonna have to figure out some other way to steal money and make it their own...
Daniel_Cousens
not rated yet Aug 05, 2010
Quickest way to get the impression the entire worlds a ghost town. Interesting software none the less.
david13579
not rated yet Aug 06, 2010
Ghost free? you can see a very very faint image of the person, the most visible being the one with the umbrella.

I think that anything that happens out, on the street is not private and should not be censored.
_Scott_
Aug 06, 2010
This comment has been removed by a moderator.
Jeremyh
not rated yet Aug 06, 2010
I would think that as the spatial information of the background is computable, it would be possible to remove the ghosting produced by the (pixle swapping)method. It should be possible to work out which parts of the image are moving, and which are static, as the spatial information should be able to show depth as well as any movement between shots. Using all of this information it should even be possible to make virtual 3D environments for all the Google street map information, thus allowing people to walk up to something or actualy walk on the side walk down the street using an avatar of sorts.
callywally
not rated yet Aug 06, 2010
“This is a cute idea that, as far as we know, has not been explored,” said Belongie.


Haha, funny. The idea of identifying people in images and then replacing them with proper context would be readily available to anyone skilled in the art of image processing.

What about this project? http://grail.cs.w...ncement/

I'm sure this will be implemented soon enough in google streetview, but cute application for a job in google.
mrlewish
Aug 06, 2010
This comment has been removed by a moderator.
rgwalther
not rated yet Aug 07, 2010
Welcome to Photo Shop 1998

What about Photoshop 1990? (v.07). Your reference to 1998 is either random or makes you