Combination of crowdsourcing and computer vision could identify individuals within endangered populations

May 1, 2013, Massachusetts Institute of Technology
MIT algorithms analyze the complex patterns of jeweled geckos to match up images of particular individuals. Credit: Dr. Paddy Ryan/The National Heritage Collection

Keeping track of individuals in an endangered population of animals is a cumbersome and time-consuming task. Conservationists physically tag animals in the wild to better follow them over time. But tagging can be intrusive for many species, and difficult to accomplish in larger populations. As an alternative, scientists have photographed animals in their natural environments and catalogued the images, along with information such as individuals' dimensions and geographic locations.

However, as accumulate, picking out individuals from among thousands of pictures can be a monumental task. Sai Ravela, a principal research scientist in MIT's Department of Earth, Atmospheric and Planetary Sciences, estimates that manually sifting through a catalog of 10,000 images can take one person 15 years.

"It's an enormous amount of time," Ravela says. "You're reaching the edges of what people want to do with their lives."

Now Ravela and his colleagues at MIT have developed computer software that automates much of the image-matching process. The system, which they've named SLOOP, sifts through thousands of images, using pattern-recognition algorithms to analyze features in each image, such as an animal's arrangement of stripes or spots. The system then identifies an average of 20 most likely matches for an individual.

From there, the researchers turned to crowdsourcing, asking online users to pick the most similar pair. Based on such feedback, the system reorders the list, paring it down to fewer images likely to depict the same individual.

"It's sort of like , in that you type in a and you get back potential matches, but ultimately you're the judge of what's the best match," Ravela says. "It makes ' lives a lot easier than having to go through an entire catalog."

Ravela, along with undergraduates James Duyck and Chelsea Finn, are applying the system to various endangered and threatened species, including , and skinks. The group will present details of the system at the Mexican Conference on Pattern Recognition in June.

Beyond facial recognition

In recent years, pattern-recognition algorithms have mostly been developed for facial recognition, matching individuals by features such as eyes, noses and mouths. But Ravela says these algorithms are not sophisticated enough to parse out the enormous complexity in animal patterning.

"To distinguish an individual, like a salamander Bob from a salamander Jill, is tough," Ravela says. "On the whole, the variability we see within species really tests our assumptions of what makes a good algorithm."

His group has developed multiple algorithms to identify matching patterns, including several that adjust for changes in an animal's lighting, orientation and geometry, and other algorithms that overlay images, comparing the positioning of spots or stripes. The researchers use combinations of algorithms to match images, depending on a species's particular features.

A user can upload an image to the system, along with any accompanying information such as an individual's weight, size and location. Depending on the type of animal being catalogued, the user performs a few simple additional steps, such as marking reference points like a lizard's eyes, nose and shoulder. The system then takes over, using algorithms to adjust the image and ranking its similarity to the rest of the catalog's images.

Animal tracking

The image-matching system is currently used by New Zealand's Department of Conservation to track threatened populations of skinks—small lizards that, thanks to a recovery program, have multiplied in recent years. The species's rebound, while a good sign, poses a more difficult tracking problem for as the population grows—a problem that SLOOP is helping to solve.

"We had reached the point where two-thirds of our monitoring effort was spent in front of the computer screen, and only one-third in the field directly monitoring the lizards," says Andy Hutcheon, program manager for the Grand and Otago Skink Recovery Plan in New Zealand. "That's a lot of eyestrain."

Using the system, Hutcheon and his colleagues have quickly sorted out individuals from among more than 26,000 existing images. So far, they have found 15 cases in which human error incorrectly identified an individual as two distinct lizards.

Ultimately, SLOOP may help scientists answer broader questions about animal behavior, such as on species' breeding habits and migration patterns. For example, Hutcheon says the image-matching system has picked out at least six individuals among the entire population that have migrated between study sites, in some cases traveling up to several miles.

"Many New Zealand natives are characterized by lack of detailed data," Hutcheon says. "New application of technology that can help us to understand their numbers and life history can only help with their conservation."

Crowd conservation

Going a step further, Ravela's team looked at the potential for crowdsourcing to further speed up the image-matching process.

As an experiment, the researchers posted thousands of images of salamanders, in groups of four, on Amazon's Mechanical Turk, an online crowdsourcing marketplace. The researchers asked users to rank the images in order of similarity.

To "rate" a user's ability, the system was programmed to know the answer to three of every four images. If a user correctly ranked these known images, the system accepted the user's fourth answer. As incentive to participate, the researchers offered users a modest monetary reward for correct answers.

"We found about a third of the people who came were really good pattern-matchers," Ravela says. "One guy had a 99.96 percent performance, and stayed for 3,000 comparisons."

By combining computer-vision algorithms with crowdsourcing, the team is able to quickly identify image matches among thousands of photos with 97 percent accuracy.

The researchers are now working to further automate the system. For example, Ravela's students are developing algorithms that will automatically separate and outline an animal from an image background. The problem is a difficult one, as the system would have to distinguish between, for instance, a lizard's leg and a nearby twig. To solve these problems, Finn is developing matching algorithms using feature geometry, while Duyck is combining multiple pattern-matching algorithms to improve image-ranking.

"These are incredibly hard problems," Ravela says. "On the other hand, doing these things manually is extraordinarily time-consuming. Is there a sweet spot between the two that allows us to solve real problems? That's what SLOOP is trying to do."

Explore further: Iris recognition report evaluates 'needle in haystack' search capability

Related Stories

Special software helps to save species

August 1, 2011

With the aim of better protecting endangered species, game wardens are studying the behavior of surviving great apes in the wild. This is often painstaking work because it is difficult to distinguish between different individuals. ...

Passports for penguins

June 28, 2008

Ground-breaking technology that will enable biologists to identify and monitor large numbers of endangered animals, from butterflies to whales, without being captured, will be shown to the public for the first time at this ...

Google image search gets a 'swirl'

November 17, 2009

Google Labs on Tuesday brought more focus to finding pictures online, adding a "Swirl" tool that automatically groups similar images into categories presented on results pages.

Making crowdsourcing easier

August 24, 2012

Crowdsourcing is a technique for farming out labor-intensive tasks over the Internet by splitting them into small chunks that dozens, hundreds or even thousands of people complete at their desks for a few cents each.

Recommended for you

AI and 5G in focus at top mobile fair

February 24, 2018

Phone makers will seek to entice new buyers with better cameras and bigger screens at the world's biggest mobile fair starting Monday in Spain after a year of flat smartphone sales.

Google Assistant adds more languages in global push

February 23, 2018

Google said Friday its digital assistant software would be available in more than 30 languages by the end of the years as it steps up its artificial intelligence efforts against Amazon and others.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.