Researchers are working on new smart cameras to help the visually impaired
Standing in the grocery store, you scan the peanut butter jars looking for the one on your list. Your eyes flit from label to label until they land on the familiar red, blue and green jar, and you reach to pick it up and place it in your cart. Then, it's on to the next item on your list.
Picking something up at the grocery store seems simple, but it's a task that relies heavily on your sense of sight. You need to find the right item, pick it up and then place it safely in your basket. But for those with visual impairments, it's a task that's often difficult or impossible for them to do on their own.
To help make shopping easier for the visually impaired, Jack Carroll—a distinguished professor in the College of Information Sciences and Technology (IST) at Penn State—has completed a study that explored how smart cameras could eventually guide visually impaired shoppers to find the items they need. Carroll worked on the study with Mary Beth Rosson, interim dean of the College of IST, as well as graduate student Jake Weidman as part of a $10 million National Science Foundation-funded project that's seeking to replicate the human vision system using information technology (IT).
"This is one of the biggest computing projects Penn State's been involved with," said Carroll. "I've been working with what we do with the technology after the camera is available, specifically in a shopping scenario. Shopping is so universal, and it signifies normalcy—it's important that the visually impaired be able to do it, too."
The project—dubbed Visual Cortex on Silicon—is led by Penn State Computer Science and Engineering Distinguished Professor Vijaykrishnan Narayanan, and Penn State is the lead university alongside seven other universities.
"This project brings together the strengths and efforts of technical leaders in multiple disciplines," said Narayanan. "It draws on the expertise of several industry and national lab partners. A major goal is advancing the technologies available for the visually impaired."
Visual Cortex on Silicon aims to create sophisticated "smart" cameras that will replicate or even surpass the abilities of the human vision system, which can interpret complex scenes and complete complicated tasks while using less than 20 watts of power. The cameras will be used in mobile devices, such as smartphones, so energy efficiency is key to fitting a complex system into a small piece of hardware.
Researchers—who are each tackling individual pieces of the larger project—hope the cameras will not just be able to record images but also interpret them. For example, the cameras will not just recognize an item as a jar of peanut butter but also be able to determine whether it's the specific kind the user needs—a scenario that was the focus of Carroll's study.
Carroll and his team mocked up a grocery aisle in his lab, complete with food and other items filling the shelves. He wanted to see how the visually impaired participants would interact with the camera, which would scan the items on the shelves and alert the participant when the item they were searching for was near.
But, since the new cameras haven't been finished yet, an actual prototype wasn't available to test. So, Carroll and his team used the "Wizard of Oz" technique, in which the participants interact with an apparatus that's being controlled by an out-of-sight person.
"In the movie, the wizard was actually an illusion controlled by a man behind the curtain. These types of experiments are similar," said Carroll. "For this one, the cameras aren't ready so we had to simulate how they might work in the future. It allows us to see how participants would interact with the camera and tweak any problems that arise. We don't want to wait until the cameras are finished being developed and then start from scratch on the apparatus."
Participants strapped iPods on their foreheads that would send a video feed of the shelf to a graduate student who would then list the items currently in view. This simulates what they would eventually be able to do with one of the new cameras.
Once the participant knew they were standing in front of the item they needed, the next step was guiding them to pick it off the shelf. This proved a bit more difficult than the first step.
"At first we tried using verbal cues to help them locate it—'higher' or 'lower,' for example, as the participants moved their hands," says Carroll. "This worked well, but we didn't want to stop there and are also looking into other guiding methods."
Carroll is now investigating a method using haptic sensors, which give tactile feedback through vibration. The "buzzes" would become stronger as the participants' hands got closer to their desired items. Carroll says he likes this method because a lot of speech is needed for even small directions and other people's speech can interfere with feedback.
"Jack's work is key because it enables the new cameras being researched and built in this project to meet expectations and understand what kind of assistance visually impaired people will need from these new technologies," Narayanan said.
Shortly after Carroll's study was completed, the Visual Cortex on Silicon team was able to use research completed by other team members on hardware design and vision algorithms to build an actual prototype of the apparatus. The team demonstrated the system at the project's annual review.
Narayanan says the Visual Cortex on Silicon project—which is funded through September 2018—came about after a previous effort designing cameras.
"In a prior project, we used brain-inspired algorithms and configurable architectures to detect and track objects for aerial cameras," Narayanan said. "The system was more energy efficient compared to other current approaches, and it made us want to work toward cameras that are applicable across a wider spectrum of vision-related tasks."