3D motion tracking system could streamline vision for autonomous tech

3D motion tracking system could streamline vision for autonomous tech
A graphene-based transparent photodetector array (acting as two layers of sensors in a camera) measures the focal stack images of a point object simulated by focusing a green laser beam onto a small spot in front of the lens inside Ted Norris’ lab on North Campus in Ann Arbor, MI on January 27, 2021. Norris and his team fabricated a prototype of transparent photodetector arrays with graphene on glass and used two pieces of the detector arrays separated slightly apart and behind an imaging lens to demonstrate its potential applications in 3D object tracking tasks. With this prototype’s application this will aid in autonomous driving and robotics, responding to moving objects that require their perception units to obtain not only what they are but also where and how far they are in real time. Credit: Robert Coelius/Michigan Engineering, Communications and Marketing

A new real-time, 3D motion tracking system developed at the University of Michigan combines transparent light detectors with advanced neural network methods to create a system that could one day replace LiDAR and cameras in autonomous technologies.

While the technology is still in its infancy, future applications include automated manufacturing, biomedical imaging and autonomous driving. A paper on the system is published in Nature Communications.

The imaging system exploits the advantages of transparent, nanoscale, highly sensitive graphene photodetectors developed by Zhaohui Zhong, U-M associate professor of electrical and computer engineering, and his group. They're believed to be the first of their kind.

"The in-depth combination of graphene nanodevices and machine learning algorithms can lead to fascinating opportunities in both science and technology," said Dehui Zhang, a doctoral student in electrical and computer engineering. "Our system combines computational power efficiency, fast tracking speed, compact hardware and a lower cost compared with several other solutions."

The graphene photodetectors in this work have been tweaked to absorb only about 10% of the light they're exposed to, making them nearly transparent. Because graphene is so sensitive to light, this is sufficient to generate that can be reconstructed through computational imaging. The photodetectors are stacked behind each other, resulting in a compact system, and each layer focuses on a different focal plane, which enables 3D imaging.

But 3D imaging is just the beginning. The team also tackled real-time motion tracking, which is critical to a wide array of autonomous robotic applications. To do this, they needed a way to determine the position and orientation of an object being tracked. Typical approaches involve LiDAR systems and light-field cameras, both of which suffer from significant limitations, the researchers say. Others use metamaterials or multiple cameras. Hardware alone was not enough to produce the desired results.

They also needed deep learning algorithms. Helping to bridge those two worlds was Zhen Xu, a doctoral student in electrical and computer engineering. He built the optical setup and worked with the team to enable a neural network to decipher the positional information.

3D motion tracking system could streamline vision for autonomous tech
A graphene-based transparent photodetector array (acting as two layers of sensors in a camera) measures the focal stack images of a point object simulated by focusing a green laser beam onto a small spot in front of the lens inside Ted Norris’ lab on North Campus in Ann Arbor, MI on January 27, 2021. Norris and his team fabricated a prototype of transparent photodetector arrays with graphene on glass and used two pieces of the detector arrays separated slightly apart and behind an imaging lens to demonstrate its potential applications in 3D object tracking tasks. With this prototype’s application this will aid in autonomous driving and robotics, responding to moving objects that require their perception units to obtain not only what they are but also where and how far they are in real time. Credit: Robert Coelius/Michigan Engineering, Communications and Marketing

The neural network is trained to search for specific objects in the entire scene, and then focus only on the object of interest—for example, a pedestrian in traffic, or an object moving into your lane on a highway. The technology works particularly well for stable systems, such as automated manufacturing, or projecting human body structures in 3D for the .

"It takes time to train your neural network," said project leader Ted Norris, professor of electrical and computer engineering. "But once it's done, it's done. So when a camera sees a certain scene, it can give an answer in milliseconds."

Doctoral student Zhengyu Huang led the algorithm design for the . The type of algorithms the team developed are unlike traditional signal processing algorithms used for long-standing imaging technologies such as X-ray and MRI. And that's exciting to team co-leader Jeffrey Fessler, professor of electrical and computer engineering, who specializes in medical imaging.

"In my 30 years at Michigan, this is the first project I've been involved in where the technology is in its infancy," Fessler said. "We're a long way from something you're going to buy at Best Buy, but that's OK. That's part of what makes this exciting."

The team demonstrated success tracking a beam of light, as well as an actual ladybug with a stack of two 4x4 (16 pixel) graphene photodetector arrays. They also proved that their technique is scalable. They believe it would take as few as 4,000 pixels for some practical applications, and 400x600 pixel arrays for many more.

While the technology could be used with other materials, additional advantages to are that it doesn't require artificial illumination and it's environmentally friendly. It will be a challenge to build the manufacturing infrastructure necessary for mass production, but it may be worth it, the researchers say.

"Graphene is now what silicon was in 1960," Norris said. "As we continue to develop this technology, it could motivate the kind of investment that would be needed for commercialization."

The paper is titled "Neural Network Based 3D Tracking with a Graphene Transparent Focal Stack Imaging System."


Explore further

A 3-D camera for safer autonomy and advanced biomedical imaging

Journal information: Nature Communications

Citation: 3D motion tracking system could streamline vision for autonomous tech (2021, April 23) retrieved 9 May 2021 from https://phys.org/news/2021-04-3d-motion-tracking-vision-autonomous.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
92 shares

Feedback to editors

User comments