Plant phenotyping: Deep learning and 3D point cloud technology in overcoming reconstruction challenges

3D point cloud technology revolutionizes non-invasive measurement of plant phenotypic parameters, offering vital data for agriculture and research.

Current research focuses on overcoming the limitations of 2.5D imaging and occlusions. Methods such as structure from motion, multi-view stereo, and advanced active 3D reconstruction techniques are being explored for this purpose. However, issues persist with incomplete data acquisition and the inaccuracy of phenotypic parameter extraction due to plant occlusions and environmental factors.

Recently, integrating deep learning with point cloud analysis and innovative techniques like Reinforcement Learning Agent Controlled GAN Network and multi-scale geometry-aware Transformer networks show promise in addressing these challenges by enhancing point cloud completion and maintaining the integrity of the original data.

In November 2023, Plant Phenomics published a research article titled "Point Cloud Completion of Plant Leaves under Occlusion Conditions Based on Deep Learning."

This research utilized neural network-based point cloud completion, specifically the PF-Net algorithm, to reconstruct 3D models of flowering Chinese Cabbage leaves, which are challenging due to their complex structures. A dataset of cabbage leaf point clouds was created and trained using PF-Net, with incomplete point clouds obtained using an Azure Kinect camera.

The neural network supplemented these incomplete point clouds, achieving full 3D reconstruction, which was then compared with the MVS-SFM algorithm for accuracy.

The results indicated that the PF-Net successfully completed point clouds of various shapes and bending degrees from both MVS-SFM and Azure Kinect datasets, showcasing its strong ability to complete complex structures. However, the completion was less effective in areas with multiple missing sections. As the missing ratio increased, the completion effect diminished, particularly in the holes of the leaf point cloud, suggesting that while the network learned structural relationships effectively, it struggled with larger missing areas.

Under natural occlusion conditions, the PF-Net improved the completeness of leaf point clouds compared to the MVS-SFM algorithm, but struggled with multiple occlusions or high missing ratios. The research also examined the reconstruction precision and leaf area extraction results.

About half of the point pairs in the reconstructed point clouds had a distance error of less than 2mm, indicating moderate consistency with the MVS-SFM method. The leaf area extraction analysis revealed that the accuracy of leaf area estimation improved after point cloud completion, with results close to those obtained using the MVS-SFM algorithm.

Despite these promising results, the study acknowledged gaps in completing real-world incomplete point clouds and the challenge of point cloud orientation due to the rotation invariance characteristic of point clouds.

Future improvements include developing more realistic training datasets and addressing the orientation estimation challenge, possibly through a multi-stage completion network that can learn and adjust the model's proper pose before completing it.

Overall, the research offers a new perspective for plant phenotype studies using active sensors and confirms the potential of deep learning in enhancing phenotypic parameter estimation precision.