AI depicts 3D social interactions between animals

Accurate quantification of multi-animal behavior plays a pivotal role in unraveling the intricacies of animal social interactions with far-reaching applications in neuroscience and ecology.

In a study published in Nature Machine Intelligence, researchers from the Brain Cognition and Brain Disease Institute (BCBDI) of the Shenzhen Institute of Advanced Technology of the Chinese Academy of Sciences have proposed a few-shot learning artificial intelligence (AI) framework, the Social Behavior Atlas (SBeA), for multi-animal three-dimensional (3D) social pose estimation, identification, and behavior embedding.

While recent advances in deep learning methods such as multi-animal DeepLabCut, SLEAP, and SIPEC have improved the accessibility of quantifying high-dimensional social behaviors in animals, including pose estimation, identity recognition, and behavior classification, the applications of these methods are restricted by the availability of insufficiently annotated datasets.

In this study, researchers developed a continuously occluded copy-paste algorithm (COCA) as a universal data augmenter to reduce data annotation to about 400 frames in the multi-animal pose estimation step. The number of required data is equivalent to single-animal annotations. COCA promotes SBeA to reach higher performance than state-of-the-art methods. Additionally, SBeA can reconstruct the 3D poses of social animals combined with the camera array.

The proposed bidirectional transfer learning in SBeA can recognize each animal's identity during social interaction without the need for manual annotations. This resolves the problem of perplexing identity recognition for animals with similar appearances, even for professional human annotators.

The 3D social poses with identities were further decomposed and clustered by the unsupervised social behavior classification of SBeA, which classifies social behavior without predefined categories.

SBeA helps researchers identify undefined subtle social behavior modules in Shank3B mutant mice, an animal model used to simulate autism spectrum disorders. It indicates the existence of unknown neural modulation mechanisms behind subtle social behaviors. In addition to mice, SBeA effectively identifies subtle social behavior in other species such as birds and dogs. Neuroscience and ecology would benefit from the accurate animal social behavior quantification provided by SBeA.