Mandarin language learners get a boost from AI

IBM Research and Rensselaer Polytechnic Institute (RPI) are collaborating on a new approach to help students learn Mandarin. The strategy pairs an AI-powered assistant with an immersive classroom environment that has not been used previously for language instruction. The classroom, called the Cognitive Immersive Room (CIR), makes students feel as though they are in restaurant in China, a garden, or a Tai Chi class, where they can practice speaking Mandarin with an AI chat agent. The CIR was developed by the Cognitive and Immersive Systems Lab (CISL), a research collaboration between IBM Research and RPI.

When learning a new language, especially one as difficult as Mandarin, it's important that students have many opportunities to speak and practice their conversational skills. Acquiring a new language naturally, through cultural immersion, may be more effective than non-immersive practices. Yet, experiential learning is unavailable to most new learners of Mandarin. That's why we developed the CIR.

An immersive gamification classroom environment

The CIR brings together several state-of-the-art technologies such as speech-to-text, natural language understanding, and computer vision to enable immersion and natural multi-modal dialogue. The room includes a 360-degree panoramic display system, an audio system, multiple cameras, multiple Kinect devices, and multiple microphones, as well as computer systems to support the AI technologies, some located in the room and others in the cloud.

Our goal is to combine cognitive, immersive technologies with game-playing elements to enable students to experience a cultural environment, practice daily tasks, and get help from intelligent agents. With the Mandarin Project, we use IBM Watson within the CIR as a conversational agent to engage students while they learn the language. Our approach involves IBM Watson speech recognition and natural language understanding technologies for English and Chinese.

One of the biggest obstacles in learning a foreign language through immersion is students' fears of being judged by native speakers. In the CIR, however, students can work with virtual conversation partners to practice vocabulary and pronunciation without the pressures of a real-world setting.

Many language learning researchers are working with virtual or augmented reality, but we are investigating human-scale, immersive, gaming environments where students can physically walk around without having to wear specialized equipment. This reflects a broader trend in human-computer interaction, as our engagement with information in our daily lives becomes ever more immersive and our interactions with intelligent machines shift towards partnership.

Our system was tested in a Chinese 1 class on RPI's campus in the late fall of 2017. This fall, we will continue using AI as a teaching tool in Chinese 1, 2 and 3 classes, with the goal of offering a new class merging Chinese 1-3 in the summer of 2019.

The research behind—and beyond—the classroom

As part of the IBM AI Horizons Network, CISL creates and uses such technologies as context-aware systems, pitch tone contour analysis, language-switch between Chinese and English in a natural language conversation between people and AI agents, multi-user multimodal interaction, reasoning and planning in the decision-making process and multimodal storytelling with automatic generation of narrative, visualization and sonification.

The CIR is one of four use cases in CISL. The lab is also working to augment group intelligence for critical decision-making in real-world environments like boardrooms. In the future, these new cognitive and immersive systems could fundamentally change the way people live and work. The early prototypes of CIR have also provided opportunities for user interaction research, enabling us to find and address new challenges in language learning.