Team breaks world record for fast, accurate AI training

November 7, 2018, Hong Kong Baptist University
Diagram showing data transmission of a 5-layer model. Credit: HKBU

Researchers at Hong Kong Baptist University (HKBU) have partnered with a team from Tencent Machine Learning to create a new technique for training artificial intelligence (AI) machines faster than ever before while maintaining accuracy.

During the experiment, the team trained two popular deep neural networks called AlexNet and ResNet-50 in just four minutes and 6.6 minutes respectively. Previously, the fastest time was 11 minutes for AlexNet and 15 minutes for ResNet-50.

AlexNet and ResNet-50 are built on ImageNet, a large-scale dataset for visual recognition. Once trained, the system was able to recognise and label an object in a given photo. The result is significantly faster than previous records and outperforms all other existing systems.

Machine learning is a set of mathematical approaches that enable computers to learn from data without explicitly being programmed by humans. The resulting algorithms can then be applied to a variety of data and tasks used in AI.

The HKBU team comprises Professor Chu Xiaowen and Ph.D. student Shi Shaohuai from the Department of Computer Science. Professor Chu said, "We have proposed a new optimised training method that significantly improves the best output without losing accuracy. In AI training, researchers strive to train their networks faster, but this can lead to a decrease in accuracy. As a result, training machine-learning models at high speed while maintaining accuracy and precision is a vital goal for scientists."

Professor Chu said the time required to train AI is affected by both computing time and communication time. The research team attained breakthroughs in both aspects to create this record-breaking achievement.

This included adopting a simpler computational method known as FP16 to replace the more traditional one, FP32, making computation much faster without losing . As communication is affected by the size of data blocks, the team came up with a communication technique named "tensor fusion," which combines smaller pieces of data into larger ones, optimising the transmission pattern and thereby improving the efficiency of communication during AI training.

This can be adopted in large-scale image classification, and it can also be applied to other AI applications, including machine translation; natural language processing (NLP) to enhance interactions between human language and computers; medical imaging analysis; and online multiplayer battle games.

Explore further: Supercomputing speeds up deep learning training

More information: Xianyan Jia et al. Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes. arXiv:1807.11205 [cs.LG].

Related Stories

Supercomputing speeds up deep learning training

November 13, 2017

A team of researchers from the University of California, Berkeley, the University of California, Davis and the Texas Advanced Computing Center (TACC) published the results of an effort to harness the power of supercomputers ...

Training artificial intelligence with artificial X-rays

July 6, 2018

Artificial intelligence (AI) holds real potential for improving both the speed and accuracy of medical diagnostics. But before clinicians can harness the power of AI to identify conditions in images such as X-rays, they have ...

Restoring balance in machine learning datasets

October 11, 2018

If you want to teach a child what an elephant looks like, you have an infinite number of options. Take a photo from National Geographic, a stuffed animal of Dumbo, or an elephant keychain; show it to the child; and the next ...

Scientists improve deep learning method for neural networks

August 17, 2018

Researchers from the Institute of Cyber Intelligence Systems at the National Research Nuclear University MEPhI (Russia) have recently developed a new learning model for the restricted Boltzmann machine (a neural network), ...

Recommended for you

When more women make decisions, the environment wins

March 21, 2019

When more women are involved in group decisions about land management, the group conserves more—particularly when offered financial incentives to do so, according to a new University of Colorado Boulder study published ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.