Scientists improve deep learning method for neural networks

August 17, 2018, National Research Nuclear University
Credit: CC0 Public Domain

Researchers from the Institute of Cyber Intelligence Systems at the National Research Nuclear University MEPhI (Russia) have recently developed a new learning model for the restricted Boltzmann machine (a neural network), which optimizes the processes of semantic encoding, visualization and data recognition. The results of this research are published in the journal Optical Memory and Neural Networks.

Today, deep neural networks with different architectures, such as convolutional, recurrent and autoencoder networks, are becoming an increasingly popular area of research. A number of high-tech companies, including Microsoft and Google, are using deep neural networks to design intelligent systems.

In systems, the processes of feature selection and configuration are automated, which means that the networks can choose between the most effective algorithms for hierarchal feature extraction on their own. Deep learning is characterized by learning with the help of large samples using a single optimization algorithm. Typical optimization algorithms configure the parameters of all operations simultaneously, and effectively estimate every neural parameter's effect on error with the help of the so-called backpropagation method.

"The neural networks' ability to learn on their own is one of their most intriguing properties," explained Vladimir Golovko, professor at the MEPhI Institute of Cyber Intelligence Systems. "Just like biological systems, neural networks can model themselves, seeking to develop the best possible model of behavior."

In 2006, the sphere of saw a breakthrough when Geoffrey Hinton published a research paper on pre-training neural networks. He stated that multilayer neural networks could be pre-trained by training one layer at a time with the help of the restricted Boltzmann machine and then fine-tuning them using backpropagation. These networks were named deep belief networks, or DBN.

Golovko analyzed the main issues and paradigms of deep machine learning and suggested a new learning method for the restricted Boltzmann machine. The researcher demonstrated that the classical rule of training this neural network is a particular case of the method he developed.

"American scientists Minsky and Papert once showed that from the point of view of pattern classification, the single layer perceptron with the threshold activation function forms a linear separating surface, which is the reason why it cannot solve the 'exclusive or' problem," Golovko noted. "This led to pessimistic conclusions about the further development of neural networks. However, the last statement is only true for a single layer perceptron with a threshold or a monotonic continuous activation function, for instance, a sigmoid function. When one uses the signal activation function, the single layer perceptron can solve the 'exclusive or' problem, since it can divide the area of ones and zeros into classes with the help of two straight lines."

The research also involved an analysis of the prospects of using for compression, visualization and recognition of data. Moreover, Golovko also suggested a new approach to implementation of semantic encoding, or hashing, which is based on the use of deep auto-associative neural networks.

This deep learning method might be very useful for training search engines' , the author states, as it will improve the speed of searching for relevant images.

These findings have great practical value: they have already found application in the spheres of computer vision, speech recognition and bioinformatics.

Explore further: DeepMind uses neural network to help explain meta-learning in people

More information: V. A. Golovko. Deep learning: an overview and main paradigms, Optical Memory and Neural Networks (2017). DOI: 10.3103/S1060992X16040081

Related Stories

Training artificial intelligence with artificial X-rays

July 6, 2018

Artificial intelligence (AI) holds real potential for improving both the speed and accuracy of medical diagnostics. But before clinicians can harness the power of AI to identify conditions in images such as X-rays, they have ...

Recommended for you

Technology near for real-time TV political fact checks

January 18, 2019

A Duke University team expects to have a product available for election year that will allow television networks to offer real-time fact checks onscreen when a politician makes a questionable claim during a speech or debate.

Privacy becomes a selling point at tech show

January 7, 2019

Apple is not among the exhibitors at the 2019 Consumer Electronics Show, but that didn't prevent the iPhone maker from sending a message to attendees on a large billboard.

China's Huawei unveils chip for global big data market

January 7, 2019

Huawei Technologies Ltd. showed off a new processor chip for data centers and cloud computing Monday, expanding into new and growing markets despite Western warnings the company might be a security risk.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.