The robot that learns everything from scratch

June 8, 2015 by Steinar Brandslet, Norwegian University of Science and Technology
Axel Tidemann, [self.] and Øyvind Brandtsegg. Credit: Ole Morten Melgård, NTNU

Two researchers at NTNU have made a robot that learns like a young child. At least, that's the idea. The machine starts with nothing—it has to learn everything from scratch.

"We're still pretty far away from accurately modelling all aspects of a living child's brain, but the algorithms that handle sound and image processing are inspired by biology," says Professor Øyvind Brandtsegg at NTNU.

The machine is called [self.]. It analyses sound through a system based on the human ear, and learns to recognize images using a of how nerve cells in the brain handle sensory impressions. It is designed to learn entirely from sensory input with no pre-defined knowledge database, so that its learning process will resemble that of a human child in early life.

"We've given it almost no pre-defined knowledge on purpose," Brandtsegg says.

Interdisciplinary project

The 'we' that Brandtsegg refers to is himself and postdoc Axel Tidemann – because this is without a doubt an interdisciplinary project. The machine is so complex that cooperation between different research fields is an absolute necessity to get it to work. Brandtsegg is at the Department of Music, while Tidemann is at the Department of Computer and Information Science. But they have overlapping interests.

"We understand just enough of each other's fields of study to see what is difficult, and why," Brandtsegg says. Naturally, his main interest is music.

But he is also an accomplished programmer, and uses this knowledge to make music. Conversely, Tidemann made a drumming for his doctoral project. The robot simulated the playing styles of living drummers.

Knows nothing

In the beginning, their robot knew nothing. It 'hears' sounds from a person speaking, and can connect these to a simultaneous video feed of the speaker.

The robot picks a sound that the person appears to be emphasizing, and responds by playing other sounds that it associates with this, while projecting a neural representation of its association between the sound and pictures. It doesn't show a video, but rather how its 'brain' connects sounds and images.


The robot has already been on display in Trondheim and Arendal, where visitors were able to affect its learning. It was in Trondheim for a month before Christmas, and in Arendal for two weeks in January.

Interacting with a diverse audience allowed the researchers to see exactly how it learns.

There was a lot of "My name is…" and "What is your name?" from the audience, but some people sang, and others read poems.

This resulted in a period where a lot of similar sounds and connected people got mixed up, a chaos of the machine making strange connections. But this changed the more it learned.

The robot gradually absorbed more and more impressions of different people. Certain people, like guides, affected it more, because it 'saw' them often. The robot also learned to filter input.

If a word is said in a certain way five times, and then in a different way once, it learned to filter away the standout and concentrate on the most common way, which is presumably correct. This processing happens during the robot's downtime.

"We say that the machine 'dreams' at night," Brandtsegg says.

After a while, the robot was able to connect words and pictures together in a more complex manner— you could say that it associates sounds with images and connects them by itself.


The robot is constantly under development, and Brandtsegg and Tidemann have lost a lot of sleep over it.

"The day before it was put on display in Trondheim, we worked through the night until eight in the morning. Then we went home, ate breakfast, and went back to work at 11," Brandtsegg says.

Between the two displays, they worked on improving the way the robot organizes its memories.

"Every little change we make takes a lot of time, at least if we want to make sure that we don't destroy any of the things it already has learned," Brandtsegg explains.

The result is a robot that shows how it makes associations in a very pedagogical manner. It doesn't resemble any living organisms on purpose— you're supposed to concentrate on its learning and the process behind it.

"The robot looks rough," as Brandtsegg says.

Thinking on its own?

[self.] is an art project, and raises questions that may be very relevant in the years to come. When is a robot thinking on its own? When is appropriate to call a machine 'living'?

"Many say that intelligence can be determined by specific behaviour," Tidemann says.

He names the Turing Test, in which a machine is considered to be thinking if it is able to convince a human that it is human as well, through text-based questions, at least thirty per cent of the time. Based on this definition, computers that play chess, like IBM's Deep Blue, can be defined as intelligent, because they are very good at chess.

But this is symbolic reasoning that isn't necessarily transferrable to real-world scenarios. Specialized robots for things like precise machining and industrial work have been better at certain tasks than humans for decades. But these robots are far from being able to learn. Not to mention doing things like running up stairs or jumping rope. There is also no machine that is as good at analysing a football match or writing a novel as a human.

Not in a vacuum

"Many artificial intelligence (AI) researchers, myself included, believe that true intelligence can't occur in a vacuum— it is a consequence of adapting and living in a dynamic environment," Tidemann explains. "You could see our intelligence as a byproduct of our adaptability."

"Because we have developed the ability to plan and remember, we get cognition as a sort of package deal," he says.

Cognition is the combined ability to perceive your surroundings, reason based on this perception, communicate with your surroundings and recall and act reasonably according to the information you have on hand.

"What is independent thinking? What is artificial life? These are the big questions," Tidemann says. "But we believe that the right way to reach for the 'holy grail' of AI is to implement biologically inspired models in a machine, let it operate in a physical environment and see if we can observe intelligent behaviour."

Researchers use the phrase 'technological singularity' to describe the point where human intellectual capacity is surpassed by . This is still a long time coming, however— Brandtsegg and Tidemann's goal with [self.] is that it will be able to learn through interacting with humans as well as possible.

Explore further: New 'deep learning' technique enables robot mastery of skills via trial and error

Related Stories

'Robobarista' can figure out your new coffee machine

April 15, 2015

In the near future we may have household robots to handle cooking, cleaning and other menial tasks. They will be teachable: Show the robot how to operate your coffee machine, and it will take over from there.

Musio: Your AI friend in back-and-forth exchange

June 2, 2015

Many people have hardly reached saturation point in being drawn to videos of cute robots that avoid the uncanny-valley risk of looking uncomfortably human; instead they cross the lines between cute animal pets and cartoonish ...

Kids and robots learn to write together (w/ Video)

March 4, 2015

Who is the teacher: the student or the machine? By showing a robot how to write letters, children improve their writing skills and gain self-‐confidence. This system, called CoWriter, was developed by EPFL researchers.

Evolving robot brains

March 2, 2015

Researchers are using the principles of Darwinian evolution to develop robot brains that can navigate mazes, identify and catch falling objects, and work as a group to determine in which order they should exit and re-enter ...

Recommended for you

Two new planets discovered using artificial intelligence

March 26, 2019

Astronomers at The University of Texas at Austin, in partnership with Google, have used artificial intelligence (AI) to uncover two more hidden planets in the Kepler space telescope archive. The technique shows promise for ...

Infertility's roots in DNA packaging

March 26, 2019

Pathological infertility is a condition affecting roughly 7 percent of human males, and among those afflicted, 10 to 15 percent are thought to have a genetic cause. However, pinpointing the precise genes responsible for the ...

Facebook is free, but should it count toward GDP anyway?

March 26, 2019

For several decades, gross domestic product (GDP), a sum of the value of purchased goods, has been a ubiquitous yardstick of economic activity. More recently, some observers have suggested that GDP falls short because it ...

Droughts could hit aging power plants hard

March 26, 2019

Older power plants with once-through cooling systems generate about a third of all U.S. electricity, but their future generating capacity will be undercut by droughts and rising water temperatures linked to climate change. ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.