Researchers train computer to create games by watching YouTube

Georgia Institute of Technology researchers have developed a computing system that views gameplay video from streaming services like YouTube or Twitch, analyzes the footage and then is able to create original new sections of a game.

The team tested their discovery, the first of its kind, with the original Super Mario Brothers, a well-known two-dimensional platformer game that will allow the new automatic-level designer to replicate results across similar games.

The system focuses on the gaming terrain (not the playable character) and the positioning between elements on-screen – be it pipes, blocks, coins or Goombas – and it determines the required relationship or level design rule. For example, pipes in the Mario games tend to stick out of the ground, so the system learns this and prevents any pipes from being flush with grassy surfaces. It also prevents "breaks" by using spatial analysis – e.g. no impossibly long jumps for the hero.

"An initial evaluation of our approach indicates an ability to produce level sections that are both playable and close to the original without hand coding any design criteria," says Matthew Guzdial, lead researcher and Ph.D. student in Computer Science at Georgia Tech.

Key to the process is watching the players in action to see where they actually spend most of their time in the game. After recording on-screen locations of sprites, Georgia Tech's algorithms determine what are high-interaction areas – those spots where players spend more time to collect bonus items or master a challenge. The automatic-level designer specifically targets these areas to gain design information. The system is then able to build a new level section, element by element.

"Our system creates a model or template, and it's able to produce level sections that have never been seen before, do not appear random and can be traversed by the player," says Mark Riedl, associate professor of Interactive Computing. "One could say that the system 'studies' the design of Mario levels until it is able to create new playable areas."

The Georgia Tech system output 151 distinct level sections from 17 samples in the original game, controlling for overall playability and style variables. Output increased to 334 level sections as the system lessened the constraints. The new levels can be played easily by porting them into a game engine.

Riedl says this is the first time he is aware of a gameplay video being used to design levels for a Mario game. By applying the technique across a number of different platformer games, a system can theoretically learn genre knowledge, which can be beneficial for procedurally creating games of a given genre. The technique may also extend to other game genres beyond platformers. The researchers next plan to develop full-scale levels and evaluate how gamers interact in those levels compared to the original gameplay videos.

The research, "Toward Game Level Generation from Gameplay Videos," is featured June 22-25 at the Foundations of Digital Games Conference in Pacific Grove, Calif.