Intelligent software assigns appropriate background music for pictures

February 17, 2012

Cebit 2012: Intelligent software assigns appropriate background music for pictures

Enlarge

Newly developed software called Picasso succeeds in arranging pictures with appropriate compositions in an instant, by utilizing the technical skills of movie directors. Credit: bellhäuser - das bilderwerk

Previously, setting a picture or whole series of pictures to suitable music required expert knowledge and a great deal of time. Newly developed software called Picasso succeeds in arranging pictures with appropriate compositions in an instant, by utilizing the technical skills of movie directors. Scientists at the Cluster of Excellence on "Multimodal Computing and Interaction" at Saarland University developed the software, and they will present the program with its associated smart phone app "PicasSound" at the Cebit computer expo. Cebit will take place from the 6th through the 10th of March at the fairground in Hanover, Germany.

"Usually, directors select consistent melodies for a particular movie scene," explains Sebastian Michel, head of a junior research group at the Cluster of Excellence at the Saarland University. He designed the Picasso software together with Ph.D. student Aleksander Stupar. The program utilizes the expertise of movie producers in assigning appropriate for pictures, enabling users to select music that will harmonize with their impressions of their last summer vacation without putting in an extraordinary amount of effort.

A three-level carries out the elaborate process. First, the picture the user has chosen is compared with a huge database of movie scenes and their corresponding soundtracks. The software creates a ranking of the scenes that look most similar to the user's picture. This database was compiled by the two researchers Michel and Stupar by splitting 50 movies into screenshots and their accompanying soundtracks. In the next step, the software creates a list of the selected tracks. And finally, the total number of tracks gets reduced to a few selections through a mathematical calculation; these are at last proposed to the user.

Here, a list of suggestions makes sense because particular pictures bring up different associations for different users, according to Michel. "Some people might connect a picture of a little house surrounded by an idyllic landscape with a romantic weekend for two, while others might think about loneliness," Michel explains.

Interested people can experience the software on the demo website and get an idea of Picasso's hit rate. Or they can use the free smartphone app "PicasSound," which is programmed to pick out an appropriate soundtrack using the music which is already saved on the smartphone. It is free available for the mobile operating systems iOSX and Android.

Next, Michel and Stupar will try to expand the system to support automatic sound recording of audiobooks on a textual basis. Furthermore, the researchers are planning to integrate a function inside the system that will take the individual preferences of the users into account.

More information: Online demo of the Picasso software, including videos, for free use: http://picasso.mmc … and.de/demo/
Personal website of Sebastian Michel: http://qid3.mmci.u … bastian.html

Provided by Saarland University

view popular send feedback to editors
3.3 /5 (7 votes)  

Filter


Move the slider to adjust rank threshold, so that you can hide some of the comments.


Display comments: newest first

Lurker2358
Feb 17, 2012

Rank: 5 / 5 (1)
Soundtrack in a can.

Great, as if everything else wasn't artificial enough.

Would be more interesting to design a software to compose a unique musical piece for the video. Ha, that'd I'd find impressive, particularly if it output something half as awesome as a true masterpiece.
antialias_physorg
Feb 17, 2012

Rank: 3 / 5 (2)
You mean like this?
http://www.miller...ack2.mp3

(listen to the entire thing...not bad for a 'few lines of code')

Entirely composed by software (Emmy) created by David Cope
http://www.miller...er-8507/
Lurker2358
Feb 17, 2012

Rank: 2.3 / 5 (3)
yes, well, very good...good indeed, but not quite what I'm thinking.

I was initially just joking, and this Emmy is very, very far along.

I don't play music, so I was sort of just thinking about the best CUSTOM musical pieces from movies or video games, where the author composes a piece of music specifically for a character, or a scene, or the opening score of Star Wars, etc.

So the idea would be more complicated I guess.

I mean, a machine could never compose Terra's theme without understanding the logic of the character and then tie it to music.

A machine could never compose Dancing Mad without understanding the insanity of Kefka.

Now the star wars theme is just "epic," but doesn't tell a story by itself, so I suppose a machine could make something in that style just by following some rules.

But to compose a theme for a movie character or scene, or a video game character or scene would take an even more advanced machine to really capture the essence of the moment.
Lurker2358
Feb 17, 2012

Rank: 5 / 5 (1)
cont...

Like if you want to know what I'm talking about, research the soundtracks of Final Fantasy 6 characters and bosses.

Each is a masterpiece of music in it's own right, and would be awesome just as a stand-alone piece, but in addition, it somehow manages to capture the essence of the character or scene it represents.

"Terra"
youtube.com/watch?v=W7RPY-oiDAQ

"Catastrophe"
youtube.com/watch?v=58hev00AmPA

"Metamorphosis"
youtube.com/watch?v=gKUOyqZX1QU

"Battle to the Death"
youtube.com/watch?v=KH40gYZhwWI

"One Winged Angel"
youtube.com/watch?v=t7wJ8pE2qKU

This last one, much like Dancing Mad, is even done in the form of a twisted Hymn and has lyrics in Latin.

But the point here is that not only are all of these pieces great, they were specifically composed for and tell the story of the character, scene, or boss fight that they accompany.

All composed by Nobuo Uematsu.

If you listen to them, notice how each is carefully crafted for it's sole role in the story.
antialias_physorg
Feb 17, 2012

Rank: 2.3 / 5 (3)
I mean, a machine could never compose Terra's theme without understanding the logic of the character and then tie it to music.

Ever played "X Bexond the frontier" (1999). Since then situationally appropriate music that blends seamlessly has been pretty much standard in all strategy and 4X computer games (its so ubiquitous that we don't even register it anymore as novel)

But to compose a theme for a movie character or scene, or a video game character or scene would take an even more advanced machine to really capture the essence of the moment

That's just subjective. IMO Dancing Mad or Terra's theme are completely inappropriate to the characters mentioned.

And don't get me started on the FF series. I can't stand the soundtracks. Asians just have no idea of music - especially when using classical european instruments.
Lurker2358
Feb 17, 2012

Rank: 5 / 5 (1)

"Devil's Lab" (Magitek Research Facility)
youtube.com/watch?v=JAf087gh1c4

What makes this interesting is you are inside a research and production facility, and the music really makes it sound like there are things being made, and of course you see stuff moving around on the conveyor, etc.

*sigh* what these guys could do with modern computer technology...
Lurker2358
Feb 17, 2012

Rank: 5 / 5 (1)
Ever played "X Bexond the frontier" (1999).


No.

That's just subjective. IMO Dancing Mad or Terra's theme are completely inappropriate to the characters mentioned.

And don't get me started on the FF series. I can't stand the soundtracks. Asians just have no idea of music - especially when using classical european instruments.


Well, if it's subjective then...

when you search polls for best video game music of all time, FF6 themes come up top ten in almost all of them.

But the truth is, unfortunately, the games weren't as popular as they might have been. RPGs were never the most popular genre to begin with, and FF6 and Chrono Trigger each only sold a few million copies world wide. So there actually wasn't nearly as much exposure as there might have been.

A lot of people might vote for a Zelda game or a Mario game's soundtrack very high, but then again, they came with the console for free, so more people heard them...

I'm off topic now, but I think you see my point.
Lurker2358
Feb 17, 2012

Rank: 5 / 5 (1)
Ok, if you don't like the MIDI versions from the SNES, you can always listen to the orchestrated versions.

antialias_physorg
Feb 17, 2012

Rank: 2.3 / 5 (3)
FF6 themes come up top ten in almost all of them.

Asians making games for asians...who would have guessed that the game designers would try to make music that asians like?
kaasinees
Feb 18, 2012

Rank: 1 / 5 (1)
I would be more impressed if they made an algorithm that creates music based on a picture.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
FF6 themes come up top ten in almost all of them.

Asians making games for asians...who would have guessed that the game designers would try to make music that asians like?


@antialias
How racist of you. Have you forgotten all the Asian-Americans and Asians who have lived all their lives in Europe, Australia, and even Africa? How do you know exactly what each Asian in the world prefers in music? And, are you thinking of Asians only in terms of Chinese, Japanese or Korean? It appears that you are lumping all the Asians together as a collective in their tastes in music. There are many different types of what is called Asian. . .which is actually a geographic area of the Earth with different regions and differing cultures in those regions. Western music is not limited to Western civilizations and cultures. Elvis Presley was very popular in many Asian countries, as was the Beatles and other groups.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
You are probably assuming that game designers will only choose music that will fit the culture for which it's intended. But those designers know that their games can also be sold elsewhere in other cultures and markets and that an exclusively Chinese musical sound, e.g. will limit their sales.
There are many Asian musicians and composers who play and write music that is decidedly Western and it has nothing to do with their racial background. Yo Yo Ma is a good example.
antialias_physorg
Feb 18, 2012

Rank: 3 / 5 (2)
How racist of you.

Why? Europeans make soccer games for european markets. Americans make battle simulations. Asians make final fantasy type games for asian markets. It just makes economic sense to
a) make what you know/can empathize with
b) tailor to the market where your product will most likely make the most revenue

It's not about individuals in manufacture/marketing. It's about what pleases the MOST people of your target audience. Games (and music...and especially music in games) are no exception to this whatsoever.
Lurker2358
Feb 18, 2012

Rank: 3 / 5 (2)
Asians make final fantasy type games for asian markets. It just makes economic sense to


Ok, see, you must not have gotten the symbolism of FF6 and FF7, because it's clearly mid-eastern and western.

Kefka and Sephiroth are parodies of the Biblical Satan.

Dancing Mad and the final fight with Kefka's 3 tiered mini-boss, and then Kefka himself makes this painfully obvious.

In the case of Sephiroth, the symbols come directly from the throne of God in the Book of Revelation, as he is a hybrid of a Seraphim and "God" as seen on the throne. The halo and the golden rainbow above his head announces him as making claim to deity. The Song is "One Winged Angel," but Sephiroth actually has 7 wings, six white and one corrupted wing. Six is the number of man, coming short of God's perfection. Seven is the number of God's perfection, but Sephiroth is exposed as a fraud because his seventh wing, the "one wing" is corrupted.

It parodies Satan's insane, failed attempt to transcend God.
Lurker2358
Feb 18, 2012

Rank: 5 / 5 (1)
Now in Chrono Trigger, it's a bit of a different matter.

Lavos is effectively the dangerous, "What if God is an advanced, evil alien," sort of reasoning.

Depending on which characters you use at the end, and which side quests you've done, you find that Lavos is an incredibly ancient, yet super advanced planetary parasite. He seeds life on planets and comes back eons later to both "harvest" the DNA of the evolved creatures, in addition to draining the energy of the planet's core. In this way it improves and re-invents itself as it replicates.

I suppose the only symbol here is that Lavos is effectively the opposite of the Holy Trinity, but even that is a stretch.

But again, the music for the Lavos fight and the Magus fight "fits".

Am I arguing that no music would ever be better? no.

But then again, it would need to be something truly unbelievably exceptional before any enthusiast would say, "Yes, that should be the new theme in a remake."
Lurker2358
Feb 18, 2012

Rank: 5 / 5 (1)
If you think it's bad choices, fine. Then find a better alternative, or a machine or composer to custom make a better alternative.

In all cases, it is not a mockery of God, although there are many references to polytheism and other religions of all cultures.

But the common theme is "no finite or mortal being can ever become God, no matter how smart or powerful they may believe themselves to be."

In the end, they are always defeated by a group "mere" mortals.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
@antialias
That is a nice biography of David Cope and his Emmy and Emily Howell. Thanks for the link. I'm not a musicologist but I do enjoy listening to good music, whether classical or modern, and I can tell the difference between excellent music and junk.

"All the computer is is just an extension of me, Cope says. Theyre nothing but wonderfully organized shovels. I wouldnt give credit to the shovel for digging the hole. Would you?

Cope is honest as he perceives, and rightly so, that HE IS the originator of all that his computer algorithms are capable of doing. He is the god/programmer/creator of all that went in to Emmy and Emily Howell, for had it not been for David Cope and his many talents, the creation of his AI could not have occurred successfully in his time and place.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
Somehow, it reminds me of "Data". the android of the Star Trek: The Next Generation TV series and movies, although Data is a far cry from the simpler programming of Cope's musical composition programs. But the random decision-making according to Cope's input is a bit similar.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
IMHO, David Cope has spent far too much time and effort on AI rather than on his own efforts of composition that can be attributable to himself. He is a talented, but very frustrated composer. . .frustrated from his own lack of personal and meaningful music and the praise and adulation that would have come with it had he not become so upset with his, probably temporary, writer's block. However, he has proven, at least for the music industry, that AI is capable of producing very good music with the right input. But, he has also shown that there is nothing new under the sun as pertains to the mathematical computations of composing a fine piece of music. It has all been done before within so many variations by humans. . .and with far more soul. Only the rhythms can be original, but even those rhythms may have been done already in some remote culture.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
How racist of you.

Why? Europeans make soccer games for european markets. Americans make battle simulations. Asians make final fantasy type games for asian markets. It just makes economic sense to
a) make what you know/can empathize with
b) tailor to the market where your product will most likely make the most revenue

It's not about individuals in manufacture/marketing. It's about what pleases the MOST people of your target audience. Games (and music...and especially music in games) are no exception to this whatsoever.


RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
The only barrier to selling an excellent game worldwide is the language barrier. But that can be overcome by using subtitles or voice-over in different languages. A game designer who intentionally designs only for the Chinese-Asian market will find that Asia not only is comprised of Chinese that includes Mandarin and many other dialects, but also Thai, Korean, Indonesian, etc., not to mention the English speakers. The musical background also should be acceptable to various cultures and languages, unless the game designer has no intention of selling the game worldwide. If that is the intent, then the designer and his staff will lose out on that extra revenue even if their game is embraced by the intended market. Not everyone would appreciate that singsong quality and the sound of pounding of coconut shells in Chinese music. There would be a lack of diversity in such a game for all to understand.

antialias_physorg
Feb 18, 2012

Rank: 5 / 5 (1)
The only barrier to selling an excellent game worldwide is the language barrier.

not quite. There's a culturale barrier. Have you looked at the types of games they market in Japan? And I don't mena the kinky kind but the completely 'normal' games for kids and adolescents. This stuff is so bizarre to our eyes that you'd never be able to sell it here.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
Wellll. . .that is the reason for the ratings on the game box. . .E for everyone, T for teen, etc. That's for the American market and possibly European. I can't imagine sane parents buying such violent games for their kids, knowing the content of the games. . . .unless they are NOT aware of the content. Does that make them bad parents? Not necessarily. . .just ignorant and maybe they buy it for themselves in Japan and then allow the kids to play the game, not realizing how bad it is. I don't know any Japanese, so I can't determine their thinking processes. They can't be that much different from us. But kids almost always go for the advertising and then beg mom and dad to buy the game. If the parents don't have the wherewithal to take the time to search the packaging for content details, then they may or may not care if their kids turn out to be monsters. I've bought American designed games for myself that I wouldn't allow my own kids to use. But that's me and my own upbringing.
antialias_physorg
Feb 18, 2012

Rank: 5 / 5 (2)
I'm not talking violent (or pronographic). I'm talking: weird

hile the following link might encompass the games tha take the cake I think you'll get the idea.
http://www.weirdw...m-japan/

And these are games that are SUCCESSFUL there. Can you imagine any of these being successful anywhere else? Even with a perfect translation?
Lurker2358
Feb 18, 2012

Rank: 5 / 5 (1)
WTF>

Kid Icarus is a CLASSIC.

One of the first games with a password save feature, along with Metroid. Use all spaces followed by "Justin Bailey" for shortcut password here to get everything, in case you forgot your password.

I still remember the password: "Icarus Fights Medusa Angels" (all caps). this'll get ya to like the last level with almost everything for free.

The first level of Kid Icarus really is very, very hard. I'd say it's one of the hardest levels in gaming that isn't "broken," and I don't even remember much about it.

I'd agree the others on that list are very weird, and don't look particularly interesting.

Mario spin-offs and sequels got old very long time ago, and some of these others are just silly "clones" of other games.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
I can't seem to access that page. I'll try again later. The thing is, what appears to be weird to one generation, may be perfectly normal to the younger generation, and every game you've ever played as a young person may be not considered much of a challenge by your own kids. What may seem horrific to your own eyes may have an enticing effect on theirs. I think that these games that depict the battles between evil and good where good is supposed to win and most often does, could be at least a little uplifting psychologically, no matter how weird or violent they may be. I have yet to see a game where both sides are evil and, no matter who wins, the end is still bad.
But, you've got to wonder about the designers of these weird games and the story boards. Where do they get such nightmarish ideas and what kind of dreams do they have. Outwardly, they may seem as normal as you and I (not too sure about you. . .LOL) :)) but do they come up with such weirdness for the sake of their art?
Lurker2358
Feb 18, 2012

Rank: 3 / 5 (2)
and every game you've ever played as a young person may be not considered much of a challenge by your own kids


It's the opposite.

I don't have kids of my own, but I know teens and kids and tweens who cannot play the old games I played, or the more difficult RPGs (FF ones are actually easy,) because their skills have been dumbed down by modern games. Now if they had only that to work with....

also, "Execution based" games, such as platform, adventure, and fighting games, and even FPS have generally gotten far, far easier over the years. There are a few exceptions, like the A.I. in Crysis is extremely good compared to older FPS, but that's hard to really judge, because I'm getting too old to play a FPS at a decent skill level.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
I remember that during the 1970s there were artists who did sci-fi posters that were considered weird at the time. Posters of strange and beautiful female humanoids with wings or handsome men riding on eagle-like birds and strange plant life. I collected them at the time because the art was hauntingly beautiful. I never thought about the mindset of the artists, just that the art was incredibly weird and wonderful for that time. What made it different was the fact that no violence was depicted at all, except for one poster that had a monster plant's tendrils wrapped around a beautiful "girl's" very shapely leg. Some say that those artists were tripping out while painting. . .maybe a little cannabis or acid, who knows? Anyway, those posters, though considered by some as weird, were to my liking and I have a nice collection. Wish I could find more.
Lurker2358
Feb 18, 2012

Rank: 3 / 5 (2)
I have yet to see a game where both sides are evil and, no matter who wins, the end is still bad


I actually don't recommend the game, because it's so dark and sadistic, but in Baldur's Gate 2 you can play a bad guy who really is evil. the ending isn't much different, but you can go around town picking fights and seeing how many factions your character can single handedly defeat in a fight. Like kill someone in town, and 4 Cowled Wizards will attack you. If your reputation is bad enough, an additional platoon of the Knights of the Sacred Heart will attack you simultaneously. It's supposed to be an "impossible" fight, but I've won it on several occasions.

anyway, that one gives you a chance to play the bad guy.

Neverwinter nights 2: Mask of the Betrayer lets you play a bad guy too, but again, ending is only slightly different.

I found this expansion too easy as did hardcore D&D players, but most other players complained it was far too hard.
Lurker2358
Feb 18, 2012

Rank: 3 / 5 (2)
Actually, in Mask of the Betrayer, your character IS evil.

You can just choose from basic options though. You can be the "Bad guy who tries to be good" or you can be the "bad guy who embraces his curse", and it changes dialogues and bonus feats that become available throughout the game, as well as minor changes to the endings, as there are multiple endings.

It can be really annoying though, so unless you've absolutely mastered character building in the original campaign, it's actually hard due to some mechanics they force on the protagonist.

Even with a perfect character build, you will be forced to absolutely fly through the game, keeping a high pace in all levels and encounters, else you will "starve" and die.
RitchieGuy
Feb 18, 2012

Rank: 2.3 / 5 (3)
Lurker. . .sorry but I'm not into the new games. Last one I played was a deer hunting game that was boring as hell. I used to enjoy the Jedi Knight games that I played on my old WinXP even though the graphics was awful. My kids had some terrific games, but I was too busy to play and my computer at the time had an older OS that wouldn't accept the newer games.
Rank 3.3 /5 (7 votes)
more news
Related Stories
created Jun 23, 2011 comments 0

Smartphone app helps you find friends in a crowd

created Nov 28, 2011 comments 0

WalkSafe app shields smartphone pedestrians (w/ video)

created Apr 13, 2011 comments 0

Researchers create privacy mode to help secure Android smartphones

created Feb 08, 2007 comments 0

A tidy disk, a tidy mind

Tags

mathematical calculation, smart phones, cebit, background music, algorithm, software
Relevant PhysicsForums posts

More news stories

Browser wars flare in mobile space

The browser wars are heating up again, but this time the fight is for dominance of the mobile Internet.

Technology / Software

created 20 hours ago | popularity 4 / 5 (4) | comments 3

Probability of contamination from severe nuclear reactor accidents is higher than expected: study

Catastrophic nuclear accidents such as the core meltdowns in Chernobyl and Fukushima are more likely to happen than previously assumed. Based on the operating hours of all civil nuclear reactors and the number ...

Technology / Energy & Green Tech

created May 22, 2012 | popularity 3.6 / 5 (25) | comments 56 | with audio podcast

HyperSolar shows dirty water no barrier to power world

(Phys.org) -- The Santa Barbara, California, company, HyperSolar, is set to transparently share the ups and downs of its research experiences toward the company’s ultimate vision, successfully producing ...

Technology / Energy & Green Tech

created May 24, 2012 | popularity 4.8 / 5 (16) | comments 17 | with audio podcast report

Tesla to launch electric sedan in US on June 22

Tesla Motors said Tuesday it would begin deliveries of "the world's first premium electric sedan" on June 22, slightly ahead of schedule.

Technology / Energy & Green Tech

created May 22, 2012 | popularity 4.5 / 5 (12) | comments 18

SpotterRF debuts Radar Backpack Kit (w/ Video)

(Phys.org) -- SpotterRF has announced a special radar backpack kit designed to enhance situational awareness for soldiers on the ground. The company says its special radar is designed for warfighters as part ...

Technology / Hi Tech & Innovation

created May 26, 2012 | popularity 5 / 5 (5) | comments 13 | with audio podcast report


Stunning image of smallest possible five-ringed structure

Scientists have created and imaged the smallest possible five-ringed structure – about 100,000 times thinner than a human hair – and you'll probably recognise its shape.

'Unzipped' carbon nanotubes could help energize fuel cells, batteries

Multi-walled carbon nanotubes riddled with defects and impurities on the outside could replace some of the expensive platinum catalysts used in fuel cells and metal-air batteries, according to scientists at ...

Change in developmental timing was crucial in the evolutionary shift from dinosaurs to birds: study

At first glance, it's hard to see how a common house sparrow and a Tyrannosaurus Rex might have anything in common. After all, one is a bird that weighs less than an ounce, and the other is a dinosaur that ...

Computer model used to pinpoint prime materials for efficient carbon capture

When power plants begin capturing their carbon emissions to reduce greenhouse gases – and to most in the electric power industry, it's a question of when, not if – it will be an expensive undertaking.

T cells 'hunt' parasites like animal predators seek prey, study shows

By pairing an intimate knowledge of immune-system function with a deep understanding of statistical physics, a cross-disciplinary team at the University of Pennsylvania has arrived at a surprising finding: T cells use a movement ...

Yale study concludes public apathy over climate change unrelated to science literacy

Are members of the public divided about climate change because they don't understand the science behind it? If Americans knew more basic science and were more proficient in technical reasoning, would public consensus match ...