Intelligent software assigns appropriate background music for pictures

Feb 17, 2012
Newly developed software called Picasso succeeds in arranging pictures with appropriate compositions in an instant, by utilizing the technical skills of movie directors. Credit: bellhäuser - das bilderwerk

Previously, setting a picture or whole series of pictures to suitable music required expert knowledge and a great deal of time. Newly developed software called Picasso succeeds in arranging pictures with appropriate compositions in an instant, by utilizing the technical skills of movie directors. Scientists at the Cluster of Excellence on "Multimodal Computing and Interaction" at Saarland University developed the software, and they will present the program with its associated smart phone app "PicasSound" at the Cebit computer expo. Cebit will take place from the 6th through the 10th of March at the fairground in Hanover, Germany.

"Usually, directors select consistent melodies for a particular movie scene," explains Sebastian Michel, head of a junior research group at the Cluster of Excellence at the Saarland University. He designed the Picasso software together with Ph.D. student Aleksander Stupar. The program utilizes the expertise of movie producers in assigning appropriate for pictures, enabling users to select music that will harmonize with their impressions of their last summer vacation without putting in an extraordinary amount of effort.

A three-level carries out the elaborate process. First, the picture the user has chosen is compared with a huge database of movie scenes and their corresponding soundtracks. The software creates a ranking of the scenes that look most similar to the user's picture. This database was compiled by the two researchers Michel and Stupar by splitting 50 movies into screenshots and their accompanying soundtracks. In the next step, the software creates a list of the selected tracks. And finally, the total number of tracks gets reduced to a few selections through a mathematical calculation; these are at last proposed to the user.

Here, a list of suggestions makes sense because particular pictures bring up different associations for different users, according to Michel. "Some people might connect a picture of a little house surrounded by an idyllic landscape with a romantic weekend for two, while others might think about loneliness," Michel explains.

Interested people can experience the software on the demo website and get an idea of Picasso's hit rate. Or they can use the free smartphone app "PicasSound," which is programmed to pick out an appropriate soundtrack using the music which is already saved on the smartphone. It is free available for the mobile operating systems iOSX and Android.

Next, Michel and Stupar will try to expand the system to support automatic sound recording of audiobooks on a textual basis. Furthermore, the researchers are planning to integrate a function inside the system that will take the individual preferences of the users into account.

Explore further: Tecnalia designs an app to help elderly people get around on public transport

More information: Online demo of the Picasso software, including videos, for free use: picasso.mmci.uni-saarland.de/demo/
Personal website of Sebastian Michel: qid3.mmci.uni-saarland.de/people/sebastian.html

Provided by Saarland University

3.3 /5 (7 votes)

Related Stories

Smartphone app helps you find friends in a crowd

Jun 23, 2011

Can a smartphone app enable meaningful, face-to-face conversation? Engineers are trying to find out, with software that helps people locate their friends in a crowd – and make new friends who share similar interests.

WalkSafe app shields smartphone pedestrians (w/ video)

Nov 28, 2011

(PhysOrg.com) -- Smartphone users who as pedestrians are not very smart about crossing and looking both ways now have a protective shield in the form of an Android app which they can download for free. A research ...

A tidy disk, a tidy mind

Feb 08, 2007

How to master the chaos in an endless sea of photographs? In the present age of digital photography, the sheer number of images creates a great need in the archiving of pictures. At this year’s CeBIT trade ...

World's first Walkman® phone

Mar 02, 2005

Sony Ericsson kick-started its entry into the mobile music market with the announcement of the W800, the first Walkman® branded mobile phone. It will now be possible to listen to music, handle phone calls ...

Recommended for you

Google worker shows early-draft glimpse of Chrome OS

Jul 20, 2014

The Chrome OS is in for a future look. Athena, a Chromium OS project, will bring forth the new Chrome OS user experience. Google's François Beaufort on Friday, referring to the screenshot he posted, said," ...

Google eyes Chrome on Windows laptop battery drain

Jul 19, 2014

Google Chrome on Microsoft Windows has been said to have a problem for some time but this week comes news that Google will give it the attention others think the problem quite deserves. Namely, Google is to ...

Mental-health monitoring goes mobile

Jul 16, 2014

Behavioral health analytics startup Ginger.io sees smartphones as "automated diaries" containing valuable insight into the mental well-being of people with mental illnesses.

User comments : 31

Adjust slider to filter visible comments by rank

Display comments: newest first

Lurker2358
5 / 5 (1) Feb 17, 2012
Soundtrack in a can.

Great, as if everything else wasn't artificial enough.

Would be more interesting to design a software to compose a unique musical piece for the video. Ha, that'd I'd find impressive, particularly if it output something half as awesome as a true masterpiece.
antialias_physorg
3 / 5 (2) Feb 17, 2012
You mean like this?
http://www.miller...ack2.mp3

(listen to the entire thing...not bad for a 'few lines of code')

Entirely composed by software (Emmy) created by David Cope
http://www.miller...er-8507/
Lurker2358
2.3 / 5 (3) Feb 17, 2012
yes, well, very good...good indeed, but not quite what I'm thinking.

I was initially just joking, and this Emmy is very, very far along.

I don't play music, so I was sort of just thinking about the best CUSTOM musical pieces from movies or video games, where the author composes a piece of music specifically for a character, or a scene, or the opening score of Star Wars, etc.

So the idea would be more complicated I guess.

I mean, a machine could never compose Terra's theme without understanding the logic of the character and then tie it to music.

A machine could never compose Dancing Mad without understanding the insanity of Kefka.

Now the star wars theme is just "epic," but doesn't tell a story by itself, so I suppose a machine could make something in that style just by following some rules.

But to compose a theme for a movie character or scene, or a video game character or scene would take an even more advanced machine to really capture the essence of the moment.
Lurker2358
5 / 5 (1) Feb 17, 2012
cont...

Like if you want to know what I'm talking about, research the soundtracks of Final Fantasy 6 characters and bosses.

Each is a masterpiece of music in it's own right, and would be awesome just as a stand-alone piece, but in addition, it somehow manages to capture the essence of the character or scene it represents.

"Terra"
youtube.com/watch?v=W7RPY-oiDAQ

"Catastrophe"
youtube.com/watch?v=58hev00AmPA

"Metamorphosis"
youtube.com/watch?v=gKUOyqZX1QU

"Battle to the Death"
youtube.com/watch?v=KH40gYZhwWI

"One Winged Angel"
youtube.com/watch?v=t7wJ8pE2qKU

This last one, much like Dancing Mad, is even done in the form of a twisted Hymn and has lyrics in Latin.

But the point here is that not only are all of these pieces great, they were specifically composed for and tell the story of the character, scene, or boss fight that they accompany.

All composed by Nobuo Uematsu.

If you listen to them, notice how each is carefully crafted for it's sole role in the story.
antialias_physorg
2.3 / 5 (3) Feb 17, 2012
I mean, a machine could never compose Terra's theme without understanding the logic of the character and then tie it to music.

Ever played "X Bexond the frontier" (1999). Since then situationally appropriate music that blends seamlessly has been pretty much standard in all strategy and 4X computer games (its so ubiquitous that we don't even register it anymore as novel)

But to compose a theme for a movie character or scene, or a video game character or scene would take an even more advanced machine to really capture the essence of the moment

That's just subjective. IMO Dancing Mad or Terra's theme are completely inappropriate to the characters mentioned.

And don't get me started on the FF series. I can't stand the soundtracks. Asians just have no idea of music - especially when using classical european instruments.
Lurker2358
5 / 5 (1) Feb 17, 2012

"Devil's Lab" (Magitek Research Facility)
youtube.com/watch?v=JAf087gh1c4

What makes this interesting is you are inside a research and production facility, and the music really makes it sound like there are things being made, and of course you see stuff moving around on the conveyor, etc.

*sigh* what these guys could do with modern computer technology...
Lurker2358
5 / 5 (1) Feb 17, 2012
Ever played "X Bexond the frontier" (1999).


No.

That's just subjective. IMO Dancing Mad or Terra's theme are completely inappropriate to the characters mentioned.

And don't get me started on the FF series. I can't stand the soundtracks. Asians just have no idea of music - especially when using classical european instruments.


Well, if it's subjective then...

when you search polls for best video game music of all time, FF6 themes come up top ten in almost all of them.

But the truth is, unfortunately, the games weren't as popular as they might have been. RPGs were never the most popular genre to begin with, and FF6 and Chrono Trigger each only sold a few million copies world wide. So there actually wasn't nearly as much exposure as there might have been.

A lot of people might vote for a Zelda game or a Mario game's soundtrack very high, but then again, they came with the console for free, so more people heard them...

I'm off topic now, but I think you see my point.
Lurker2358
5 / 5 (1) Feb 17, 2012
Ok, if you don't like the MIDI versions from the SNES, you can always listen to the orchestrated versions.

antialias_physorg
2.3 / 5 (3) Feb 17, 2012
FF6 themes come up top ten in almost all of them.

Asians making games for asians...who would have guessed that the game designers would try to make music that asians like?
kaasinees
1 / 5 (1) Feb 18, 2012
I would be more impressed if they made an algorithm that creates music based on a picture.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
FF6 themes come up top ten in almost all of them.

Asians making games for asians...who would have guessed that the game designers would try to make music that asians like?


@antialias
How racist of you. Have you forgotten all the Asian-Americans and Asians who have lived all their lives in Europe, Australia, and even Africa? How do you know exactly what each Asian in the world prefers in music? And, are you thinking of Asians only in terms of Chinese, Japanese or Korean? It appears that you are lumping all the Asians together as a collective in their tastes in music. There are many different types of what is called Asian. . .which is actually a geographic area of the Earth with different regions and differing cultures in those regions. Western music is not limited to Western civilizations and cultures. Elvis Presley was very popular in many Asian countries, as was the Beatles and other groups.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
You are probably assuming that game designers will only choose music that will fit the culture for which it's intended. But those designers know that their games can also be sold elsewhere in other cultures and markets and that an exclusively Chinese musical sound, e.g. will limit their sales.
There are many Asian musicians and composers who play and write music that is decidedly Western and it has nothing to do with their racial background. Yo Yo Ma is a good example.
antialias_physorg
3 / 5 (2) Feb 18, 2012
How racist of you.

Why? Europeans make soccer games for european markets. Americans make battle simulations. Asians make final fantasy type games for asian markets. It just makes economic sense to
a) make what you know/can empathize with
b) tailor to the market where your product will most likely make the most revenue

It's not about individuals in manufacture/marketing. It's about what pleases the MOST people of your target audience. Games (and music...and especially music in games) are no exception to this whatsoever.
Lurker2358
3 / 5 (2) Feb 18, 2012
Asians make final fantasy type games for asian markets. It just makes economic sense to


Ok, see, you must not have gotten the symbolism of FF6 and FF7, because it's clearly mid-eastern and western.

Kefka and Sephiroth are parodies of the Biblical Satan.

Dancing Mad and the final fight with Kefka's 3 tiered mini-boss, and then Kefka himself makes this painfully obvious.

In the case of Sephiroth, the symbols come directly from the throne of God in the Book of Revelation, as he is a hybrid of a Seraphim and "God" as seen on the throne. The halo and the golden rainbow above his head announces him as making claim to deity. The Song is "One Winged Angel," but Sephiroth actually has 7 wings, six white and one corrupted wing. Six is the number of man, coming short of God's perfection. Seven is the number of God's perfection, but Sephiroth is exposed as a fraud because his seventh wing, the "one wing" is corrupted.

It parodies Satan's insane, failed attempt to transcend God.
Lurker2358
5 / 5 (1) Feb 18, 2012
Now in Chrono Trigger, it's a bit of a different matter.

Lavos is effectively the dangerous, "What if God is an advanced, evil alien," sort of reasoning.

Depending on which characters you use at the end, and which side quests you've done, you find that Lavos is an incredibly ancient, yet super advanced planetary parasite. He seeds life on planets and comes back eons later to both "harvest" the DNA of the evolved creatures, in addition to draining the energy of the planet's core. In this way it improves and re-invents itself as it replicates.

I suppose the only symbol here is that Lavos is effectively the opposite of the Holy Trinity, but even that is a stretch.

But again, the music for the Lavos fight and the Magus fight "fits".

Am I arguing that no music would ever be better? no.

But then again, it would need to be something truly unbelievably exceptional before any enthusiast would say, "Yes, that should be the new theme in a remake."
Lurker2358
5 / 5 (1) Feb 18, 2012
If you think it's bad choices, fine. Then find a better alternative, or a machine or composer to custom make a better alternative.

In all cases, it is not a mockery of God, although there are many references to polytheism and other religions of all cultures.

But the common theme is "no finite or mortal being can ever become God, no matter how smart or powerful they may believe themselves to be."

In the end, they are always defeated by a group "mere" mortals.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
@antialias
That is a nice biography of David Cope and his Emmy and Emily Howell. Thanks for the link. I'm not a musicologist but I do enjoy listening to good music, whether classical or modern, and I can tell the difference between excellent music and junk.

"All the computer is is just an extension of me, Cope says. Theyre nothing but wonderfully organized shovels. I wouldnt give credit to the shovel for digging the hole. Would you?

Cope is honest as he perceives, and rightly so, that HE IS the originator of all that his computer algorithms are capable of doing. He is the god/programmer/creator of all that went in to Emmy and Emily Howell, for had it not been for David Cope and his many talents, the creation of his AI could not have occurred successfully in his time and place.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
Somehow, it reminds me of "Data". the android of the Star Trek: The Next Generation TV series and movies, although Data is a far cry from the simpler programming of Cope's musical composition programs. But the random decision-making according to Cope's input is a bit similar.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
IMHO, David Cope has spent far too much time and effort on AI rather than on his own efforts of composition that can be attributable to himself. He is a talented, but very frustrated composer. . .frustrated from his own lack of personal and meaningful music and the praise and adulation that would have come with it had he not become so upset with his, probably temporary, writer's block. However, he has proven, at least for the music industry, that AI is capable of producing very good music with the right input. But, he has also shown that there is nothing new under the sun as pertains to the mathematical computations of composing a fine piece of music. It has all been done before within so many variations by humans. . .and with far more soul. Only the rhythms can be original, but even those rhythms may have been done already in some remote culture.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
How racist of you.

Why? Europeans make soccer games for european markets. Americans make battle simulations. Asians make final fantasy type games for asian markets. It just makes economic sense to
a) make what you know/can empathize with
b) tailor to the market where your product will most likely make the most revenue

It's not about individuals in manufacture/marketing. It's about what pleases the MOST people of your target audience. Games (and music...and especially music in games) are no exception to this whatsoever.


RitchieGuy
2.3 / 5 (3) Feb 18, 2012
The only barrier to selling an excellent game worldwide is the language barrier. But that can be overcome by using subtitles or voice-over in different languages. A game designer who intentionally designs only for the Chinese-Asian market will find that Asia not only is comprised of Chinese that includes Mandarin and many other dialects, but also Thai, Korean, Indonesian, etc., not to mention the English speakers. The musical background also should be acceptable to various cultures and languages, unless the game designer has no intention of selling the game worldwide. If that is the intent, then the designer and his staff will lose out on that extra revenue even if their game is embraced by the intended market. Not everyone would appreciate that singsong quality and the sound of pounding of coconut shells in Chinese music. There would be a lack of diversity in such a game for all to understand.

antialias_physorg
5 / 5 (1) Feb 18, 2012
The only barrier to selling an excellent game worldwide is the language barrier.

not quite. There's a culturale barrier. Have you looked at the types of games they market in Japan? And I don't mena the kinky kind but the completely 'normal' games for kids and adolescents. This stuff is so bizarre to our eyes that you'd never be able to sell it here.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
Wellll. . .that is the reason for the ratings on the game box. . .E for everyone, T for teen, etc. That's for the American market and possibly European. I can't imagine sane parents buying such violent games for their kids, knowing the content of the games. . . .unless they are NOT aware of the content. Does that make them bad parents? Not necessarily. . .just ignorant and maybe they buy it for themselves in Japan and then allow the kids to play the game, not realizing how bad it is. I don't know any Japanese, so I can't determine their thinking processes. They can't be that much different from us. But kids almost always go for the advertising and then beg mom and dad to buy the game. If the parents don't have the wherewithal to take the time to search the packaging for content details, then they may or may not care if their kids turn out to be monsters. I've bought American designed games for myself that I wouldn't allow my own kids to use. But that's me and my own upbringing.
antialias_physorg
5 / 5 (2) Feb 18, 2012
I'm not talking violent (or pronographic). I'm talking: weird

hile the following link might encompass the games tha take the cake I think you'll get the idea.
http://www.weirdw...m-japan/

And these are games that are SUCCESSFUL there. Can you imagine any of these being successful anywhere else? Even with a perfect translation?
Lurker2358
5 / 5 (1) Feb 18, 2012
WTF>

Kid Icarus is a CLASSIC.

One of the first games with a password save feature, along with Metroid. Use all spaces followed by "Justin Bailey" for shortcut password here to get everything, in case you forgot your password.

I still remember the password: "Icarus Fights Medusa Angels" (all caps). this'll get ya to like the last level with almost everything for free.

The first level of Kid Icarus really is very, very hard. I'd say it's one of the hardest levels in gaming that isn't "broken," and I don't even remember much about it.

I'd agree the others on that list are very weird, and don't look particularly interesting.

Mario spin-offs and sequels got old very long time ago, and some of these others are just silly "clones" of other games.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
I can't seem to access that page. I'll try again later. The thing is, what appears to be weird to one generation, may be perfectly normal to the younger generation, and every game you've ever played as a young person may be not considered much of a challenge by your own kids. What may seem horrific to your own eyes may have an enticing effect on theirs. I think that these games that depict the battles between evil and good where good is supposed to win and most often does, could be at least a little uplifting psychologically, no matter how weird or violent they may be. I have yet to see a game where both sides are evil and, no matter who wins, the end is still bad.
But, you've got to wonder about the designers of these weird games and the story boards. Where do they get such nightmarish ideas and what kind of dreams do they have. Outwardly, they may seem as normal as you and I (not too sure about you. . .LOL) :)) but do they come up with such weirdness for the sake of their art?
Lurker2358
3 / 5 (2) Feb 18, 2012
and every game you've ever played as a young person may be not considered much of a challenge by your own kids


It's the opposite.

I don't have kids of my own, but I know teens and kids and tweens who cannot play the old games I played, or the more difficult RPGs (FF ones are actually easy,) because their skills have been dumbed down by modern games. Now if they had only that to work with....

also, "Execution based" games, such as platform, adventure, and fighting games, and even FPS have generally gotten far, far easier over the years. There are a few exceptions, like the A.I. in Crysis is extremely good compared to older FPS, but that's hard to really judge, because I'm getting too old to play a FPS at a decent skill level.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
I remember that during the 1970s there were artists who did sci-fi posters that were considered weird at the time. Posters of strange and beautiful female humanoids with wings or handsome men riding on eagle-like birds and strange plant life. I collected them at the time because the art was hauntingly beautiful. I never thought about the mindset of the artists, just that the art was incredibly weird and wonderful for that time. What made it different was the fact that no violence was depicted at all, except for one poster that had a monster plant's tendrils wrapped around a beautiful "girl's" very shapely leg. Some say that those artists were tripping out while painting. . .maybe a little cannabis or acid, who knows? Anyway, those posters, though considered by some as weird, were to my liking and I have a nice collection. Wish I could find more.
Lurker2358
3 / 5 (2) Feb 18, 2012
I have yet to see a game where both sides are evil and, no matter who wins, the end is still bad


I actually don't recommend the game, because it's so dark and sadistic, but in Baldur's Gate 2 you can play a bad guy who really is evil. the ending isn't much different, but you can go around town picking fights and seeing how many factions your character can single handedly defeat in a fight. Like kill someone in town, and 4 Cowled Wizards will attack you. If your reputation is bad enough, an additional platoon of the Knights of the Sacred Heart will attack you simultaneously. It's supposed to be an "impossible" fight, but I've won it on several occasions.

anyway, that one gives you a chance to play the bad guy.

Neverwinter nights 2: Mask of the Betrayer lets you play a bad guy too, but again, ending is only slightly different.

I found this expansion too easy as did hardcore D&D players, but most other players complained it was far too hard.
Lurker2358
3 / 5 (2) Feb 18, 2012
Actually, in Mask of the Betrayer, your character IS evil.

You can just choose from basic options though. You can be the "Bad guy who tries to be good" or you can be the "bad guy who embraces his curse", and it changes dialogues and bonus feats that become available throughout the game, as well as minor changes to the endings, as there are multiple endings.

It can be really annoying though, so unless you've absolutely mastered character building in the original campaign, it's actually hard due to some mechanics they force on the protagonist.

Even with a perfect character build, you will be forced to absolutely fly through the game, keeping a high pace in all levels and encounters, else you will "starve" and die.
RitchieGuy
2.3 / 5 (3) Feb 18, 2012
Lurker. . .sorry but I'm not into the new games. Last one I played was a deer hunting game that was boring as hell. I used to enjoy the Jedi Knight games that I played on my old WinXP even though the graphics was awful. My kids had some terrific games, but I was too busy to play and my computer at the time had an older OS that wouldn't accept the newer games.