What if all software was open source? A code to unlock the desktop

Mar 30, 2010 by Hannah Hickey

(PhysOrg.com) -- What if all software was open source? Anybody would then be able to add custom features to Microsoft Word, Adobe Photoshop, Apple iTunes or any other program. A University of Washington project may make this possible.

"Microsoft and Apple aren't going to open up all their stuff. But they all create programs that put pixels on the screen. And if we can modify those pixels, then we can change the program's apparent behavior," said James Fogarty, a UW assistant professor of computer science and engineering.

His approach hijacks the display to customize the user's interaction with the program. He will demonstrate his system April 14 in Atlanta at the Association for Computing Machinery's Conference on Human Factors in .

"We really see this as a first step toward a scenario where anybody can modify any application," Fogarty said. "In a sense, this has happened online. You've got this mash-up culture on the Web because everybody can see the HTML. But that hasn't been possible on the desktop."

These days a Web page might include a map from , an embedded video from and a list of recent headlines. This is not yet possible on the personal computer.

"Let's say I'm writing a paper in Microsoft Word but I want to listen to music at the same time," explained co-author Morgan Dixon, a UW doctoral student in computer science and engineering.

Right now he would have to click back and for the between Word and iTunes, but the system he helped create can simply add a few iTunes buttons to the Word toolbar.

"I'm using some program that I love," Dixon said, "and I'm going to stick in some features from some other program that I love, so I have a more unified interface."

More importantly, having more control over widely used programs would allow people to benefit from accessibility tools that have been gathering dust in academic research labs.

An example is target-aware pointing, which can make many interfaces easier for people with muscular dystrophy, Parkinson's disease, cerebral palsy or other motor-control disabilities. One such tool, the bubble cursor, highlights the button closest to it, making it easier for people with disabilities to click a button without having to hit it dead on. Fogarty and Dixon show the first implementation of a bubble cursor in various commercial applications.

"The human-computer interaction community has done 30 years of research on how to make computers more accessible to people with disabilities. But no one change is perfect for everybody," Fogarty said. "That's why you don't see these tools out there."

His research allows people to personalize programs based on their needs.

The UW tool, named Prefab, takes advantage of the fact that almost all displays are made from prefabricated blocks of code such as buttons, sliders, check boxes and drop-down menus. Prefab looks for those blocks as many as 20 times per second and alters their behavior.

The researchers are continuing to develop Prefab and are exploring options for commercialization.

Prefab unlocks previously inaccessible interfaces, allowing people to add the same usability tool to all the applications they run on their desktop. The system could translate a program's interface into a different language, or reorder menus to bump up favorite commands.

The authors hope Prefab will spur development of new innovations.

"If you come up with a new technology, too often it's evaluated in a test environment," Fogarty said. "This lets researchers put it into practice in something real, like Photoshop or iTunes."

Prefab can also produce more advanced effects. One demonstration that will be presented at the conference creates multiple previews of a single image in Photoshop. Behind the scenes, Prefab moves the sliders to different points, captures the output and then displays all of them on a single screen. This could save time by showing a range of effects the user frequently adjusts.

The system could also allow programs to move from computer screens to mobile devices, which do not have a standard operating system.

"It dramatically lowers the threshold to getting new innovation into existing, complex programs," Fogarty said.

Research has been funded by the Hacherl Endowed Graduate Fellowship in the UW Department of & Engineering, a fellowship from the Seattle chapter of the Achievement Rewards for College Scientists, and Intel.

Explore further: Taking great ideas from the lab to the fab

More information: -- More information about Prefab: www.cs.washington.edu/homes/jf… rty/research/prefab/
-- "Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure" To appear in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2010). Winner of Best Paper award. uwnews.org/relatedcontent/2010… 6581_thisID56586.pdf

Related Stories

Picture-driven computing (w/ Video)

Jan 20, 2010

Until the 1980s, using a computer program meant memorizing a lot of commands and typing them in a line at a time, only to get lines of text back. The graphical user interface, or GUI, changed that. By representing ...

Conquering the chaos in modern, multiprocessor computers

Mar 10, 2010

(PhysOrg.com) -- Computers should not play dice. That, to paraphrase Einstein, is the feeling of a University of Washington computer scientist with a simple manifesto: If you enter the same computer command, you should get ...

Microsoft moving Office online in duel with Google

Jul 13, 2009

Microsoft on Monday said that the 2010 version of its popular Office software will feature online collaboration as the technology giant duels "in the cloud" with Internet titan Google.

Recommended for you

Taking great ideas from the lab to the fab

9 hours ago

A "valley of death" is well-known to entrepreneurs—the lull between government funding for research and industry support for prototypes and products. To confront this problem, in 2013 the National Science ...

SR Labs research to expose BadUSB next week in Vegas

10 hours ago

A Berlin-based security research and consulting company will reveal how USB devices can do damage that can conduct two-way malice, from computer to USB or from USB to computer, and can survive traditional ...

US warns retailers on data-stealing malware

12 hours ago

US government cybersecurity watchdogs warned retailers Thursday about malware being circulated that allows hackers to get into computer networks and steal customer data.

User comments : 15

Adjust slider to filter visible comments by rank

Display comments: newest first

Corban
5 / 5 (1) Mar 30, 2010
QA will hate this.
gunslingor1
3 / 5 (2) Mar 30, 2010
Good work and clever, however, this will only provide the effective apperance of semi-open source control and will be entirely limited to how much their program can do. True open source is limitless, you can do anything. I can gaurantee EFFECTIVE performance loss; I say effective because, for the above example of running itunes as a control in word, you would have to have iTunes running, Word running and the their UI program running as well. So, in reality, the system load will be as it should be, however, it will appear to be running slow because the oblivious user will think he only has word open.
Also, if it were true open source, you could do the above and combine only a small part of itunes in with word, which would (if designed correctly) reduce system load below that of just running word and itunes in parallel.

This will be useful for small desired changes, but by no means is it anywhere close to true open source nor the appearance of true open source for that matter. "Cute"
PieRSquare
not rated yet Mar 30, 2010
QA will hate this.

Tech support will REALLY hate this.
KingDWS
5 / 5 (1) Mar 30, 2010
QA will hate this.

Tech support will REALLY hate this.


Hackers will love this. Just ignore the man behind the screen, its the giant talking head that is real.
THoKling
not rated yet Mar 30, 2010
This is the lazy-man's API. It would be interesting to see what the non-programmers might do with this.
MorituriMax
1 / 5 (1) Mar 30, 2010
The hackers will just LOOOOOVE this idea. How about lets find an effective way to secure the software before we open it up to any hacker with a toolkit.
Megadeth312
not rated yet Mar 30, 2010
The hackers will just LOOOOOVE this idea. How about lets find an effective way to secure the software before we open it up to any hacker with a toolkit.


The hackers will just LOOOOOVE this idea. How about lets find an effective way to secure the software before we open it up to any hacker with a toolkit.


Two very good points, on the one side, perhaps them non-technical folks can get their hands dirty with a little customization here and there that may shave a few seconds off their daily routine, but really? Does ANYONE need another way for black hats to exploit the already ravaged ignorant public?

I say not.

ArtflDgr
1 / 5 (2) Mar 30, 2010
more important would be to make odd engineering type design tools available. if you want to stimulate the creation of new wonderful things, connecting the impoverished lone or small time designer with the right test tools (think like spice for electronics), and then let them go...

until photoshop was affordable and so many people had it such amazing graphics were rare...

magnetic simulations, and structural ones in which you can load materials and put together ideas... but not the price of inventor/autocad (and of course not as featured or specialized)

enable creation of more technology, not disable the very market you need to support such improvements

EvgenijM
not rated yet Mar 31, 2010
Open Source is good, because you can change existing feature to use a different algorithm. Can this be done in prefab?
glene77is
not rated yet Mar 31, 2010
Gunslingor is right.
This is API stuff,
and is not heading in the direction of the Open Source Concept.
Feldagast
not rated yet Mar 31, 2010
Open source is a dream, people dont invest in something that can be had for free. They want return on investment, hence microsoft and apple products, closed systems that can be sold. If open source was the end all and be all they would have you believe it would have the 90% of market share of the os market already. What open source does do is provide an option that will keep microsoft and apple a little more honest with their products by provided a minimal of competition by providing a alternative to their product. If they dont produce something viable people will vote with their wallets and that WILL have a direct effect on what they are willing to do.
fourthrocker
Mar 31, 2010
This comment has been removed by a moderator.
CSharpner
not rated yet Mar 31, 2010
This is not open source by any stretch of the imagination, of course.

They're POLLING the screen??? and doing it about 20 times a second? Why not just intercept the draw commands? That way, their code only runs when something changes on the screen.

You can also accomplish some of this by modifying the embedded resources (the "resource fork" on Macs).

An "open source" solution would be something that can decompile the compiled code to produce source code. Those have existed for decades, but what they produce looks nothing like the original source and is generally more difficult to mess with than writing a competing product from scratch. There ARE exceptions, but there's no universal decompiler that produces manageable source code for everything. And this prefab product is not that type of product.
ricegf
5 / 5 (2) Apr 01, 2010
Open source is a dream, people dont invest in something that can be had for free.


LOL. Tell that to IBM and Red Hat (cf. Linux), Mozilla (cf. Firefox), Google (cf. Python and Android), Nokia (cf. Symbian), Intel (cf. MeeGo), Apple (cf. Darwin and MacRuby), and Microsoft (cf. CodePlex). Or did you think open source software was mostly written by amateurs or something? While you slept, free software conquered the world...
stealthc
1 / 5 (1) Apr 04, 2010
Replacing form elements is not that big of an accomplishment. There is a specific dll that has all those controls in it, forms.dll?

Anyways scanning for access to a specific dll would work better, or even better perhaps creating your own custom dll with trapping code for each element would mean that scans only occur when this thing is accessed. I doubt their idea is anywhere near this efficient.

I think this is a step in the right direction, but it would be nice to extend this behavior a little to make a more flexible utility.
gwrede
1 / 5 (1) Apr 05, 2010
Hmm. How about inserting a new version of the Password Entry Field?