What if all software was open source? A code to unlock the desktop

March 30, 2010 by Hannah Hickey, University of Washington

(PhysOrg.com) -- What if all software was open source? Anybody would then be able to add custom features to Microsoft Word, Adobe Photoshop, Apple iTunes or any other program. A University of Washington project may make this possible.

"Microsoft and Apple aren't going to open up all their stuff. But they all create programs that put pixels on the screen. And if we can modify those pixels, then we can change the program's apparent behavior," said James Fogarty, a UW assistant professor of computer science and engineering.

His approach hijacks the display to customize the user's interaction with the program. He will demonstrate his system April 14 in Atlanta at the Association for Computing Machinery's Conference on Human Factors in .

"We really see this as a first step toward a scenario where anybody can modify any application," Fogarty said. "In a sense, this has happened online. You've got this mash-up culture on the Web because everybody can see the HTML. But that hasn't been possible on the desktop."

These days a Web page might include a map from , an embedded video from and a list of recent headlines. This is not yet possible on the personal computer.

"Let's say I'm writing a paper in Microsoft Word but I want to listen to music at the same time," explained co-author Morgan Dixon, a UW doctoral student in computer science and engineering.

Right now he would have to click back and for the between Word and iTunes, but the system he helped create can simply add a few iTunes buttons to the Word toolbar.

"I'm using some program that I love," Dixon said, "and I'm going to stick in some features from some other program that I love, so I have a more unified interface."

More importantly, having more control over widely used programs would allow people to benefit from accessibility tools that have been gathering dust in academic research labs.

An example is target-aware pointing, which can make many interfaces easier for people with muscular dystrophy, Parkinson's disease, cerebral palsy or other motor-control disabilities. One such tool, the bubble cursor, highlights the button closest to it, making it easier for people with disabilities to click a button without having to hit it dead on. Fogarty and Dixon show the first implementation of a bubble cursor in various commercial applications.

"The human-computer interaction community has done 30 years of research on how to make computers more accessible to people with disabilities. But no one change is perfect for everybody," Fogarty said. "That's why you don't see these tools out there."

His research allows people to personalize programs based on their needs.

The UW tool, named Prefab, takes advantage of the fact that almost all displays are made from prefabricated blocks of code such as buttons, sliders, check boxes and drop-down menus. Prefab looks for those blocks as many as 20 times per second and alters their behavior.

The researchers are continuing to develop Prefab and are exploring options for commercialization.

Prefab unlocks previously inaccessible interfaces, allowing people to add the same usability tool to all the applications they run on their desktop. The system could translate a program's interface into a different language, or reorder menus to bump up favorite commands.

The authors hope Prefab will spur development of new innovations.

"If you come up with a new technology, too often it's evaluated in a test environment," Fogarty said. "This lets researchers put it into practice in something real, like Photoshop or iTunes."

Prefab can also produce more advanced effects. One demonstration that will be presented at the conference creates multiple previews of a single image in Photoshop. Behind the scenes, Prefab moves the sliders to different points, captures the output and then displays all of them on a single screen. This could save time by showing a range of effects the user frequently adjusts.

The system could also allow programs to move from computer screens to mobile devices, which do not have a standard operating system.

"It dramatically lowers the threshold to getting new innovation into existing, complex programs," Fogarty said.

Research has been funded by the Hacherl Endowed Graduate Fellowship in the UW Department of & Engineering, a fellowship from the Seattle chapter of the Achievement Rewards for College Scientists, and Intel.

Explore further: Online service lets blind surf the Internet from any computer, anywhere

More information: -- More information about Prefab: www.cs.washington.edu/homes/jf … rty/research/prefab/
-- "Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure" To appear in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2010). Winner of Best Paper award. uwnews.org/relatedcontent/2010 … 6581_thisID56586.pdf

Related Stories

Picture-driven computing (w/ Video)

January 20, 2010

Until the 1980s, using a computer program meant memorizing a lot of commands and typing them in a line at a time, only to get lines of text back. The graphical user interface, or GUI, changed that. By representing programs, ...

Conquering the chaos in modern, multiprocessor computers

March 10, 2010

(PhysOrg.com) -- Computers should not play dice. That, to paraphrase Einstein, is the feeling of a University of Washington computer scientist with a simple manifesto: If you enter the same computer command, you should get ...

Microsoft moving Office online in duel with Google

July 13, 2009

Microsoft on Monday said that the 2010 version of its popular Office software will feature online collaboration as the technology giant duels "in the cloud" with Internet titan Google.

Recommended for you

Team breaks world record for fast, accurate AI training

November 7, 2018

Researchers at Hong Kong Baptist University (HKBU) have partnered with a team from Tencent Machine Learning to create a new technique for training artificial intelligence (AI) machines faster than ever before while maintaining ...


Adjust slider to filter visible comments by rank

Display comments: newest first

5 / 5 (1) Mar 30, 2010
QA will hate this.
3 / 5 (2) Mar 30, 2010
Good work and clever, however, this will only provide the effective apperance of semi-open source control and will be entirely limited to how much their program can do. True open source is limitless, you can do anything. I can gaurantee EFFECTIVE performance loss; I say effective because, for the above example of running itunes as a control in word, you would have to have iTunes running, Word running and the their UI program running as well. So, in reality, the system load will be as it should be, however, it will appear to be running slow because the oblivious user will think he only has word open.
Also, if it were true open source, you could do the above and combine only a small part of itunes in with word, which would (if designed correctly) reduce system load below that of just running word and itunes in parallel.

This will be useful for small desired changes, but by no means is it anywhere close to true open source nor the appearance of true open source for that matter. "Cute"
not rated yet Mar 30, 2010
QA will hate this.

Tech support will REALLY hate this.
5 / 5 (1) Mar 30, 2010
QA will hate this.

Tech support will REALLY hate this.

Hackers will love this. Just ignore the man behind the screen, its the giant talking head that is real.
not rated yet Mar 30, 2010
This is the lazy-man's API. It would be interesting to see what the non-programmers might do with this.
1 / 5 (1) Mar 30, 2010
The hackers will just LOOOOOVE this idea. How about lets find an effective way to secure the software before we open it up to any hacker with a toolkit.
not rated yet Mar 30, 2010
The hackers will just LOOOOOVE this idea. How about lets find an effective way to secure the software before we open it up to any hacker with a toolkit.

The hackers will just LOOOOOVE this idea. How about lets find an effective way to secure the software before we open it up to any hacker with a toolkit.

Two very good points, on the one side, perhaps them non-technical folks can get their hands dirty with a little customization here and there that may shave a few seconds off their daily routine, but really? Does ANYONE need another way for black hats to exploit the already ravaged ignorant public?

I say not.

1 / 5 (2) Mar 30, 2010
more important would be to make odd engineering type design tools available. if you want to stimulate the creation of new wonderful things, connecting the impoverished lone or small time designer with the right test tools (think like spice for electronics), and then let them go...

until photoshop was affordable and so many people had it such amazing graphics were rare...

magnetic simulations, and structural ones in which you can load materials and put together ideas... but not the price of inventor/autocad (and of course not as featured or specialized)

enable creation of more technology, not disable the very market you need to support such improvements

not rated yet Mar 31, 2010
Open Source is good, because you can change existing feature to use a different algorithm. Can this be done in prefab?
not rated yet Mar 31, 2010
Gunslingor is right.
This is API stuff,
and is not heading in the direction of the Open Source Concept.
not rated yet Mar 31, 2010
This is not open source by any stretch of the imagination, of course.

They're POLLING the screen??? and doing it about 20 times a second? Why not just intercept the draw commands? That way, their code only runs when something changes on the screen.

You can also accomplish some of this by modifying the embedded resources (the "resource fork" on Macs).

An "open source" solution would be something that can decompile the compiled code to produce source code. Those have existed for decades, but what they produce looks nothing like the original source and is generally more difficult to mess with than writing a competing product from scratch. There ARE exceptions, but there's no universal decompiler that produces manageable source code for everything. And this prefab product is not that type of product.
5 / 5 (2) Apr 01, 2010
Open source is a dream, people dont invest in something that can be had for free.

LOL. Tell that to IBM and Red Hat (cf. Linux), Mozilla (cf. Firefox), Google (cf. Python and Android), Nokia (cf. Symbian), Intel (cf. MeeGo), Apple (cf. Darwin and MacRuby), and Microsoft (cf. CodePlex). Or did you think open source software was mostly written by amateurs or something? While you slept, free software conquered the world...
1 / 5 (1) Apr 04, 2010
Replacing form elements is not that big of an accomplishment. There is a specific dll that has all those controls in it, forms.dll?

Anyways scanning for access to a specific dll would work better, or even better perhaps creating your own custom dll with trapping code for each element would mean that scans only occur when this thing is accessed. I doubt their idea is anywhere near this efficient.

I think this is a step in the right direction, but it would be nice to extend this behavior a little to make a more flexible utility.
1 / 5 (1) Apr 05, 2010
Hmm. How about inserting a new version of the Password Entry Field?

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.