You can take it with you: Transfer open applications between computer, cellphone with a snapshot

June 16, 2011 by Larry Hardesty, MIT News
When Deep Shot transfers an open application to a cellphone, it automatically resizes the application window to match the framing of the photo.Photo courtesy of Tsung-Hsiang Chang

Have you ever found yourself looking up directions on your computer just before running out the door, only to end up retyping the same addresses and mapping the same route on your phone minutes later? A new system designed by Tsung-Hsiang Chang, a graduate student in MIT’s Computer Science and Artificial Intelligence Laboratory, and Google’s Yang Li makes it much easier to transfer computing tasks between devices. Simply take a photo of your computer screen with your smartphone’s camera, and the phone automatically opens up the corresponding application in the corresponding state. The same process can also work in reverse, moving data from the phone to a desktop computer.

“People are used to using heavy tools to transfer data or synchronize two devices,” Chang says. “You have to plug in a USB cable and maybe open iTunes and synchronize a bunch of data at the same time. But sometimes you just want to send a tiny bit of information, or a single piece of information.”

The system, called Deep Shot, exploits the fact that many use a standard format, called the uniform resource identifier (URI), to describe the states they’re in. For instance, if you search for directions on Google Maps, then click the link that says “Link,” a window pops up that says “Paste link in email or IM.” The link consists of a long string of symbols, including the addresses of the starting and ending points and codes that indicate their geographical coordinates and the approximate size of the map window. That’s a URI. The data contained in a URI can vary widely, and URIs are a common feature of many Web applications — if sometimes harder to extract than they are with Google Maps.

UR here

Deep Shot requires installing some minimal software on the phone and on all of the computers with which the phone will interact: You might, for instance, want to use the same phone in conjunction with a home desktop, a home laptop, a work desktop and your spouse or children’s computers. When it’s uploading data to a phone, Deep Shot uses existing vision algorithms to identify the application open on screen, and then the software installed on the computer extracts and transmits the corresponding URI. Changing the framing of the photograph will also change the framing of the application window that opens on the phone. Conversely, when Deep Shot is downloading data from the phone, the computer vision algorithm identifies the computer that the camera is trained on.

The video will load shortly
A demonstration showing how Deep Shot worksVideo: Tsung-Hsiang Chang and Yang Li

Chang and Li developed the system last summer when Chang was doing an internship at Google, and they presented a paper on it at the Association for Computing Machinery’s CHI conference on computer-human interaction in May. At present, the system works with several common Web applications, such as Google Maps and Yelp, and with minimal additional coding it could be made to work with any application that reveals its state through URIs. Since URIs use a standardized set of codes, Deep Shot can also mediate between different applications, transferring data from one map application running on a desktop computer, say, to a different map application installed on a . In principle, the system could also work with off-the-shelf software: It would just require software developers to make some minimal modifications to their code.

Fanning out

“I find it a really compelling use case, so I would really hope that companies like Microsoft would really consider adding it,” says Jeffrey Nichols, a researcher at IBM’s Almaden research center who specializes in mobile devices. But, he points out, getting Deep Shot to work with desktop software from a lot of different manufacturers would require broad agreement on interoperability standards, which is hard to come by. “I see it being much more likely to happen with websites than with desktop applications,” Nichols says. “On the other hand, to some extent, we’re moving away from desktop applications and moving more and more to the Web, so it’s not clear to me how important it is that we really bring all the native application developers into the fold.”

Moreover, even in cases where Web developers are reluctant to reveal information about the states of their applications, “there are a lot of ways for people who are not even involved in the creation of the website to add features into a website,” Nichols says. “So you could imagine that if the technology gets a foothold in a few places, then even if some websites aren’t willing to add it, then other people might add it for them.”

Since Chang was at Google when he worked on Deep Shot, Google owns the rights to it. Google has not yet made the system publicly available, but when it does, Chang will be among the first to install it, having come to depend on it during the summer he helped invent it. “It just makes everything so much easier,” he says.

This story is republished courtesy of MIT News (, a popular site that covers news about MIT research, innovation and teaching.

Explore further: For software developers, more speed and mobility

Related Stories

For software developers, more speed and mobility

December 14, 2010

Across the globe, technology and innovation are becoming increasingly more reliant on mobility and accessibility. For software developers working on highly complex projects, that means being able to save their work quickly ...

Turning reviews into ratings

February 3, 2011

The proliferation of websites such as Yelp and CitySearch has made it easy to find local businesses that meet common search criteria -- moderately priced seafood restaurants, for example, within a quarter-mile of a particular ...

Dutch support for disaster zone phone software

April 13, 2011

Software developed by Flinders University’s Dr. Paul Gardner-Stephen which enables mobile phones to communicate during a disaster will be freely available to the public by the end of the year thanks to the support of ...

Safe 'sandbox' for the internet of the future

May 13, 2011

To better protect new Internet applications against hacker attacks and other types of manipulation, Siemens is taking part in the EU’s WebSand research project. In cooperation with partner organizations SAP, the University ...

Streamlined rules for robots

June 13, 2011

With the explosion of the Internet and the commoditization of autonomous robots (such as the Roomba) and small sensors (such as the ones in most cell phones), computer scientists have become more and more interested in distributed ...

Recommended for you

Video privacy software lets you select what others can see

June 28, 2016

Camera-equipped smartphones, laptops and other devices make it possible to share ideas and images with anyone, anywhere, often in real-time. But in our cameras-everywhere culture, the risk of accidentally leaking sensitive ...

US Navy keeps electromagnetic cannon in its sights

June 25, 2016

The US Navy is quietly pushing ahead with a radical new cannon that one day could transform how wars are fought, even though some Pentagon officials have voiced concerns over its cost and viability.

Mapping coal's decline and the renewables' rise

June 23, 2016

Even as coal-fired power plants across the U.S. are shutting down in response to new environmental regulations and policy mandates, defenders of the emissions-heavy fuel still have cost on their side. Coal, after all, is ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

4 / 5 (1) Jun 16, 2011
Very Creative!

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.