(Phys.org) —The Human Interfaces Group at Carnegie Mellon, led by the group's director Chris Harrison, an assistant professor of Human Computer Interaction, have done work that shows how traditional hand movements to perform tasks such as measuring and erasing can be naturally applied to the digital screen, improving on a natural interaction with computers. They have come up with TouchTools, a gesture design approach. With TouchTools, you manipulate tools on the screen just as you would in real life; the idea is to make software more natural to use. They are showing the world their TouchTools concept, which Gizmodo recently referred to as "skeuomorphism applied to interaction design."
Harrison said, "The core idea behind TouchTools is to draw upon user familiarity and motor skill with tools from the real world, and bring them to interactive use on computers."
The same hand gestures that you would use, for example, to run a tape measure across an item in the real world becomes the same hand gesture you use with a virtual, realistic-looking tape measure across the screen. Grab the virtual tape measure appearing on the screen, and you can roll it out, or move a market to draw, or press a camera shutter button.
Using TouchTools, users replicate a tool's corresponding real-world grasp and press it to the screen as though it was physically present. The system recognizes this pose and instantiates the virtual tool as if it was being grasped at that position. Users can rotate and manipulate the tool as they would its physical counterpart. But what's wrong with touchscreen interactions today? Why bother? The team's argument is that hand movements can be cumbersome in today's interactive environments and their approach is more natural.
"Contemporary applications often expose a toolbar that allows users to toggle between modes (e.g., pointer, pen, eraser modes) or require use of a special physical tool, such as a stylus. TouchTools can utilize the natural modality of our hands, rendering these accessories superfluous."
The way a computer user has to chord the fingers is not natural, as we do not normally perform actions in the real world by movements with certain numbers of fingers, as we do for screen tasks. Yet, as the group noted, the average person can skillfully manipulate a plethora of tools, from hammers to tweezers, with human hand actions. Despite this remarkable natural dexterity, gestures on today's touch devices are simplistic, relying primarily on the chording of fingers: one-finger pan, two-finger pinch, four-finger swipe.
"We propose that touch gesture design be inspired by the manipulation of physical tools from the real world. In this way, we can leverage user familiarity and fluency with such tools to build a rich set of gestures for touch interaction." This is what the team said in their paper, "TouchTools: Leveraging Familiarity and Skill with Physical Tools to Augment Touch Interaction," prepared for CHI 2014, which took place in Toronto from April 26 to May 1. They reported results after recruiting participants, and an iPad was used. Participants were given physical versions of various test tools to handle. "With only a few minutes of training on a proof-of-concept system, users were able to summon a variety of virtual tools by replicating their corresponding real-world grasps." The authors said they believe that "designing gestures around real-world tools improves discoverability, intelligibility and makes gestures memorable."
One might contend, why go to this effort when a tablet is so easy to use? Would TouchTools be seen as a lot of unnecessary bother? Arguments for the use of skeuomorphism in digital design involve ease of use. Digital emulation with objects and the way humans use them provide easy familiarity.
Gizmodo commented that what the Carnegie Mellon team is trying to convey is, "the library of interactions we currently use is quite thin—and by looking at the world around us, and the 'natural modality' of our 10 fingers, designers might find unexpectedly smart solutions to digital problems."
To be sure, the authors of the TouchTools paper, Chris Harrison, Robert Xiao, Julia Schwarz and Scott E. Hudson, stated that "We hope this work offers a new lens through which the HCI community can craft novel touch experiences."
Explore further: Carnegie Mellon system lets iPad users explore data with their fingers
More information: TouchTools: chrisharrison.net/index.php/Research/Touchtools
Research paper: Harrison, C., Xiao, R., Schwarz, J., and Hudson, S. TouchTools: Leveraging Familiarity and Skill with Physical Tools to Augment Touch Interaction. In Proceedings of the 32nd Annual SIGCHI Conference on Human Factors in Computing Systems (Toronto, Canada, April 26 - May 1, 2014). CHI '14. ACM, New York, NY. dx.doi.org/10.1145/2556288.2557012