Improving computer-supported work through scenario-based evaluation

Feb 07, 2005

An 18-month study by Penn State researchers of a collaborative system for managing and maintaining 800 military vehicles has helped identify how to make the multi-million-dollar system even more useful and efficient.
The researchers drew their conclusions after analyzing use scenarios of the integrated digital environment (IDE) that helps the U.S. Marine Corps manage its light-armored vehicles (LAVs) deployed worldwide. The scenarios describe the actual contexts in which the study's participants-representative engineers, scientists, logisticians, managers and others-use the IDE system.

Analysis of the scenarios yielded four types of benefits: "measurable benefits"; "tangible benefits" recognized by time savings; and "intangible benefits" such as aiding personnel management. The researchers also found "unrealized benefits" or opportunities to gain additional value from the implemented system-by-system redesign and organizational initiatives.

"Scenarios encourage consideration of the contextual factors that impact effective use of collaborative systems," said Steven Haynes, assistant professor in the School of Information Sciences and Technology (IST) and lead researcher. "When you get users to talk about a system and you follow up with specific questions, you learn about system benefits, design recommendations for future iterations and organizational barriers."

Because they support complex organizational, psychological and social processes in the workplace, computer-supported systems for multiple users are difficult to evaluate. But evaluation plays a central role in design and assessment as well as in justifying organizational investments in such systems. The Penn State researchers created the scenario-based evaluation (SBE) method as an alternative to evaluations that focus simply on feature sets and their performance attributes.

That method is detailed in a paper titled "Situating Evaluation in Scenarios of Use" given at the ACM Computer Supported Cooperative Work Conference in November. Haynes' co-authors are Sandeep Purao, an associate professor in the Penn State School of Information Sciences and Technology (IST) and Amie Skattebo, a graduate student in industrial and organizational psychology.

To evaluate the integrated digital environment or IDE system, the Penn State researchers worked on site at a Marine Corps' base, talking with different IDE users, each of whom has different requirements and expectations for the software. The IDE provides communication tools, process support, knowledge management and other functionalities for work coordination.

The researchers found that scenario analysis was effective for understanding the organizational context of the system and ways to improve the IDE design. For example, the researchers learned that participants used only 19 of the system's 76 features, suggesting additional training was needed. They also found that designers should focus on optimizing high-volume, low-value tasks such as leave requests to avoid alienating users and encourage system adoption.

Less easy to pinpoint were measurable benefits.

"It is very hard for people to articulate and pinpoint positive contributions," Haynes said. "They knew they were receiving information faster but couldn't quantify that. They knew there were time savings, but again, they couldn't quantify that." While scenarios are specific to a system in use, the researchers said this evaluation method could be generalized to other complex systems.

"One of the benefits of this method is that the results are rich and dense," Purao said. "The recommendations can inform system redesign as well as suggest organizational initiatives to ensure uptake."

Source: Penn State

Explore further: Mapping the world's linguistic diversity—scientists discover links between your genes and the language you speak

Related Stories

Hubble view: Wolf-Rayet stars, intense and short-lived

51 minutes ago

This NASA/European Space Agency (ESA) Hubble Space Telescope picture shows a galaxy named SBS 1415+437 (also called SDSS CGB 12067.1), located about 45 million light-years from Earth. SBS 1415+437 is a Wolf-Rayet ...

Five properties of physics that affect your gas mileage

59 minutes ago

Physics is inescapable. It's everywhere, making your Frisbees fly, your toilets flush and your pasta water boil at a lower temperature at altitude. We've harnessed these forces, along with chemistry and engineering, ...

Recommended for you

Lady, you're on the money

Jul 03, 2015

So far, women whose portraits appear on U.S. money have been a party of three. Excluding commemorative currency, only Sacagawea, Susan B. Anthony and Helen Keller appear on coins in general circulation, according ...

Old World monkey had tiny, complex brain

Jul 03, 2015

The brain hidden inside the oldest known Old World monkey skull has been visualized for the first time. The creature's tiny but remarkably wrinkled brain supports the idea that brain complexity can evolve ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.