Improving computer-supported work through scenario-based evaluation

Feb 07, 2005

An 18-month study by Penn State researchers of a collaborative system for managing and maintaining 800 military vehicles has helped identify how to make the multi-million-dollar system even more useful and efficient.
The researchers drew their conclusions after analyzing use scenarios of the integrated digital environment (IDE) that helps the U.S. Marine Corps manage its light-armored vehicles (LAVs) deployed worldwide. The scenarios describe the actual contexts in which the study's participants-representative engineers, scientists, logisticians, managers and others-use the IDE system.

Analysis of the scenarios yielded four types of benefits: "measurable benefits"; "tangible benefits" recognized by time savings; and "intangible benefits" such as aiding personnel management. The researchers also found "unrealized benefits" or opportunities to gain additional value from the implemented system-by-system redesign and organizational initiatives.

"Scenarios encourage consideration of the contextual factors that impact effective use of collaborative systems," said Steven Haynes, assistant professor in the School of Information Sciences and Technology (IST) and lead researcher. "When you get users to talk about a system and you follow up with specific questions, you learn about system benefits, design recommendations for future iterations and organizational barriers."

Because they support complex organizational, psychological and social processes in the workplace, computer-supported systems for multiple users are difficult to evaluate. But evaluation plays a central role in design and assessment as well as in justifying organizational investments in such systems. The Penn State researchers created the scenario-based evaluation (SBE) method as an alternative to evaluations that focus simply on feature sets and their performance attributes.

That method is detailed in a paper titled "Situating Evaluation in Scenarios of Use" given at the ACM Computer Supported Cooperative Work Conference in November. Haynes' co-authors are Sandeep Purao, an associate professor in the Penn State School of Information Sciences and Technology (IST) and Amie Skattebo, a graduate student in industrial and organizational psychology.

To evaluate the integrated digital environment or IDE system, the Penn State researchers worked on site at a Marine Corps' base, talking with different IDE users, each of whom has different requirements and expectations for the software. The IDE provides communication tools, process support, knowledge management and other functionalities for work coordination.

The researchers found that scenario analysis was effective for understanding the organizational context of the system and ways to improve the IDE design. For example, the researchers learned that participants used only 19 of the system's 76 features, suggesting additional training was needed. They also found that designers should focus on optimizing high-volume, low-value tasks such as leave requests to avoid alienating users and encourage system adoption.

Less easy to pinpoint were measurable benefits.

"It is very hard for people to articulate and pinpoint positive contributions," Haynes said. "They knew they were receiving information faster but couldn't quantify that. They knew there were time savings, but again, they couldn't quantify that." While scenarios are specific to a system in use, the researchers said this evaluation method could be generalized to other complex systems.

"One of the benefits of this method is that the results are rich and dense," Purao said. "The recommendations can inform system redesign as well as suggest organizational initiatives to ensure uptake."

Source: Penn State

Explore further: Violence rates can be halved in just 30 years, say leading experts

add to favorites email to friend print save as pdf

Related Stories

China demand to fuel Hong Kong iPhone grey market

11 hours ago

Wealthy mainland Chinese looking to buy the new iPhone 6 next week could expect to pay an eye-watering US$2,500 for the handsets in Hong Kong, following Apple's decision to delay the launch in China.

Netflix sets sights on European screens

11 hours ago

US online streaming giant Netflix will launch the second phase of its European expansion plan on Monday as it sets about seducing French viewers with a "House of Cards"-style drama set in Marseille.

MIT ATLAS robot demo shows advanced moves (w/ Video)

Sep 07, 2014

The bipedal robot ATLAS from MIT is moving on. Reacting to the recent video of "MIT Atlas truckin' with a truss," TechCrunch said, "We've seen the cute little guy walk, toddle, and climb over obstacles but ...

Recommended for you

Entrepreneurs aren't overconfident gamblers

11 hours ago

Leaving one's job to become an entrepreneur is inarguably risky. But it may not be the fear of risk that makes entrepreneurs more determined to succeed. A new study finds entrepreneurs are also concerned about what they might ...

New branch added to European family tree

13 hours ago

The setting: Europe, about 7,500 years ago. Agriculture was sweeping in from the Near East, bringing early farmers into contact with hunter-gatherers who had already been living in Europe for tens of thousands ...

User comments : 0