Improving computer-supported work through scenario-based evaluation
An 18-month study by Penn State researchers of a collaborative system for managing and maintaining 800 military vehicles has helped identify how to make the multi-million-dollar system even more useful and efficient.
The researchers drew their conclusions after analyzing use scenarios of the integrated digital environment (IDE) that helps the U.S. Marine Corps manage its light-armored vehicles (LAVs) deployed worldwide. The scenarios describe the actual contexts in which the study's participants-representative engineers, scientists, logisticians, managers and others-use the IDE system.
Analysis of the scenarios yielded four types of benefits: "measurable benefits"; "tangible benefits" recognized by time savings; and "intangible benefits" such as aiding personnel management. The researchers also found "unrealized benefits" or opportunities to gain additional value from the implemented system-by-system redesign and organizational initiatives.
"Scenarios encourage consideration of the contextual factors that impact effective use of collaborative systems," said Steven Haynes, assistant professor in the School of Information Sciences and Technology (IST) and lead researcher. "When you get users to talk about a system and you follow up with specific questions, you learn about system benefits, design recommendations for future iterations and organizational barriers."
Because they support complex organizational, psychological and social processes in the workplace, computer-supported systems for multiple users are difficult to evaluate. But evaluation plays a central role in design and assessment as well as in justifying organizational investments in such systems. The Penn State researchers created the scenario-based evaluation (SBE) method as an alternative to evaluations that focus simply on feature sets and their performance attributes.
That method is detailed in a paper titled "Situating Evaluation in Scenarios of Use" given at the ACM Computer Supported Cooperative Work Conference in November. Haynes' co-authors are Sandeep Purao, an associate professor in the Penn State School of Information Sciences and Technology (IST) and Amie Skattebo, a graduate student in industrial and organizational psychology.
To evaluate the integrated digital environment or IDE system, the Penn State researchers worked on site at a Marine Corps' base, talking with different IDE users, each of whom has different requirements and expectations for the software. The IDE provides communication tools, process support, knowledge management and other functionalities for work coordination.
The researchers found that scenario analysis was effective for understanding the organizational context of the system and ways to improve the IDE design. For example, the researchers learned that participants used only 19 of the system's 76 features, suggesting additional training was needed. They also found that designers should focus on optimizing high-volume, low-value tasks such as leave requests to avoid alienating users and encourage system adoption.
Less easy to pinpoint were measurable benefits.
"It is very hard for people to articulate and pinpoint positive contributions," Haynes said. "They knew they were receiving information faster but couldn't quantify that. They knew there were time savings, but again, they couldn't quantify that." While scenarios are specific to a system in use, the researchers said this evaluation method could be generalized to other complex systems.
"One of the benefits of this method is that the results are rich and dense," Purao said. "The recommendations can inform system redesign as well as suggest organizational initiatives to ensure uptake."
Source: Penn State
