Improving computer-supported work through scenario-based evaluation

Feb 07, 2005

An 18-month study by Penn State researchers of a collaborative system for managing and maintaining 800 military vehicles has helped identify how to make the multi-million-dollar system even more useful and efficient.
The researchers drew their conclusions after analyzing use scenarios of the integrated digital environment (IDE) that helps the U.S. Marine Corps manage its light-armored vehicles (LAVs) deployed worldwide. The scenarios describe the actual contexts in which the study's participants-representative engineers, scientists, logisticians, managers and others-use the IDE system.

Analysis of the scenarios yielded four types of benefits: "measurable benefits"; "tangible benefits" recognized by time savings; and "intangible benefits" such as aiding personnel management. The researchers also found "unrealized benefits" or opportunities to gain additional value from the implemented system-by-system redesign and organizational initiatives.

"Scenarios encourage consideration of the contextual factors that impact effective use of collaborative systems," said Steven Haynes, assistant professor in the School of Information Sciences and Technology (IST) and lead researcher. "When you get users to talk about a system and you follow up with specific questions, you learn about system benefits, design recommendations for future iterations and organizational barriers."

Because they support complex organizational, psychological and social processes in the workplace, computer-supported systems for multiple users are difficult to evaluate. But evaluation plays a central role in design and assessment as well as in justifying organizational investments in such systems. The Penn State researchers created the scenario-based evaluation (SBE) method as an alternative to evaluations that focus simply on feature sets and their performance attributes.

That method is detailed in a paper titled "Situating Evaluation in Scenarios of Use" given at the ACM Computer Supported Cooperative Work Conference in November. Haynes' co-authors are Sandeep Purao, an associate professor in the Penn State School of Information Sciences and Technology (IST) and Amie Skattebo, a graduate student in industrial and organizational psychology.

To evaluate the integrated digital environment or IDE system, the Penn State researchers worked on site at a Marine Corps' base, talking with different IDE users, each of whom has different requirements and expectations for the software. The IDE provides communication tools, process support, knowledge management and other functionalities for work coordination.

The researchers found that scenario analysis was effective for understanding the organizational context of the system and ways to improve the IDE design. For example, the researchers learned that participants used only 19 of the system's 76 features, suggesting additional training was needed. They also found that designers should focus on optimizing high-volume, low-value tasks such as leave requests to avoid alienating users and encourage system adoption.

Less easy to pinpoint were measurable benefits.

"It is very hard for people to articulate and pinpoint positive contributions," Haynes said. "They knew they were receiving information faster but couldn't quantify that. They knew there were time savings, but again, they couldn't quantify that." While scenarios are specific to a system in use, the researchers said this evaluation method could be generalized to other complex systems.

"One of the benefits of this method is that the results are rich and dense," Purao said. "The recommendations can inform system redesign as well as suggest organizational initiatives to ensure uptake."

Source: Penn State

Explore further: To improve STEM diversity, fix higher education, scholar says

Related Stories

Thawing permafrost feeds climate change

9 minutes ago

Carbon, held in frozen permafrost soils for tens of thousands of years, is being released as Arctic regions of the Earth warm and is further fueling global climate change, according to a Florida State University ...

Heat still on despite warming slowdown

12 minutes ago

The recent slowdown in the rise of global average air temperatures will make no difference to how much the planet will warm by 2100, a new study has found.

Can sound help us detect 'earthquakes' on Venus?

12 minutes ago

Detecting an "earthquake" on Venus would seem to be an impossible task. The planet's surface is a hostile zone of crushing pressure and scorching temperatures—about 874 degrees F, hot enough to melt lead—that ...

Recommended for you

Study finds we think better on our feet, literally

20 hours ago

A study from the Texas A&M Health Science Center School of Public Health finds students with standing desks are more attentive than their seated counterparts. In fact, preliminary results show 12 percent ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.