Improving computer-supported work through scenario-based evaluation

Feb 07, 2005

An 18-month study by Penn State researchers of a collaborative system for managing and maintaining 800 military vehicles has helped identify how to make the multi-million-dollar system even more useful and efficient.
The researchers drew their conclusions after analyzing use scenarios of the integrated digital environment (IDE) that helps the U.S. Marine Corps manage its light-armored vehicles (LAVs) deployed worldwide. The scenarios describe the actual contexts in which the study's participants-representative engineers, scientists, logisticians, managers and others-use the IDE system.

Analysis of the scenarios yielded four types of benefits: "measurable benefits"; "tangible benefits" recognized by time savings; and "intangible benefits" such as aiding personnel management. The researchers also found "unrealized benefits" or opportunities to gain additional value from the implemented system-by-system redesign and organizational initiatives.

"Scenarios encourage consideration of the contextual factors that impact effective use of collaborative systems," said Steven Haynes, assistant professor in the School of Information Sciences and Technology (IST) and lead researcher. "When you get users to talk about a system and you follow up with specific questions, you learn about system benefits, design recommendations for future iterations and organizational barriers."

Because they support complex organizational, psychological and social processes in the workplace, computer-supported systems for multiple users are difficult to evaluate. But evaluation plays a central role in design and assessment as well as in justifying organizational investments in such systems. The Penn State researchers created the scenario-based evaluation (SBE) method as an alternative to evaluations that focus simply on feature sets and their performance attributes.

That method is detailed in a paper titled "Situating Evaluation in Scenarios of Use" given at the ACM Computer Supported Cooperative Work Conference in November. Haynes' co-authors are Sandeep Purao, an associate professor in the Penn State School of Information Sciences and Technology (IST) and Amie Skattebo, a graduate student in industrial and organizational psychology.

To evaluate the integrated digital environment or IDE system, the Penn State researchers worked on site at a Marine Corps' base, talking with different IDE users, each of whom has different requirements and expectations for the software. The IDE provides communication tools, process support, knowledge management and other functionalities for work coordination.

The researchers found that scenario analysis was effective for understanding the organizational context of the system and ways to improve the IDE design. For example, the researchers learned that participants used only 19 of the system's 76 features, suggesting additional training was needed. They also found that designers should focus on optimizing high-volume, low-value tasks such as leave requests to avoid alienating users and encourage system adoption.

Less easy to pinpoint were measurable benefits.

"It is very hard for people to articulate and pinpoint positive contributions," Haynes said. "They knew they were receiving information faster but couldn't quantify that. They knew there were time savings, but again, they couldn't quantify that." While scenarios are specific to a system in use, the researchers said this evaluation method could be generalized to other complex systems.

"One of the benefits of this method is that the results are rich and dense," Purao said. "The recommendations can inform system redesign as well as suggest organizational initiatives to ensure uptake."

Source: Penn State

Explore further: Understanding faith, teaching evolution not mutually exclusive

add to favorites email to friend print save as pdf

Related Stories

Fresh nuclear leak detected at Fukushima plant

5 hours ago

Sensors at the Fukushima nuclear plant have detected a fresh leak of highly radioactive water to the sea, the plant's operator announced Sunday, highlighting difficulties in decommissioning the crippled plant.

Spacewalking astronauts route cable in 1st of 3 jobs

5 hours ago

(AP)—Spacewalking astronauts routed more than 300 feet (90 meters) of cable outside the International Space Station on Saturday, tricky and tiring advance work for the arrival of new American-made crew ...

Driverless shuttle will be on the move in UK

6 hours ago

(Phys.org) —"Autonomous public transport" is on the minds of planners who envision self-driving vehicles that would cross over short distances, suited for airport transport, industrial sites, theme parks ...

Superfish points fingers over ad software security flaws

9 hours ago

A little-known Silicon Valley startup was caught in a firestorm of criticism this week for making software that exposed Lenovo laptop users to hackers bent on stealing personal information. But Superfish Inc. ...

Recommended for you

Ancient wheat points to Stone Age trading links

12 hours ago

(AP)—Britons may have discovered a taste for bread thousands of years earlier than previously thought, thanks to trade with more advanced neighbors on the European continent.

Humour in the 13th century characterized by ridicule

15 hours ago

We tend to think of the Middle Ages as grotesque and dreary. However, 13th century elites made use of laughter quite deliberately – and it resounded most loudly when it was at someone else's expense.

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.