Updated NIST software uses combination testing to catch bugs fast and easy

Nov 10, 2010
NIST's software for testing computer systems -- ACTS -- takes advantage of research that shows that virtually all software failures appear to be caused by six or fewer interactions. Credit: NIST

Researchers at the National Institute of Standards and Technology have released an updated version of a computer system testing tool that can cut costs by more efficiently finding flaws. A tutorial on using the tool accompanies the new release.

Catching "bugs" before a program is released enhances because hackers often exploit these flaws to introduce malware, including viruses, to disrupt or take control of computer systems. But it's difficult. A widely cited 2002 study prepared for NIST* reported that even though 50 percent of software development budgets go to testing, flaws in software still cost the U.S. economy $59.5 billion annually.

Exhaustive checking of all possible combinations of input actions that could cause software failure is not practical, explained NIST's Raghu Kacker, because of the huge number of possibilities, but it's also not necessary. Based on studies of software crashes in applications, including medical devices and Web browsers, NIST's Rick Kuhn and other researchers determined that between 70 and 95 percent of software failures are triggered by only two variables interacting and practically 100 percent of software failures are triggered by no more than six. "Testing every combination up to six variables can be as good as exhaustive testing," said Kacker.

Working with researcher Jeff Yu Lei and his students from the University of Texas at Arlington, NIST designed Advanced Combinatorial Testing System (ACTS), a freely distributed to generate plans for efficiently testing combinations of two to six interacting variables. The method goes beyond the commonly used "pairwise" approach to software testing, which tests combinations of two variables, so it can detect more obscure flaws. (See "'Combinatorial' Approach Squashes Software Bugs Faster, Cheaper" http://phys.org/news116693828.html)

Kuhn describes the process "as packing as many combinations into a set of tests as efficiently as we know how." For example, testing all possible interactions for a product with 34 on and off switches would require 17 billion tests. Using ACTS, all three-way interactions can be evaluated using only 33 tests and all six-way combinations with just 522 tests, instead of 17 billion.

The first version of ACTS was released in 2008. Since then, it has been distributed at no cost to 465 organizations and individuals in industry, academia and government. "About half of our users are in IT, but other heavy users are in the financial, defense and telecommunications sectors," said Kuhn. In August, NIST and Lockheed Martin initiated a Cooperative Research and Development Agreement to study the application of ACTS in the company's large and complex software applications. The two groups will jointly publish the results.

NIST released the latest update of ACTS in October. The new version includes an improved user interface and a better method of specifying relationships between parameters for testing. This can eliminate the problem, for example, of spending time on tests for invalid combinations, such as using Internet Explorer on a Linux system. Information for requesting ACTS is available at http://csrc.nist.gov/groups/SNS/acts/index.html.

Just released is a new tutorial, Practical Combinatorial Testing, that introduces key concepts and methods along with explaining the use of software tools for generating combinatorial tests. Cost and other practical considerations are addressed. The tutorial is designed to be accessible to undergraduate students in computer science or engineering and includes extensive references. NIST Special Publication 800-142 (link to http://csrc.nist.gov/groups/SNS/acts/documents/SP800-142-101006.pdf) can be downloaded at crsc.nist.gov/acts.

Explore further: Computerized emotion detector

More information: * Research Triangle Institute, The Economic Impacts of Inadequate Infrastructure for Software Testing, NIST Planning Report 02-3, May 2002.

add to favorites email to friend print save as pdf

Related Stories

Faster and more efficient software for the US Air Force

Jan 11, 2010

Researchers at the University of Nebraska in Lincoln have addressed the issue of faulty software by developing an algorithm and open source tool that is 300 times faster at generating tests and also reduces current software ...

Software Helps Developers Get Started with PIV Cards

Jul 10, 2008

The National Institute of Standards and Technology has developed two demonstration software packages that show how Personal Identity Verification (PIV) cards can be used with Windows and Linux systems to perform ...

How to Protect Your Web Server from Attacks

Oct 11, 2007

The National Institute of Standards and Technology has released a new publication that provides detailed tips on how to make web servers more resistant to potential attacks. Called “Guidelines on Securing Public Web Servers,” ...

NIST releases major update of popular REFPROP database

Apr 13, 2007

The National Institute of Standards and Technology has released an expanded and upgraded version of a popular database, a computer package for calculating the properties and modeling the behavior of fluids.

Recommended for you

Computerized emotion detector

13 hours ago

Face recognition software measures various parameters in a mug shot, such as the distance between the person's eyes, the height from lip to top of their nose and various other metrics and then compares it with photos of people ...

Cutting the cloud computing carbon cost

Sep 12, 2014

Cloud computing involves displacing data storage and processing from the user's computer on to remote servers. It can provide users with more storage space and computing power that they can then access from anywhere in the ...

Teaching computers the nuances of human conversation

Sep 12, 2014

Computer scientists have successfully developed programs to recognize spoken language, as in automated phone systems that respond to voice prompts and voice-activated assistants like Apple's Siri.

Mapping the connections between diverse sets of data

Sep 12, 2014

What is a map? Most often, it's a visual tool used to demonstrate the relationship between multiple places in geographic space. They're useful because you can look at one and very quickly pick up on the general ...

User comments : 0