Beyond peer review: NIST and five journals find a way to manage errors in research data

Sep 11, 2013
Beyond peer review: NIST and five journals find a way to manage errors in research data
NIST chemists Michael Frenkel and Robert Chirico led a collaborative study with five journals to find ways to ensure the quality of thermophysical property data used in chemical process technologies. Credit: Burrus/NIST

Traditional peer review is not enough to ensure data quality amid the recent boom in scientific research findings, according to results of a 10-year collaboration between the National Institute of Standards and Technology (NIST) and five technical journals.

While production of research data is growing about 7 percent annually, about one-third of papers submitted to participating journals contained erroneous or incomplete chemical property data, according to a report by 32 authors from NIST and the collaborating journals. Poor data can lead to mistakes in equipment selection, over-design of industrial plant components, difficulty simulating and discovering new processes, and poor regulatory decisions, the report notes.

The traditional process is under pressure to work too fast to evaluate fully all new experimental data, the NIST-journal collaboration found. The authors' solution is a set of customized software tools and procedures for validating experimental data and eliminating errors after a paper is approved by peer review, but before a journal formally accepts the paper.

The collaboration focused on thermophysical property data used in chemical process technologies such as distillation, extraction and crystallization. Thermophysical properties include boiling and melting points, density, viscosity, solubility and many other physical values related to temperature, including those for mixtures. The study findings also may be of broad value to scientific data publishing in general.

"The solutions we offer, while centered on the field of thermodynamics, should be applicable in principle to other areas of science and engineering," says Michael Frenkel, a co-author of the new paper and leader of NIST's Thermodynamics Research Center.

Managing thermophysical property data is particularly challenging because some 100-year-old data remain useful today for engineering purposes. Efforts to establish data-reporting standards for this field began more than 50 years ago but could not succeed until recently, with the development of specialized computer hardware and software tools.

The collaboration cites a variety of factors contributing to poor data quality. Advances in measurement science have boosted data collection, but increased automation has resulted in the loss of personnel expertise and knowledge required to run manual systems. In addition, equipment manufacturers sometimes make invalid uncertainty claims. And word processing software, with functions such as cut-and-paste and "fill down" in spreadsheets, has led to many published errors. The most common problem found in papers analyzed by the collaboration was missing or underestimated uncertainties for reported data.

After several tries, the collaboration developed a rapid, cost-effective process for identifying and eliminating errors. NIST developed a new online tool (NIST ThermoLit) that allows researchers to generate a literature report containing relevant references retrieved from a NIST database. Researchers can combine this capability with an older experiment planning system (NIST ThermoPlan) at both the conceptualization and publication stages of their work. If the submitted paper passes a journal's peer review, NIST generates a report noting any inconsistencies between the new experimental data and critically evaluated data based on past research. Data are extracted from the submitted paper and validated by NIST's expert software system for data evaluation (NIST ThermoData Engine).

Explore further: Japan orders air bag maker to conduct probe

More information: Chirico, R. et al. Improvement of quality in publication of experimental thermophysical property data: Challenges, assessment tools, global implementation, and online support, Journal of Chemical and Engineering Data, Sept. 6, 2013. DOI: 10.1021/je400569s

add to favorites email to friend print save as pdf

Related Stories

Peer review option proposed for biodiversity data

Oct 25, 2012

Data publishers should have the option of submitting their biodiversity datasets for peer review, according to a discussion paper commissioned by the Global Biodiversity Information Facility (GBIF).

NIST releases major update of popular REFPROP database

Apr 13, 2007

The National Institute of Standards and Technology has released an expanded and upgraded version of a popular database, a computer package for calculating the properties and modeling the behavior of fluids.

NIST chip measurement advance earns 'oscar of innovation'

Jul 15, 2013

A fundamental advance in measurement capabilities that could save semiconductor manufacturers billions of dollars annually has earned a 2013 R&D 100 Award for its National Institute of Standards and Technology (NIST) inventors.

Air conditioner 'evolves' in novel study

Sep 04, 2013

Played out on a computer over hundreds of generations, a survival-of-the-fittest programming method adapted by National Institute of Standards and Technology (NIST) researchers has spawned, of all things, ...

Recommended for you

Japan orders air bag maker to conduct probe

Nov 21, 2014

Japan's transport ministry said Friday it has ordered air bag maker Takata to conduct an internal investigation after cases of its air bags exploding triggered safety concerns in the United States and other countries.

Senators get no clear answers on air bag safety

Nov 20, 2014

There were apologies and long-winded explanations, but after nearly four hours of testimony about exploding air bags, senators never got a clear answer to the question most people have: whether or not their ...

Winter-like temps can reduce tire pressure

Nov 19, 2014

The polar plunge that has chilled much of the nation does more than bring out ice scrapers and antifreeze. It can trigger vehicles' tire pressure monitoring systems overnight, sending nervous drivers to dealers ...

US: Gov't aircraft regulations apply to drones (Update)

Nov 18, 2014

The U.S. government has the power to hold drone operators accountable when they operate the remote-control aircraft recklessly, a federal safety board ruled Tuesday in a setback to small drone operators chafing ...

Mapping the crisis of displaced peoples

Nov 17, 2014

Population displacement is a global problem, one that historically has been insufficiently quantified and analyzed, especially given its wide-ranging effects. Displacement can result from a number of factors, ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.