Measuring science investments

A 2010 administrative memorandum calls on U.S. federal agencies and executive departments to develop tools to "better assess the impact of [...] science and technology investments."

Translation: There is increasing pressure to document the results of [...] research investments in a scientific manner, writes Julia Lane, Science of Science and Innovation Policy program director at the National Science Foundation (NSF) and her co-author Stefano Bertuzzi, Office of Science Policy at the National Institutes of Health (NIH).

They make the observation in, "Measuring the Results of Science Investments," a Policy Forum paper appearing in this week's edition of the journal Science.

Lane and Bertuzzi examine critical questions for measuring the effects of scientific research. Is it possible to create such a system? What would be the inputs, outputs and structure of the system? What scientific disciplines should inform the formulation of such a model?

"It is important that the scientific community, together with the federal government, engage in describing the results," said Lane of the special burdens placed on measuring the fundamental and exploratory research that NSF is responsible for funding, but does not always show immediate results.

"Fundamental research is a long way from the market place, so there are long lags before any results are seen," she said emphasizing the need to be diligent and sensitive to the exploratory nature of basic research.

The coauthors note other countries already have begun developing systematic ways to describe the results of science investments.

The Higher Education Funding Councils in England has been conducting Research Assessment Exercises to gauge the quality and impact of funded research since 1986. In addition, last year, the Japanese government began creating a program to advance the science of science and innovation.

In the United States, STAR METRICS attempts to address some of these same challenges. The Science and Technology for America's Reinvestment: Measuring the Effects of Research on Innovation, Competitiveness, and Science program is being developed by NIH and NSF with the backing of the White House Office of Science Technology and Policy.

The Department of Energy and the Environmental Protection Agency are joining the consortium.

"This is a collaborative approach," said Lane. "The goal is to work together with research institutions to build a scientific data infrastructure that gathers inputs, outputs and outcomes from a variety of sources in an open a fashion as possible."

Another major aim of STAR METRICS is to reduce, as much as possible, manual reporting by research institutions and principal investigators.

"Researchers currently spend as much as 42 percent of their time in administrative reporting," said Lane. "That's not a good use of their time."

Lane and Bertuzzi say they hope the active engagement of the federal science policy community through STAR METRICS will help ensure scientific advances in science measurement.

Provided by National Science Foundation