Association of Public Data Users

Science and Technology for America’s Reinvestment Measuring the Effects of Research on Innovation, Competitiveness, and Science (STAR METRICS)

George Chacko, Director of the Office of Planning, Analysis, and Evaluation, Center for Scientific Review
National Institutes of Health
Jim Onken, Special Assistant to the Deputy Director for Extramural Research
National Institutes of Health
Susan Winter, Acting Deputy Director, Office of Cyberinfrastructure
National Science Foundation

STAR METRICS is a consortium of federal agencies in collaboration with research institutions led by the National Institutes of Health (NIH), the National Science Foundation (NSF), and the White House Office of Science and Technology Policy (OSTP). The goal of this program is to provide a better empirical basis for science policy through the development of an enabling data infrastructure, which can be used by federal agencies, research institutions, and researchers to document and analyze federal investments in science. STAR METRICS is organized around two related efforts: Level I- which is focused on collecting data to describe the national scientific workforce supported by federal funds and Level II- which will focus on collecting data to document the output from science investments.

Background: The STAR METRICS program was created in direct response to the reporting requirements associated with the 2009 American Recovery and Reinvestment Act (ARRA), which required recipients of Federal funding to document the initial jobs resulting from Federal expenditures. STAR METRICS Level I was designed to facilitate this reporting. STAR METRICS is also responsive to Open Government memoranda requiring agencies to publish their information online in an open format that can be retrieved, downloaded, and searched and to use modern technology to disseminate useful information1. In addition, the White House Office of Management and Budget (OMB) and OSTP have requested that federal agencies should support the development and use of "science of science policy" tools that can improve management of their R&D portfolios and enable better assessment of the impact of their science, technology and innovation investments2.

The STAR METRICS project was officially approved in October 2010 and now comprises six federal agencies and 86 research institutions. These 86 institutions represent a significant fraction of research expenditure. For example, research institutions participating in STAR METRICS represent 33% of NIH’s grants budget. The six federal participants are the NIH, NSF, OSTP, the Environmental Protection Agency (EPA), the United States Department of Agriculture (USDA), and the Department of Energy (DOE).

In early 2012, NIH was designated as the host institution and a new governance system was put in place to help guide STAR METRICS as it transitions from a pilot to an operational phase. A Lead Entity Executive in consultation with a senior Executive Council and an Interagency Working Group directs the program. Dr. Sally Rockey, Deputy Director for Extramural Research at NIH serves as the Lead Entity Executive and chairs the Executive Council along with Dr. Carl Wieman from OSTP. The NIH and NSF representatives on the Interagency Working Group serve as Program Managers for STAR METRICS.

We are currently trying to document the levels and trends in the scientific workforce supported by federal funding. This goal stemmed from the need to respond to ARRA reporting. The STAR METRICS team developed an automated approach in consultation with key stakeholders including the Federal Demonstration Project (FDP), the Association of American Universities (AAU), and the Association of Public and Land Grant Universities (APLU). The approach taken was to collect de-identified data from administrative systems of records from participating institutions. The data provide information about employees supported by grants from federal agencies, the nature of the positions that these persons occupy, as well as expenditures on vendors, subcontractors, and institutional overhead. This information was not previously available in any standardized form and is limited to 14 data elements drawn directly from institutions’ human resources systems, thus minimizing administrative burden. Eighty six institutions are currently enrolled in Level I.

Immediate plans for Level I are to:

  • Develop, test, and implement a workforce report that describes personnel supported by federal research funds
  • Continue to enroll institutions in STAR METRICS until the consortium includes enough institutions to account for at least 50% of the extramural research funds distributed by NIH, NSF, and USDA
  • Solicit participation of minority serving institutions to ensure that we are able to accurately describe all segments of the workforce
  • Commission a statistical study to determine the enrollment targets, beyond the 50% target, that would permit reliable estimates of the scientific workforce at a national level

The goal of Level II is to build upon the limited Level I data collection to expand the data infrastructure needed to address a broader array of questions concerning the federally supported research enterprise. The overall goal is to develop an open, automated data infrastructure and tools that will enable the documentation and analysis of the inputs, outputs, and outcomes resulting from federal investments in science. The first step in this ambitious undertaking is to aggregate awards information (grants and contracts) from the participating agencies into a common database. When combined with new methods in statistical inference and modeling, these data will enable a greater understanding of the outputs and outcomes of federal science investments.

Comments or questions? Please feel free to contact the author at chackoge@csr.nih.gov.