Realistic Evaluation and Communication of Indications, Conclusions, and Uncertainty from Analysis of Government Data

• Activity Number: 44 - Tukey (1962) and the Subsequent Sixty Years of Data Analysis: Perspectives from Government and Social Science Applications
• Type: Invited
• Date/Time: Sunday, August 7, 2022 : 4:00 PM to 5:50 PM
• Sponsor: Government Statistics Section
• Author(s): John L. Eltinge*
• Companies: U.S. Census Bureau
• Keywords: Design of Statistical Information Systems; Foundations for Evidence-Based Policymaking Act of 2018; Inference; Integration of Multiple Data Sources; Quantifiable and Inherently Qualitative Components of Data Quality; Wicked Problems

Abstract

This paper explores concepts and techniques from literature on data analysis and data science that are especially important for realistic responses to current government needs for high-quality statistical information. That exploration directs special attention to three ideas highlighted in Tukey (1962) and subsequent literature. First, practical designs of large-scale statistical information production and analysis procedures rely heavily on optimization concepts, but also are subject to substantial levels of uncertainty. Second, realistic interpretations of analysis results depend heavily on the framing of the underlying scientific and policy questions; and require carefully nuanced distinctions between “indications” and “conclusions.” Third, public communication and application of analytic results will occur within operational contexts that vary markedly across time and societal settings, and that can depend on expectations regarding scientific and institutional integrity and credibility. These three concepts are illustrated with cases from the integration of multiple data sources and from recent work in response to the Foundations for Evidence-Based Policymaking Act of 2018.