JSM 2015 Preliminary Program

Online Program Home
My Program

Abstract Details

Activity Number: 487
Type: Contributed
Date/Time: Wednesday, August 12, 2015 : 8:30 AM to 10:20 AM
Sponsor: Section on Statistical Education
Abstract #316872
Title: A Randomized Trial in a Massive Online Open Course Shows People Don't Know What a Statistically Significant Relationship Looks Like, but They Can Learn
Author(s): Aaron Fisher* and G. Brooke Anderson and Roger Peng and Jeff Leek
Companies: The Johns Hopkins University and Colorado State University and The Johns Hopkins University and The Johns Hopkins University
Keywords: Evidenced based data analysis ; p-values ; MOOC ; Randomized trial ; Data visualization ; Statistical significance
Abstract:

Scatterplots are the most common way for data analysts to visually detect relationships between measured variables. At the same time, and despite controversy, P-values remain the most commonly used measure to statistically justify relationships identified between variables. Here we measure the ability to detect statistically significant relationships from scatterplots in a randomized trial of 2,039 students in a statistics massive open online course (MOOC). Each subject was shown a random set of scatterplots and asked to visually determine if the underlying relationships were statistically significant at the P < 0.05 level. Subjects correctly classified only 47.4% (95% CI [45.1%-49.7%]) of statistically significant relationships, and 74.6% (95% CI [72.5%-76.6%]) of non-significant relationships. Classification accuracy in some scenarios improved on repeat attempts of the survey. Our results suggest that data analysts have incorrect intuition about what statistically significant relationships look like, particularly for small effects. We propose that evidence-based data analysis can be used to identify weaknesses in theoretical procedures in the hands of average users.


Authors who are presenting talks have a * after their name.

Back to the full JSM 2015 program





For program information, contact the JSM Registration Department or phone (888) 231-3473.

For Professional Development information, contact the Education Department.

The views expressed here are those of the individual authors and not necessarily those of the JSM sponsors, their officers, or their staff.

2015 JSM Online Program Home