Robustness in Metaanalysis: An Empirical Comparison of Point and Interval Estimates of Standardized Mean Differences and Cliff's Delta

• Activity Number: 330
• Type: Contributed
• Date/Time: Tuesday, August 9, 2005 : 2:00 PM to 3:50 PM
• Sponsor: Social Statistics Section
• Author(s): Jeffrey Kromrey and Kristine Hogarty*+ and John Ferron and Constance Hines and Melinda Hess
• Companies: University of South Florida and University of South Florida and University of South Florida and University of South Florida and University of South Florida
• Keywords: meta-analysis ; robustness ; effect size ; Monte Carlo study

Abstract

With the growing popularity of metaanalytic techniques to analyze and synthesize results across sets of empirical studies have come concerns about the sensitivity of traditional tests in metaanalysis to violations of assumptions. This is particularly distressing as the tenability of such assumptions in primary studies is often impossible to evaluate unless sufficient details are reported. Robust estimates of effect size, such as Cliff's Delta, may yield superior inferences about the population effect size. The purpose of this study was to compare standardized mean differences (Cohen's d and Hedges' g) and Delta in terms of the accuracy and precision of interval estimates of population mean effect size in metaanalysis. Factors investigated in the Monte Carlo study included characteristics of both the populations from which samples were drawn (i.e., distribution shape, variance heterogeneity, and population effect size) and the corpus of studies in each metaanalysis (i.e., sample size and number of studies). Although d and g provided relatively unbiased estimates when the assumptions were met, inferences with Delta evidenced less bias under nonnormality and variance heterogeneity.