Detection and Treatment of Careless Responses in Survey Research

• Activity Number: 165
• Type: Topic Contributed
• Date/Time: Monday, August 4, 2014 : 10:30 AM to 12:20 PM
• Sponsor: Committee on Applied Statisticians
• Author(s): Ying Cheng*+ and Jeffrey Patton
• Companies: and University of Notre Dame
• Keywords: careless responses ; survey research ; item response theory ; fit statistics

Abstract

In psychological and survey research, the prevalence of careless responses from unmotivated participants has been repeatedly reported. Datasets contaminated by such responses may lead to serious consequences. In psychological and survey research, these consequences include low scale reliability, attenuated effects between predictor and outcome variables, and erroneous conclusions from hypothesis testing. In educational testing, researchers have reported biased item parameter estimates. When item parameter estimates are biased, the resulting ability estimates and item/test information functions are affected.

This study proposes two methods to detect and treat careless responses, in the hope of improving item parameter estimation. The first method iteratively cleanse the calibration sample based on the person-fit index. The second method is robust calibration based on Mahalanobis distance (MHD). A series of simulation studies are conducted to evaluate the performance of the two proposed methods. The first method was found to be very effective. The second method requires careful fine-tuning of the weights. Overall, the iterative cleansing method is recommended in survey research.