The Survey of Non-Medical Use of Prescription Drugs Program (NMURx) is a system that collects information on drug use and abuse using an online survey instrument. Validity of responses is often a concern for anonymous online surveys, and NMURx is further challenged by collecting information on rare outcomes related to behaviors that may encounter social stigmas. To improve data quality and key outcome estimates, criteria that excluded 0.5% of all respondents for providing careless responses were applied to NMURx data collected in 2018. After consideration of several literature-based and data-driven methods, criteria were based on time spent on demographic questions, response pattern indices, even-odd consistency, and outlier analysis of the total number of drug products endorsed. In certain low volume drug groups, over 20% of endorsements came from careless respondents. 47.1% of the careless respondents provided contradictory answers, and the median time spent on drug use questions was significantly lower among careless respondents (p < 0.0001). These internally consistent results indicate that the criteria identified diverse behaviors in respondents that should be removed.