Optimal Adaptive Sequential Design with Application to Crowdsourcing

• Activity Number: 79
• Type: Contributed
• Date/Time: Sunday, August 9, 2015 : 4:00 PM to 5:50 PM
• Sponsor: Social Statistics Section
• Author(s): Xiaoou Li*
• Companies: Columbia University
• Keywords: mastery test ; computerized adaptive testing ; sequential analysis ; adaptive testing ; Rasch model ; stopping rule

Abstract

Commercial crowdsourcing service has gained prominence for obtaining machine learning labels recently. As the crowdsourcing workers are paid for each label they provide, it is desirable to reduce total budget by selecting proper workers adaptively. In this talk, we investigate properties of optimal adaptive sequential designs for worker selection that have minimal Bayes risk. This paper combines several major techniques in statistics and applies them to design optimal crowdsourcing procedure. In particular, we employ techniques in adaptive testing, sequential probability ratio test, stochastic control and empirical Bayes.