|
Abstract:
|
A/B testing and multivariate testing have become the bread-and-butter for applications as varied as website optimization, mobile apps and email campaigns. An emerging problem in this area is how to conduct online experiments simultaneously across multiple platforms such as desktops, tablets, smartphones (and maybe wearable gadgets). A simple way is to run tests for different platforms separately. An obvious drawback of this approach is that it does not borrow information from all the sources. To overcome this limitation, we propose a new experimental design framework that allows for optimizing testing for different platforms but also enabling comparing and pooling data across the platforms. Based on a new criterion called sliced effect hierarchy, this framework introduces a "sliced design" idea to partition test units into homogenous slices. The different slices are then used to optimize testing for each platform and the combined design makes it possible to benchmark and correlate testing performance of the different platforms. We demonstrate the effectiveness of the proposed method by using email design application for an online magazine.
|
