A Multi-Resolution Theory for Approximating Infinite-P-Zero-N: Transitional Inference, Individualized Predictions, and a World Without Bias-Variance Trade-Off

• Activity Number: 38 - Inference, Optimization, and Computation on Discrete Structures
• Type: Invited
• Date/Time: Sunday, August 8, 2021 : 3:30 PM to 5:20 PM
• Sponsor: IMS
• Author(s): Xinran Li and Xiao-Li Meng*
• Companies: University of Illinois and Harvard University
• Keywords: Double descent; Machine Learning; Multiple descents; Personalized medicine; Sieve methods; Sparsity

Abstract

Transitional inference is an empiricism concept, rooted and practiced in clinical medicine since ancient Greece. Knowledge and experiences gained from treating one entity are applied to treat a related but distinctively different one. The uniqueness of entities is the result of an infinite number of attributes (p), which entails zero direct training sample size (n) because genuine guinea pigs do not exist. Wavelets and sieve methods suggest an multi-resolution (MR) theory for transitional inference, where we use the resolution level to index the degree of approximation to ultimate individuality (Meng, 2014). MR inference relies on an infinite-term ANOVA-type decomposition, providing an alternative way to model sparsity via the decay rate of resolution bias. Unexpectedly, this decomposition reveals a world without variance when the outcome is a deterministic function of infinitely many predictors. In this deterministic world, the optimal resolution can prefer over-fitting. There can also be double or many ``descents'' in the prediction error curve, when predictors are inhomogeneous and the ordering of their importance does not align with the order of their inclusion in prediction.