Diversity in a Sample and Large Number of Small Probabilities: The Case of Questionnaires

• Activity Number: 210
• Type: Invited
• Date/Time: Monday, August 2, 2010 : 2:00 PM to 3:50 PM
• Sponsor: Section on Nonparametric Statistics
• Author(s): Estate V. Khmaladze*+
• Companies: Victoria University of Wellington
• Keywords: Karlin-Rouault law ; probabilities of unseen events ; Good-Turing indices ; Zipf's law

Abstract

Consider a questionnaire with $q$ binary questions, which is filled by $N$ individuals, thus providing $N$ ``opinions" $\overrightarrow x = \{x_i\}_{i=1}^q$, with each $x_i$ being $0$ or $1$. For $q$ large, most probabilities $p(\overrightarrow x)$ are small and many of them are very small. However, how many how small probabilities do we have typically is not quite arbitrary and under general assumptions follows the asymptotics $$ \sum_{\overrightarrow x} {\mathbb I}_{\{2^q p(\overrightarrow x)>z\}} \sim c_q z^{-u}, \; {\rm as} \: q\to\infty $$ with specified $c_q$. If $\mu_q$ and $\mu_q(k)$ denote the number of different opinions and the number of opinions occurring $k$ times in the sample, we show that \eqref{inv} is equivalent to the convergence $$ \frac{\mu_q(k)}{\mu_q}\to \frac {u\Gamma(k-u)}{\Gamma(k+1)\Gamma(1-u)} $$