What to Do About Observational Studies
Keywords: Multiple testing, Bias, Multiple model building,
It has been variously estimated that claims coming from observational studies are undependable. There is a need to systematically evaluate how often claims replicate when tested again. Observational claims tested in randomized clinical trials are evaluated, 20 papers and 86 claims. It is found that over 90% of such claims fail to replicate. Technical reasons for this failure to replicate are presented. Some policy implications of this failure to replicate are discussed.
is clustered and treatment differences within clusters are graphically displayed to evaluate the study. These differences can be related to demographic characteristics of the clusters. An example data set of ~86k records is used to explicate the method.