Detection of safety signals from routinely collected adverse event data in clinical trials is critical in drug development. How to deal with the multiplicity issue and rare adverse event (AE) data in such a setting is a challenging statistical problem. Without multiplicity considerations, there is a potential for an excess of false positive signals. On the other hand, traditional ways of adjusting for multiplicity often fail to flag important signals. Bayesian hierarchical modeling is appealing in dealing with this problem. In this presentation, we illustrate the use of Bayesian hierarchical binomial and Poisson models for binary and subject-year adjusted outcomes, respectively by explicitly modeling the existing AE coding structure. We further extend the work by Berry and Berry [2004] by providing guidance on how to choose the signal detection threshold in an effort to balance the type I and type II error rates. We also show some effective graphics for displaying flagged signals when analyzing hundreds or thousands of AEs.