Mediation analysis has a long history in the social sciences as a mechanism for providing explanations for observed effects. The goal in a mediation analysis is to decompose the effect of a treatment that is observed into a direct effect of the treatment on the outcome and an indirect effect through a mediator that is itself directly related to the outcome. Recently the causal mediation framework literature has extended the concepts of total, direct, and indirect effects within the context of the potential outcome framework. However, the causal mediation framework, in attempting to quantify effects, downplays the goal of many applied researchers which is primarily to know if there is evidence that putative mediators may be the mechanism for the observed effect. In this paper we develop criteria for a variable to be a potential mediator within the causal mediation framework. We then propose to evaluate these criteria using variable importance measures through procedures that compare the relative loss of models trained using data sets with permutations of key variables. We demonstrate the proposed method using simulated and data from an application.