Algorithmic decision making impacts our lives in mundane and consequential ways. From movie recommendations to medical diagnoses, practitioners make use of a myriad of algorithms at times without delving into the “why”. The reasoning behind the use of any ML algorithm in these applications usually boils down to its superior performance compared to others. Such algorithms, however, often fail to provide an interpretable understanding of the model. Recently, rule-based algorithms have been used to address this. There, an 'or-of-ands' based classification technique (Wang et al., 2015) is used that allows for rule mining of a single class in a binary classification. In this work, we extend this idea to provide classification rules for both classes simultaneously. We also present a novel and complete taxonomy of classifications that clearly quantify the inherent ambiguity in noisy binary problems in the real world. We show that it leads to a granular formulation of the likelihood and a simulated-annealing based optimization achieves performance comparable to the state of the art. We apply our method to synthetic as well as real world data that demonstrate the utility of our proposal.