Target identification is crucial in many defense and national security domains, such as target tracking, surveillance, and gaining situational awareness. In practice, a variety of multi-modal sensors are deployed to support such applications. A critical challenge is to exploit multi-sensor information to robustly identify targets in noisy and imperfect data, such as low contrast imagery under adverse weather conditions. For example, a sensor may identify a set of targets with high confidence, but another sensor may poorly resolve a potential target. The intent is to combine information from all available sensors for robust identification. We tackle the problem by exploiting contextual information based on target coocuurences in different scenarios of interest, such as desert, forest, urban terrain, under-water zones, cultural festivals, etc. In this paper, we develop conditional random field based scenario specific contextual models using past labeled sensor data. Subsequently we use these models to quantify the likelihood of an ill-identified target, and determine the most likely identification of a current low confidence target by probabilistic inference using the random field models.