All Times ET
Keywords: machine learning, classification, regression, mlops, monitoring
Machine learning models provide value to industry through their ability to identify patterns in underlying data and predict future scenarios based on these historical patterns. These models, however, are not always robust to changes external to the system. When these models predict on data that is dissimilar to training data, or when market conditions change the underlying association between dependent and independent variables, predictions become unusable. As machine learning models become integral to business decisions, monitoring of production models becomes a necessary function. When model performance begins to slip, retraining is often necessary, but repeated retraining of offline models can create an unstable environment and lower the end user’s trust in model results. In this session, an automated active monitoring process is reviewed with discussion of multiple real-world case studies. The process identifies shifts in the behavior of submitted behavior versus training behavior, dead and changing prediction path frequencies, and changes in the predictions made. A report card is generated for each model, displaying an overall rate of stability, and surfacing possible causes of model disruption, allowing for faster resolution of model issues. This process allows for generalized monitoring of tree-based regression and classification models in production and is actively being used in production models across a variety of industries.