All Times EDT
An important aspect of ensuring the long-term safety of a machine learning model is understanding the changes in the model’s environment of use that can affect its performance. These changes can arise for several reasons, including shifts in the underlying patient population, changes in technology used to acquire the input data, and changes in the behavior of clinician users of the model. Thus, the ability to isolate specific shifts that harm a model’s performance helps enable targeted surveillance of a variety of factors relevant to the model’s real-world performance. In this talk I will discuss how different types of changes in the environment can be modeled as dataset shifts (i.e., changes in the data distribution). Using this dataset shift formulation, I will then describe an evaluation framework for performing data-driven discovery of shifts that would harm a model’s performance and will discuss how this can improve model monitoring protocols.