|
Abstract:
|
How do we compare two forecasters that make predictions on the future outcomes of a regularly occurring event? In fields ranging from meteorology and economics to epidemiology and sports forecasting, different forecasters make their predictions on an event or quality over time, but their qualities are seldom compared to one another in a statistically rigorous manner. In this work, we derive a sequential inference procedure for the time-varying difference in forecast quality via confidence sequences (CS), which are a sequence of confidence intervals (CI) that have non-asymptotic coverage guarantees at arbitrary stopping times ("anytime-valid"). Importantly, the coverage guarantees for our CSs are also distribution-free, in the sense that they rely on no distributional assumptions (e.g., those of parametric models or stationarity) about how the forecasts and outcomes are generated. We examine the validity of our CSs over their fixed-time and asymptotic counterparts in synthetic experiments and demonstrate their effectiveness in real-data settings, including comparing various forecasts made on the outcome of Major League Baseball (MLB) games.
|
