Absolute and Relative Measures for Evaluating the Forecasting Performance of Time-series Models

• Activity Number: 308
• Type: Contributed
• Date/Time: Wednesday, August 11, 2004 : 8:30 AM to 10:20 AM
• Sponsor: Business and Economics Statistics Section
• Author(s): Tessema Astatkie*+
• Companies: Nova Scotia Agricultural College
• Keywords: nonlinear time series ; model validation ; coefficient of efficiency ; index of agreement

Abstract

Mean squared error (MSE) and mean absolute error (MAE) of prediction are widely used measures for evaluating the forecasting performance of time-series models. Although these absolute measures are good to compare alternative models, they do not indicate how well the models forecast the true values. Two relative measures, namely coefficient of efficiency (E) and index of agreement (d) are discussed. Also, their modified versions, known as baseline-modified coefficient of efficiency (EM) and baseline-modified index of agreement (dM) are proposed. The values of E and EM range from minus infinity to 1, while those of d and dM range from 0 to 1. For both measures, values closer to 1 are preferred. These measures are illustrated using the benchmark Jokulsa Eystri daily River flow data. The out-of-sample forecasts of one to five days in 1974 by the NAARX, NeTAR and MNITF models, calibrated using the 1972-73 data, are compared in terms of MSE, MAE, E, EM, d, and dM. NeTAR described the system best, and gave better one- and two-day ahead forecasts. MNITF was better for three-day, all three were comparable for four-day, and NAARX and NeTAR were equally better for five-day-ahead forecasts.