Daniel Borup, Philippe Goulet Coulombe, David E. Rapach, Erik Christian Montes Schütte, and Sander Schwenk-Nebbe
Working Paper 2022-16c
November 2022 (Revised September 2024)
:: Abstract:
We introduce the performance-based Shapley value (PBSV) to measure the contributions made by each of the individual predictors in fitted time-series forecasting models to the out-of-sample loss. The PBSVs for the individual predictors sum to the out-of-sample loss, so our new metric produces an exact decomposition of out-of-sample performance. In essence, the PBSV anatomizes out-of-sample forecasting accuracy, thereby providing valuable information to decision makers for interpreting fitted time-series forecasting models. The PBSV is model agnostic—so it can be applied to any fitted prediction model, including “black box” models in machine learning—and it can be used for any loss function. We also develop the TS-Shapley VI, a version of the conventional Shapley value that gauges the importance of predictors for explaining the in-sample predictions in the entire sequence of fitted prediction models that generates the time series of out-of-sample forecasts. We then propose the model accordance score to compare predictor ranks based on the TS-Shapley-VI and PBSV, thereby linking predictors' in-sample importance to their contributions to out-of-sample forecasting accuracy. We illustrate our new metrics in an application forecasting US inflation using a variety of machine-learning models and a large number of predictors.
JEL classification: C22, C45, C52, C53, E31, E37
Key words: model interpretation, machine learning, time-series data, Shapley value, loss function, inflation
https://doi.org/10.29338/wp2022-16
The authors thank seminar and conference participants at the European Commission Joint Research Center: Online Seminar, 2022 International Symposium on Forecasting, the Workshop on Advances in Alternative Data and Machine Learning for Macroeconomics and Finance, the Federal Reserve Bank of Atlanta, the 12th European Central Bank Conference on Forecasting Techniques, and the International Association for Applied Econometrics 2023 Conference, as well as Daniele Bianchi, Giulio Caperna, Todd Clark, Marco Colagrossi, Jonas N. Eriksen, Claudia Foroni (Workshop on Advances in Alternative Data and Machine Learning discussant), Nikolay Gospodinov, Andreas Joseph, Juri Marcucci, Michael McCracken, Marcelo Medeiros, Stig Møller, Mirco Rubin, and Michel van der Wel (ECB Conference on Forecasting Techniques discussant), for insightful comments. The views expressed here are those of the authors and not necessarily those of the Federal Reserve Bank of Atlanta or the Federal Reserve System. Any remaining errors are the authors' responsibility.
Daniel Borup is with Aarhus University. Philippe Goulet Coulombe is with the Université du Québec à,À Montréal. David E. Rapach is with the Federal Reserve Bank of Atlanta. Erik Christian Montes Schütte is with Aarhus University and DFI. Sander Schwenk-Nebbe is with Aarhus University. Please address questions regarding content to David E. Rapach (corresponding author), Research Department, Federal Reserve Bank of Atlanta, 1000 Peachtree Street NE, Atlanta, GA 30309.
To receive e-mail notifications about new papers, subscribe. Under "Publications" select "Working Papers."