Please enable JavaScript to view the comments powered by Disqus.

We use cookies on our website to give you the best online experience. Please know that if you continue to browse on our site, you agree to this use. You can always block or disable cookies using your browser settings. To find out more, please review our privacy policy.

About


Policy Hub: Macroblog provides concise commentary and analysis on economic topics including monetary policy, macroeconomic developments, inflation, labor economics, and financial issues for a broad audience.

Authors for Policy Hub: Macroblog are Dave Altig, John Robertson, and other Atlanta Fed economists and researchers.

Comment Standards:
Comments are moderated and will not appear until the moderator has approved them.

Please submit appropriate comments. Inappropriate comments include content that is abusive, harassing, or threatening; obscene, vulgar, or profane; an attack of a personal nature; or overtly political.

In addition, no off-topic remarks or spam is permitted.

December 5, 2016

Using Judgment in Forecasting: Does It Matter?

​​

Many professional forecasters use statistical models when making their near-term projections for real gross domestic product (GDP) growth. A 2013 special survey on the forecasting methods of the Survey of Professional Forecasters  found that 18 out of 21 respondents featured a statistical model prominently in their current-quarter economic projections. Nevertheless, there is fairly compelling evidence that many professional forecasters incorporate judgment in their forecasts of the first estimate of real GDP growth for a quarter—even when much of the source data used to construct the GDP estimate are available.

In the October 2016 Wall Street Journal Economic Forecasting Survey (WSJ), the most common panelist projection for annualized third-quarter real GDP growth was 2.5 percent, and the second most common one was 3.0 percent. The first digit after the decimal point, or tenths digit, of these two numbers are "5" and "0." Of the 58 individual forecasts of third-quarter growth in the survey, 21 had a tenths digit of "0" or "5," a total that is almost twice as large as we would expect if all tenths digits were equally likely to be submitted.

This pattern isn't unique to the most recent quarter's GDP forecast. The following chart shows the historical frequency of the tenths digit in past WSJ surveys for first estimates of real GDP growth over the period from the first quarter of 2003 to the third quarter of 2016, made about three weeks before the release.

Almost 40 percent of these 2,390 forecasts have a tenths digit of "0" or "5." In contrast, the historical distribution of published first estimates of real GDP growth from the fourth quarter of 1991 to the third quarter of 2016 and real gross national product (the most common measure of U.S. production in an earlier era) growth from the third quarter of 1965 to the third quarter of 1991 has a tenths digit of either "0" or "5" only 18 percent of the time. The historical Atlanta Fed's GDPNow forecasts have a "0" or a "5" tenths digit only 15 percent of the time.

More formally, one easily can reject the hypothesis at the 1 percent significance level that the tenths digit of the WSJ panelist forecasts are either uniformly distributed or follow the Benford distribution for tenths digits after rounding to the nearest tenth (see this paper by economists Stefan Gunnel and Karl-Heinz Todter, who found similar relative frequencies of "0s" and "5s" in professional forecasts of German GDP growth and consumer price index inflation).

If we assume that near-term GDP growth forecasts with a tenths digit of "0" or "5" typically involve more judgment than forecasts with another tenths digit, a natural question is whether these more judgmental forecasts are less accurate than others. Of the 2,390 WSJ growth forecasts mentioned above, the ones with a tenths digit of "0" or "5" (after rounding to the nearest tenth) had an average error of 0.786 percentage points without regard to sign, and the others had an average error of 0.743 percentage points. These accuracy metrics are not statistically different at even the 10 percent significance level. Moreover, because of the panel nature of WSJ forecasts, we can measure how often a forecaster has a tenths digit of "0" or "5" (after rounding). Of the 44 panelists who submitted at least 30 three-week-ahead GDP forecasts during the period of the first quarter of 2003 through the third quarter of 2016, the correlation of the panelists "0" or "5" tenth digit frequency and their average error without regard to sign is only 0.13 and not significantly different from 0.

Although at least some professional forecasters appear to make judgmental adjustments to their near-term GDP projections, the evidence presented here does not suggest it comes, on average, at the cost of accuracy.