Macroblog

May 19, 2022

An Evaluation of GDP Nowcasts during the Pandemic

On April 28, the US Bureau of Economic Analysis reported that real gross domestic product (GDP) contracted an annualized rate of 1.4 percent last quarter. This decline "surprised" GDPNow, the Atlanta Fed's GDP tracking model, which had projected a 0.4 percentage point growth rate the day before the official release. Professional forecasters, who generally expected a rate around 1 percent based on economist surveys from Reuters and the Wall Street Journal (WSJ), also turned out to be overly optimistic.

The lines in chart 1 represent errors for final forecasts of first-release estimates of real GDP growth from the WSJ Economic Forecasting Survey and GDPNow. In the five-and-a-half years before the pandemic, the WSJ survey and GDPNow both had average absolute forecast errors—that is, without regard to sign, or mean absolute errors (MAEs)—of about 0.5 percentage points. Since then, the MAEs have been 1.7 percent and 1.9 percent, respectively. The stacked bars in the chart represent a decomposition of the GDPNow line into forecast errors of subcomponent contributions to GDP growth. The chart makes it evident that the size of the bars has fallen since 2020, but continues to be larger than before the pandemic. Even GDPNow's relatively accurate forecasts for the second and fourth quarters of 2021 were largely the result of the positive and negative subcomponent errors fortuitously offsetting each other. GDPNow's error last quarter was largely concentrated in net exports. The model's final forecast of the growth rate of real final sales to private domestic purchasers—which excludes inventories and government spending in addition to net exports and has been shown to be a better leading indicator of one-quarter-ahead GDP growth than GDP growth itself is—was only 0.3 percentage points below the initial estimate of 3.7 percent.

The deterioration in GDP forecasting accuracy during the pandemic has not been isolated to the shortest horizon projections. Chart 2 shows MAEs for roughly 75-day-ahead forecasts of growth rates of real GDP and its subcomponents (as well as net exports and inventory investment contributions to growth) from both GDPNow and the Philadelphia Fed's Survey of Professional Forecasters (SPF). The decline in forecast accuracy for the SPF has been similar to the decline for GDPNow across subcomponents, with the exception of the state and local (S&L) government spending subcomponent, where the deterioration for GDPNow has been much starker. A Macroblog post from a year ago discussed the reasons for this.

The left-hand panel of chart 3 shows the MAEs for subcomponent contributions to GDP growth prior to the pandemic, and the right-hand panel shows contribution MAEs during the pandemic. Because the errors have been so much larger during the pandemic, the vertical axis is scaled to be six times larger in the right-hand panel than in the left-hand panel.

To show how forecast accuracy has evolved over a typical quarter, the figures in both panels of the charts begin with the MAE after the initial (roughly 90-day ahead) GDPNow forecast, end with the final GDPNow forecast, and use up to five other forecasts following particular data releases in between. Three of these releases are from the Institute of Supply Management's Manufacturing ISM Report On Business (its manufacturing report, specifically) for each of the three months of the quarter being forecasted, and the other two correspond to the personal income and outlays releases from the US Bureau of Economic Analysis for the first two months of the quarter.

Notice that although both panels show the subcomponent projections generally becoming more accurate over time, the rank ordering of the subcomponent accuracy has changed in some nontrivial ways during the pandemic. In particular, even though personal consumption expenditures (PCE) on services account for 45 percent of nominal GDP, it was consistently one of the smaller sources of error prior to the pandemic. But during the pandemic, it has been one of the largest. Government spending was also one of the smaller sources of error prior to the pandemic and has remained that way during the early part of pandemic quarters. But this decade, the government spending forecasts have not tended to become more accurate as quarters have progressed, so that recently, just prior to the GDP release, government spending has been nearly as large a source of error as services PCE.

In the GDPNow model, the forecasts of government spending, particularly S&L government sales to other sectors, were distorted by the large swings in the second and third quarters of 2020. This documentation on recent changes made to the GDPNow model  describes the changes made to keep these types of distortions to a minimum in the future, which is relevant to the data releases used in chart 3. Generally, a few days after the release of personal income and outlays data, the ISM releases its manufacturing data for the subsequent month, providing one of the first data snapshots for that month. The ISM data for this month are used to estimate the model's factor, that then feeds through to the forecasting equations for much of the yet-to-be-released monthly GDP source data.

Chart 4 shows the MAE from the five quarters beginning in the first quarter of 2021 for real GDP growth and the S&L government contribution to GDP growth. The dashed lines are for the version of GDPNow that had been in use at the time, and the solid lines are for the "COVID-adjusted" model in use since the first quarter of 2022. (The same data releases used in chart 3 are used here.) The modified model shows a clear improvement in the forecasts of S&L government spending. Moreover, forecast accuracy no longer deteriorates following the release of the first-month and second-month ISM manufacturing releases. After the second personal income and outlays report of the quarter—and the final one before the initial GDP report—the accuracy and forecast values of the two models became more similar.

Although this is good news for improving the forecast accuracy of GDPNow, it is not clear whether near-term forecasts of GDP growth will continue to be less accurate than they were in the decade leading up to the pandemic. Ongoing geopolitical and economic uncertainties remain elevated, which could continue to have a negative impact on forecast accuracy for some time to come.

By Pat Higgins, a policy adviser and economist, and

Austin McTier, an economist research analyst, both in the Atlanta Fed's Research Department

• May 19, 2022 in
  • Forecasts
  • GDP
• Permalink
• Comments

May 6, 2021

How Has GDPNow Performed during the Pandemic?

According to the U.S. Bureau of Economic Analysis (BEA), real gross domestic product (GDP) grew at an annualized rate of 6.4 percent in the first quarter of 2021, 1.5 percentage points below our final GDPNow model forecast. The size of this forecast error is large relative to GDPNow's prepandemic history but much smaller than the extremely large errors recorded last year. In short, the GDPNow model has struggled to accurately capture the unusual sources of variation in GDP caused by the COVID-19 pandemic and the various policy responses. The colored dashed lines in chart 1 show the evolution of GDPNow's forecast errors for each of the last five quarters. Unlike any of the four quarters of 2020, GDPNow's forecasts of real GDP growth last quarter was always within 5 percentage points of the BEA's first published estimate. Nevertheless, we can see that the GDPNow forecasts for GDP growth last quarter were still less accurate than before the pandemic.

Despite the recent deterioration in GDPNow's accuracy, its forecasts over the past 15 months share some broad similarities with professional forecasts, as chart 2 shows. From mid-March to mid-May 2020, both GDPNow and the consensus Wall Street Journal Economic Forecasting Survey (WSJ) forecasts of real GDP growth declined sharply. As described here, GDPNow did not anticipate the extent of the plunge in services consumption last March and the resulting impact on GDP growth for the first quarter of last year. The nosedive and rebound in economic activity last year were evident in GDPNow's forecasts for GDP growth in the second quarter of 2020. Forecasts for third-quarter GDP growth strengthened during much of the summer and early fall, and GDPNow performed as well or better than the consensus forecast over much of that period. However, for the last two quarters, GDPNow forecasts were more erratic, and generally less accurate, than the consensus.

It's not surprising that GDPNow's accuracy has deteriorated relative to the consensus of professional forecasts. Unlike GDPNow, professional forecasters have been able to use data on personal mobility and government mitigation measures (such as here, here, here, and here), and a number of studies have shown that these measures relate to cross-country differences in GDP growth observed in 2020 (for example, here and here). Transportation services and leisure and hospitality accounted for half the decline in consumer spending in the first two quarters of 2020, and a number of high-frequency measures of activity in these sectors (such as here, here, here, and here) are available before official spending data are. Finally, spikes in consumer spending following the distribution of stimulus payments included in the December and March fiscal packages were evident in some high-frequency measures of consumer spending (available here and here) before they were evident in the retail sales and personal income and outlays releases. In contrast, GDPNow uses a fixed methodology and source data for estimating GDP, making it much less adaptable to unusual circumstances.

Nevertheless, GDP forecasting during the pandemic has been challenging even for professional forecasters. Chart 3 shows the distributions of GDPNow and WSJ survey panelist forecast errors of real GDP growth for projections made about 20 days before the BEA's release of its initial GDP estimate. The gray cross marks represent individual WSJ panelist errors, and the blue rectangles represent interquartile ranges (IQRs), or middle halves, of the distribution of their forecast errors. The IQRs widened during the pandemic, and a number of panelists had errors in the middle two quarters of last year that were dramatically larger than usual.

Still, it is plausible that the extreme fluctuations in the data used by the GDPNow model have embedded some inaccuracies into the GDPNow forecasts. To examine this question, let's consider GDPNow's final projections of the main subcomponent contributions to overall real GDP growth since the model was released to the public in 2014. Chart 4 shows the stacked forecast errors of these projections. By construction, the sum of the bars in any quarter is approximately equal to the black dot showing the difference between the first official estimate of real GDP growth and that quarter's final GDPNow forecast. The bars have become much larger since the pandemic, but especially for services consumption and state and local government expenditures (S&L). In this post, I've noted issues related to services consumption, but most of the huge third-quarter S&L error fell into a category referred to as "sales to other sectors ." GDPNow expected these sales—which are largely health, hospital, tuition, and other educational charges and fees—to continue plummeting in the third quarter instead of rebounding strongly, as they did. (The S&L contribution error is negative because the BEA subtracts these sales from the S&L subcomponent, and from GDP, to avoid double-counting). The S&L contribution errors in the last two quarters were still quite large, and mostly the result of the GDPNow model incorrectly continuing to forecast declines of these "sales to other sectors."

In the coming months, we plan to tweak the architecture behind GDPNow in hope of mitigating these sorts of errors. For instance, the day before the most recent GDP release, a modified version of the GDPNow model (one that reduces the influence of the pandemic on some of its forecasting equations) predicted first-quarter real GDP growth of 5.5 percent. More than half of the difference between this forecast and the official final GDPNow forecast of 7.9 percent growth was the result of a lower forecast for growth in the S&L subcomponent (1.3 percent), a number much closer to the BEA's S&L growth estimate (1.7 percent) than the final GDPNow forecast (14.6 percent). Such enhancements might help improve GDPNow's forecasting accuracy down the line. But GDPNow will remain susceptible to forecast inaccuracies whenever unusual events (such as the COVID-19 pandemic) hit the economy and dramatically shift spending patterns.

By Pat Higgins, a policy adviser and economist in the Atlanta Fed's Research Department

• May 6, 2021 in
  • Forecasts
  • GDP
• Permalink
• Comments

May 12, 2020

Challenges in Nowcasting GDP Growth

Real gross domestic product (GDP) declined at an annualized rate of 4.8 percent in the first quarter, according to the first estimate from the U.S. Bureau of Economic Analysis (BEA), 3.8 percentage points more than the decline anticipated by the Atlanta Fed's final GDPNow model projection. Why was the error, which was easily the model's largest on record for final GDPNow forecasts, so big? Chart 1 looks at GDPNow's forecast errors since the model went live in mid-2014 and breaks them down into forecast errors for the various subcomponents' contributions to GDP growth. 

The clear culprit is the fact that the GDPNow model did not anticipate the record 9.5 percent monthly decline (not annualized) in real services consumption in March. At that time, GDPNow had March services data available only for electricity and natural gas use and purchased meals and beverages, as well as revised February data for net foreign travel. If the model would have correctly forecasted the March growth rate in services consumption for the subcomponents besides these, it would have actually slightly overstated the first-quarter decline in real consumer spending. (I should note that because of the timing and impact of last quarter's social distancing efforts stemming from COVID-19, the BEA used data outside of the scope of its routine procedures to estimate part of services spending in March—in particular, data about private credit card transactions for health care and recreation services.)

By design, GDPNow is a purely model-based prediction method as opposed to the models of some private forecasters, who were able to incorporate developments related to COVID-19 into their April forecasts for first-quarter GDP growth in a way that GDPNow did not and could not. As a result, their GDP predictions turned out to be relatively more accurate. For example, the consensus forecast of first-quarter GDP from the Wall Street Journal Economic Forecasting Survey in the first full week of April was a decline of 3.3 percent, and the CNBC Rapid Update survey late in the week prior to the GDP release anticipated a decline in GDP of 5.3 percent. Private forecasts will continue to be able to use news developments and high-frequency or nonstandard data sources (such as initial unemployment claims and OpenTable restaurant dining data) in a way that GDPNow and similar nowcasting models do not. The New York Fed's recently introduced Weekly Economic Index combines a set of weekly indicators into a single index with units comparable to four-quarter GDP growth, but it does not actually nowcast quarterly GDP growth.

Around the time of recessions, macroeconomic projections from professional forecasters tend to be less accurate and show more dispersion than during nonrecessionary periods. And though the National Bureau of Economic Research has not identified a 2020 business cycle peak, recent GDP forecasts show much more dispersion than they did during, or close to, past recessions. Chart 2 shows the difference between the top 10 and bottom 10 average forecasts of real GDP growth (for both the current quarter and one quarter ahead) in the Blue Chip Economics Indicators survey administered in the middle month of each quarter since 1991.

The top 10/bottom 10 difference for current-quarter GDP forecasts in the May 2020 survey is clearly much larger than around past recessions. In fact, it's larger than the difference between the highest and lowest quarterly growth rate of GDP after 1983, around the time economists date the onset of the Great Moderation—in reference to the decline in macroeconomic volatility—in the mid-1980s. Prior to the May 8 employment release, GDPNow was more optimistic about second-quarter GDP growth than most private forecasters were, but after the model forecast was revised down from a decline of 17.6 percent to a decline of 34.9 percent on the heels of that report, it fell more in line with the others.

The dispersion in the forecasts for GDP in the third quarter of 2020 is even starker. The optimistic forecasters project 2020:Q3 growth to be well above the highest rate on record (15.7 percent in 1950:Q1), and the pessimistic forecasters project contracting GDP. Of course, we will not be able to determine how accurate forecasts of second- and third-quarter GDP growth are until later in the year. Nevertheless, the wide range of forecasts implies that at least some forecasters' GDP projections will be wildly off by historical standards. As St. Louis Fed economist Michael McCracken recently noted, what the late Yankees catcher Yogi Berra said is more true than ever: "It's tough to make predictions, especially about the future." Berra's wisdom also will also apply to producing accurate and reliable economic forecasts for some time to come.

By Pat Higgins, a policy adviser and economist in the Atlanta Fed's research department

• May 12, 2020 in
  • Forecasts
  • GDP
  • Monetary Policy
• Permalink
• Comments

February 13, 2018

GDPNow's Forecast: Why Did It Spike Recently?

If you felt whipsawed by GDPNow recently, it's understandable. On February 1, the Atlanta Fed's GDPNow model estimate of first-quarter real gross domestic product (GDP) growth surged from 4.2 percent to 5.4 percent (annualized rates) after a manufacturing report from the Institute for Supply Management. GDPNow's estimate then fell to 4.0 percent on February 2 after the employment report from the U.S. Bureau of Labor Statistics. GDPNow displayed a similar undulating pattern early in the forecast cycle for fourth-quarter GDP growth.

What accounted for these sawtooth patterns? The answer lies in the treatment of the ISM manufacturing release. To forecast the yet-to-be released monthly GDP source data apart from inventories, GDPNow uses an indicator of growth in economic activity from a statistical model called a dynamic factor model. The factor is estimated from 127 monthly macroeconomic indicators, many of which are used to estimate the Chicago Fed National Activity Index (CFNAI). Indices like these can be helpful for forecasting macroeconomic data, as demonstrated here  and here.

Perhaps not surprisingly, the CFNAI and the GDPNow factor are highly correlated, as the red and blue lines in the chart below indicate. Both indices, which are normalized to have an average of 0 and a standard deviation of 1, are usually lower in recessions than expansions.

A major difference in the indices is how yet-to-be-released values are handled for months in the recent past that have reported values for some, but not all, of the source data. For example, on February 2, January 2018 values had been released for data from the ISM manufacturing and employment reports but not from the industrial production or retail sales reports. The CFNAI is released around the end of each month when about two-thirds of the 85 indicators used to construct it have reported values for the previous month. For the remaining indicators, the Chicago Fed fills in statistical model forecasts for unreported values. In contrast, the GDPNow factor is updated continuously and extended a month after each ISM manufacturing release. On the dates of the ISM releases, around 17 of the 127 indicators GDPNow uses have reported values for the previous month, with six coming from the ISM manufacturing report.

[ Enlarge ]

For months with partially missing data, GDPNow updates its factor with an approach similar to the one used in a 2008 paper by economists Domenico Giannone, Lucrezia Reichlin and David Small. That paper describes a dynamic factor model used to nowcast GDP growth similar to the one that generates the New York Fed's staff nowcast of GDP growth. In the Atlanta Fed's GDPNow factor model, the last month of ISM manufacturing data have large weights when calculating the terminal factor value right after the ISM report. These ISM weights decrease significantly after the employment report, when about 50 of the indicators have reported values for the last month of data.

In the above figure, we see that the January 2018 GDPNow factor reading was 1.37 after the February 1 ISM release, the strongest reading since 1994 and well above either its forecasted value of 0.42 prior to the ISM release or its estimated value of 0.43 after the February 2 employment release. The aforementioned rise and decline in the GDPNow forecast of first-quarter growth is largely a function of the rise and decline in the January 2018 estimates of the dynamic factor.

Although the January 2018 reading of 59.2 for the composite ISM purchasing managers index (PMI) was higher than any reading from 2005 to 2016, it was little different than either a consensus forecast from professional economists (58.8) or the forecast from a simple model (58.9) that uses the strong reading in December 2017 (59.3). Moreover, it was well above the reading the GDPNow dynamic factor model was expecting (54.5).

A possible shortcoming of the GDPNow factor model is that it does not account for the previous month's forecast errors when forecasting the 127 indicators. For example, the predicted composite ISM PMI reading of 54.4 in December 2017 was nearly 5 points lower than the actual value. For this discussion, let's adjust GDPNow's factor model to account for these forecast errors and consider a forecast evaluation period with revised current vintage data after 1999. Then, the average absolute error of the 85–90 day-ahead adjusted model forecasts of GDP growth after ISM manufacturing releases (1.40 percentage points) is lower than the average absolute forecast error on those same dates for the standard version of GDPNow (1.49 percentage points). Moreover, the forecasts using the adjusted factor model are significantly more accurate than the GDPNow forecasts, according to a standard statistical test . If we decide to incorporate adjustments to GDPNow's factor model, we will do so at an initial forecast of quarterly GDP growth and note the change here .

Would the adjustment have made a big difference in the initial first-quarter GDP forecast? The February 1 GDP growth forecast of GDPNow with the adjusted factor model was "only" 4.7 percent. Its current (February 9) forecast of first-quarter GDP growth was the same as the standard version of GDPNow: 4.0 percent. These estimates are still much higher than both the recent trend in GDP growth and the median forecast of 3.0 percent from the Philadelphia Fed's Survey of Professional Forecasters (SPF).

Most of the difference between the GDPNow and SPF forecasts of GDP growth is the result of inventories. GDPNow anticipates inventories will contribute 1.2 percentage points to first-quarter growth, and the median SPF projection implies an inventory contribution of only 0.4 percentage points. It's not unusual to see some disagreement between these inventory forecasts and it wouldn't be surprising if one—or both—of them turn out to be off the mark.

By Pat Higgins, a policy adviser and economist in the Atlanta Fed's research department

• February 13, 2018 in
  • Business Cycles
  • Forecasts
  • GDP
  • Productivity
  • This That and the Other
• Permalink
• Comments