5 minutes
Study assesses accuracy and complexity of techniques for estimating loan loss reserves.
New accounting rules for estimating loan loss reserves from the Financial Accounting Standards Board—the “current expected credit loss model” or CECL—offer general guidelines and a list of implementation possibilities, but no specific implementation recommendations.
A study sponsored by CUES Supplier member Allied Solutions, Plano, Texas, the National Association of Federally-Insured Credit Unions and OnApproach aims to assess which options have the largest impact on the final loss reserve calculation.* The study analyzes a large mortgage data set from Fannie Mae and Freddie Mac. The results quantify the pros and cons of the options for implementing the new CECL guidelines for 30-year fixed rate conforming mortgages.
Study Design
The study tested a broad range of modeling techniques: time series correlations to macroeconomic data, roll rate models, vintage models, state transition models and discrete time survival models. These models were assessed for accuracy, robustness to small data sizes, complexity, computation time, and procyclicality of lifetime loss estimates. (Procyclicality here refers to variability through the economic cycle.)
In all cases, scenarios were created with 24-month macroeconomic history followed by mean-reversion to long-run macroeconomic conditions. Undoubtedly, many practitioners will create two separate models, a near-term model with a macroeconomic scenario and a long-run through-the-cycle loss model. Using a single model with a mean-reverting macroeconomic scenario is preferable, because the active portfolio is used for the lifetime loss forecast rather than an average of past portfolios. It also avoids the necessity to validate two separate models.
The CECL guidelines mention the option of using a discounted cash flow approach. DCF is not a model so much as a system of equations for aggregation, since it requires estimates of default and attrition probabilities as estimated in the models tested in this study. Therefore, all model results were shown as direct loss aggregation, discounted loss aggregation, and DCF aggregation of cash flows simulated from the loss estimation models.
Results
The CECL guidelines suggest that many model types are potentially applicable for loss estimation. These models were tested for suitability: time series models of default and pay-down, roll rates, vintage models, state transition models, and discrete time survival models.
Accuracy
Projecting losses via time series models of default and pay-down rates produced an average 3-year cumulative error rate of 17 to 19 percent. In itself, these results will raise concerns with validators, but the accuracy is unchanging relative to the amount of training data, which can be useful for very small or noisy data sets. Vintage models were consistently high performers in terms of accuracy with 1 to 3 percent error rates. Discrete time survival models and state transition models both perform well (6.5 to 7.5 percent), but not better than vintage models, showing that loan-level modeling does not guarantee more accuracy. Vintage, state transition, and survival models all had similar scaling properties versus size of training data. Roll rate models were consistently the worst performers at 15 to 20 percent error rates. Averages of historic loss rates are unsuited to lifetime loss forecasting at 60-plus percent error rates. Overall, roll rate and historic average models should not be used for long-lived products.
Creating separate models by U.S. state did not provide greater accuracy when compared to a single national model of the same portfolio. Geographic segmentation provides advantages in business application, but not model accuracy.
The guidelines indicate that vintage modeling is not a requirement. If we assume that “vintage model” refers to any approach that adjusts credit risk and prepayment risk based on the age of the loan, then the results show significant increases in accuracy for techniques incorporating this (vintage models, state-transition models and discrete time survival models) as compared to those that do not include it (time series and roll rates).
Accuracy vs. Complexity
The loan-level models (state transition and survival) were by far the most complex in terms of numbers of coefficients and computational time. This complexity did not provide any increased accuracy relative to vintage models, but it does provide business value in account management, collections, pricing and strategic planning.
The added complexity of roll rate models when compared to time series models provided little benefit other than the chance to be more accurate for the first six months of the forecast. Vintage models were the overall winners in the accuracy versus complexity trade-off, so long as sufficient data exists for robust estimation.
Optional DCF
Discounting using a time-value-of-money concept is an option under CECL. Directly discounting of the projected monthly losses using the par rate on the mortgage results in a 20 to 30 percent decrease in the reserve amount. Estimating the principal and interest payments adjusted for the risk of default or prepayment from the loss model and then discounting with the par rate on the mortgage results in the same 20 to 30 percent reduction in the loss reserve as compared to the original lifetime loss forecast.
Old vs. New Rules
The magnitude of the change from the old loan loss rules to CECL will depend strongly on the lifetime of the asset and the point in the economic cycle when the adoption of CECL occurs. For 30-year fixed mortgages, the average life of loan is about 5.5 years, and the lifetime loss reserve will be 4 times a historic average approach with a 24-month loss emergence period. If adoption had occurred just before the onset of the last recession, the adjustment would have been a 10x increase. At the peak of the recession, the change would have been a 2x increase. Well into recovery, it would have been at parity.
Conclusion
By design, the new CECL rules provide a significant amount of flexibility in implementation. As seen from this study, even with a straightforward product like 30-year, fixed rate, conforming mortgages, the range of models listed in the CECL guidelines and the option of including discounted cash flows can produce a range of lifetime loss estimates that vary by more than a factor of 2.
Being able to choose options that will create such different answers will put the burden on lenders not only to choose the most appropriate models for their portfolios, but in doing so to also choose the level of loss via the models chosen and to defend that choice to validators, auditors and examiners.
*Any opinions expressed in this article are solely those of the author and may not represent the opinions of the sponsors.
Joseph L. Breeden, Ph.D., is the principal investigator for this study and the founder and CEO of Prescient Models, LLC, Santa Fe, New Mexico.
