Understanding CECL parallel runs and model validations
For several years now, Current Expected Credit Loss (CECL) implementation has been on the minds of financial institutions across the United States. Yet, many financial institutions are unsure about where their CECL journey should be heading as we look deeper into the rest of this year – and beyond.
There are many segments in the CECL implementation process but some of the key areas include:
- Committee assembly: Building a CECL committee that represents all aspects of the institution
- Current and historical data analysis: Determining whether the institution has the necessary data to run each of the various model options
- Model selection: Understanding what methods are available, and the pros and cons of each
- Assumption and forecasting development: Developing assumptions that range from simple to complex, depending on what model the institution has selected
- Parallel runs and stress-testing: The process of testing the model in order to weed out potential issues and errors
Many financial institutions are still at Step 1 or Step 2 of their implementation journey, but it is never too early to understand the entire process, including the benefits involved with parallel runs and stress-testing.
Model validation framework
Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. An effective validation helps ensure that models are sound. All model components, including input, processing, and reporting, should be subject to validation.
With this in mind, the idea of model risk is becoming more important. Model risk, in essence, is the potential for adverse consequences from decisions based on incorrect or misuse of model outputs and reports. It can lead to financial loss, poor business and strategic decision-making and reputation damage.
An effective validation framework should include three core elements:
- Evaluation of conceptual soundness, including developmental evidence: Assessing the quality of model design and construction
- Ongoing monitoring, including process verification and benchmarking: Confirms that the model is appropriately implemented and is being used and is performing as intended
- Outcomes analysis, including back-testing: Comparison of model outputs to corresponding actual outcomes
We also suggest approaching the validation framework in five phases, as noted below:
Phase 1:
Model governance and compliance is often missed when financial institutions perform model validations.
Currently, with the new CECL standard requiring complex methodologies, support, and documentation during every step of the process, it is more critical than ever to perform the following functions as part of a model governance and compliance review:
- Policy review
- Evaluation of internal controls
- Model documentation
- Roles and responsibilities
- Ongoing monitoring
An appropriate governance program should include board and management oversight with updated policies and procedure and defined roles and responsibilities. The program should assess the performance of the model on an ongoing basis and should clearly state the model documentation and validation standards that are to be upheld.
Phase 2:
CECL models require clean, accurate model data inputs to ensure meaningful results. The approach to this phase should focus on the following areas:
- Review of loan data
- Historical data review
- Segmentation of loan pools (including credit quality characteristic analysis)
- In particular, it is critical to be confident in the completeness and accuracy of the historical data. Inputting data that is questionable or improperly vetted is ultimately a waste of time and the start of a futile process.
Phase 3:
Model assumptions and back testing stresses the importance of incorporating forward-looking data into the calculation of expected losses over the life of a loan. Forecasted economic conditions and voluntary and involuntary (default) prepayment rates are the main drivers of an institution’s model assumption set.
As for the approach to model assumptions, we suggest validating four primary areas:
- Economic forecast assumptions
- Default rates
- Prepayment speeds
- Discounting methodology
Qualitative factors are another key assumption that will impact an institution’s CECL estimate outside of the modeling. Typically, qualitative factors are included after the modeling is complete and the organization adjusts its historical estimate based on a variety of situations. Some of the major factors consist of changes in lending policies and procedures, changes in economic and business conditions, changes in past-due loans, and changes in the value of underlying collateral to name a few. Since these factors are typically applied outside of the CECL modeling, many validations overlook this area. However, these factors will have varying impacts to an institution’s final CECL estimate based on their region, organization structure, and historical loss experience.
Additionally, many institutions are bringing over their previous qualitative factor methodology and applying it to their CECL methodology. However, institution’s should be careful to exclude certain factors that may be embedded in the historical loss and forecasting calculations. For example, many institutions may apply an unemployment or housing value forecast to their life of loan calculation. Therefore, any adjustments to economic or business conditions applied after modeling as a qualitative factor may need to be removed from the process.
Phase 4:
Model methodology and testing is, in our perspective, the most crucial part of a valuable validation implementation. There are two types of validations – standard, which is a basic testing of instruments and possibly a few shadow calculations; and replication, which requires obtaining all CECL sets and assumptions used by the institution using those data sets and assumptions to independently model a CECL estimate. and then comparing the two results, category by category.
The basic approach to model methodology and testing entails:
- Review of model framework
- Full replication of model
- Category-level reviews
- Assumption testing
- Scenario and stress testing analyses
- Back-testing
The types of findings that are commonly produced through a replication approach to model validations g are user errors, process errors, data errors and methodology differences. By replicating the model, we can uncover these errors and help resolve the issues to enhance the model for future iterations.
Additionally, the model validation offers a variety of replication benefits and value, such as giving the client full assurance on “black box” calculations and allowing the client to learn more about how assumptions and methodology impact their profile. From an internal perspective, the approach allows an institution to validate all parts of the process from start to finish. Not to mention, it helps ensure more data quality control, as noted earlier.
Phase 5:
Once an institution’s model process is set, we highly recommend performing stress tests and scenario analyses prior to implementation. These steps help gauge the impacts of movement in assumptions, such as increases and decreases in prepayment speeds, higher and lower severities, and increases and decreases in discount rates.
When running scenarios, we believe it is best practice to include a worst-case scenario (or “break the bank”) . Understanding the scenarios and the forecasting metrics will help an institution prepare not only for all real-life scenarios, but also for all types of questions from regulators, auditors, and other interested parties. Plus, the data gained from these scenario analyses can aid with capital and budgeting discussions down the road.
