FTE vs. MRM: Model Risk Management with Minimal Staff

Lean Model Risk Management

Enterprise risk management (ERM) in both the insurance and banking sectors is expected by regulators, management, the board, and other stakeholders to manage all key risks across the organization. In addition to strategic, financial, cyber, and legal risks, ERM is tasked with the management of risk associated with the company’s use of models. Quantitative models are used for decisions in pricing, valuation, stress testing, capital modeling, credit scoring, and underwriting.

ERM is, of course, a large topic on its own and the department is often thinly staffed. Because model risk management (MRM) is an expectation falling under the umbrella of ERM, the already scarce human resources available under the ERM function often face significant challenges in running an MRM program “on top” of the typical risk areas.

This article offers a comprehensive approach for “lean” MRM, i.e., the management of an MRM program with few or possibly one employee devoted to the role. I explore key objectives of an MRM program, essential process elements, tips for efficiency, and examine the use of artificial intelligence (AI) in the models themselves and as a tool to increase efficiency of the program.

Objectives of MRM

MRM aims to ensure that predictive models used in decision-making are accurate, reliable, and used appropriately to minimize financial, operational and reputational risks.

Key objectives include:

• Ensure Model Accuracy and Reliability: Test, validate and monitor models to minimize the risk of errors and erroneous decision-making, safeguarding against financial losses and regulatory penalties.
• Maintain Regulatory Compliance: Ensure all models meet industry and regulatory standards, and support clear documentation for audit purposes. In banking, the Federal Reserve Board’s regulation “SR 11-7” specifies MRM expectations, while in insurance, ORSA indicates that model risk is in the scope of ERM generally, and must be carried out in particular for capital modeling and stress test modeling.
• Mitigate Financial and Operational Risks: Identify and manage risks with model misuse, poor design, or inaccurate data.
• Enhance Transparency and Accountability: Establish clear governance and roles, maintain model inventories, and document model development, validation, and performance for internal and external stakeholders.
• Enable Ongoing Model Monitoring and Adjustment: Continually assess model performance, update documentation, and adjust approaches in response to changes in data, regulatory requirements, or business conditions.
• Strengthen Stakeholder Trust: Independent validation, thorough testing, and transparent communication build confidence among leaders, investors, and regulators in the institution’s risk management practices.

A robust MRM program supports safe, informed decision-making while protecting organizations from avoidable risks and ensuring sustainable business practices.

Essential Program Elements

Here are the primary elements of a model risk program:

• Model Governance
  • Policies and Procedures: Establishing clear, written policies that define what constitutes a model, model risk, and the processes for development, validation, and ongoing monitoring. These documents should also define roles and Responsibilities, e.g., for model developers, validators and users.
  • Oversight: Implementing committees or other governance structures to provide senior-level oversight and ensure the program's effectiveness. A board or management-level committee should receive recurring updates on MRM and be aware of models out of compliance.
• Model Inventory
  • Identification and Definition: Creating a comprehensive list of all models, including those in-house developed and third-party vendor models, to understand what is being used across the organization.
  • Categorization: Tagging models by purpose, user, inputs, assumptions, and other critical features to help in the risk assessment process. Some key fields to include in an inventory are model identification (ID) number, model owner, model validator, data of last validation, and next scheduled validation.
• Model Development and Implementation
  • Sound Methodologies: Ensuring models are built with appropriate methodologies, reliable data, and valid assumptions.
  • Integration and Updates: Planning for the integration of new models into products and processes, as well as managing updates and changes to existing models.
• Model Validation
  • Independent Review: Conducting independent, rigorous testing to verify the conceptual soundness, accuracy and resilience of models. Validation should be a periodic event with higher-risk models being validated more often, e.g., every two years.
  • Testing Methods: Utilizing methods such as backtesting, stress testing and performance benchmarking to assess model behavior under different scenarios.
• Ongoing Monitoring and Performance Tracking
  • Continuous Assessment: Regularly evaluating models after deployment to detect any drift, degradation or inaccuracies.
  • Risk Indicators: Implementing real-time risk indicators to monitor model performance and identify potential issues early.
• Risk Assessment and Mitigation
  • Quantitative and Qualitative Assessment: Performing thorough assessments to determine the level of risk each model poses to the organization.
  • Mitigation Strategies: Developing and implementing strategies to reduce or control the identified risks.
• Documentation and Reporting
  • Comprehensive Records: Maintaining thorough documentation of all aspects of the model lifecycle, including development, validation, assumptions, data and performance.
  • Issue Management: Establishing a process for documenting, tracking and remediating any identified model issues or weaknesses.
• Regulatory Compliance
  • Adherence to Requirements: Ensuring that all model risk management activities comply with relevant industry regulations and requirements.

Use the Model Lifecycle to Drive MRM

The idea of the model lifecycle is useful in several ways. It allows for an intuitive structure that can be easily understood by all the MRM contributors and stakeholders, and helps to track and manage models in an efficient way with minimal resources.

We think of a model from “birth” to “retirement” as follows.

1. Model Planning/Development: Define business objectives and scope, outlining the intended use and identifying potential risks associated with introducing the new model. Determine assumptions, input data, select modeling techniques, calibrate/training algorithms, and document methodology.
2. Pre-validation/Validation: Rigorous internal testing by model owners, followed by independent review of documentation and model logic to confirm adequacy and identify issues. Additional details on validation techniques are found in SR 11-7 and also in guidance from the American Academy of Actuaries. It is important to leverage specific validation techniques for AI and machine learning-based models.
3. Approval/Active Use: Obtain necessary stakeholder and governance sign-offs before production deployment. At this point, the model should be assigned a risk rating such as low, moderate and high. This rating should reflect the likelihood and impact of potential model issues such as faulty data, inputs, methods, outputs, etc. After the MRM team signs off on the model validation, the model is ready for deployment. It should be formally captured in the model inventory.
4. Active Use, Monitoring and Reporting: Continuously monitor performance and use, assess for drift, periodic revalidation, and report status to model risk committees or boards. The MRM policy should define what “material changes” would trigger the need for a new validation. Otherwise, validation should be based on the risk level of the model.
5. Retirement: Decommission or retire models when they are no longer fit for purpose, ensuring data retention and documenting the process for compliance. Reflect the retirement by indicating it in the model inventory.

Using the Model Lifecycle for Project Management

We will track every model’s lifecycle stage at each point in time as shown in Table 1.

Table 1

Model Lifecycle Stages: Definitions and Related MRM Requirements

This information will be captured for all models simultaneously in a single file or database. Having this information at our fingertips provides a straightforward way to manage all tasks relating to MRM and serves as the primary project management tool for MRM.

Project management is simple and intuitive using a table such as Table 2.

Table 2

Using Lifecycle to Manage MRM

When a model moves into a new lifecycle stage, we must ensure we capture that change and note the related document/file needs or action step. So we are able to use the model inventory to see the lifecycle stage of all models on a given day, and this naturally indicates the deliverable or need for each of the models. If the inventory captures responsible parties and formula-driven due dates as well, it becomes an intuitive project management tool for the entire MRM program.

Model Risk Tiering Strategy to Avoid Resource Drain

In many MRM programs, the frequency for re-validation depends on the risk rating of the model under consideration. In some cases, other tasks or the amount of time devoted to a model may also depend on this risk rating. With the goal of lean MRM, the methodology for determining a model’s rating should be: 1) Simple enough to not be time-consuming and not require specialized skill sets, and 2) limit those rated “high” risk to a manageable volume to not create resource challenges.

The model risk rating can be based on an average score across several categories, such as:

• Importance and size of the portfolio, assets, or decisions the model covers (e.g., exposure);
• model complexity and level of automation;
• sensitivity and reliability of data inputs;
• frequency of use and decisions influenced;
• regulatory relevance and potential for compliance breach; and
• customer and reputational impact.

A simple low, moderate and high (e.g., 1, 2, or 3) scale may be used to risk-rate in each category. Such scales should be described in the MRM policy and not be overly burdensome to employ. Then a simple or weighted average may be used to determine the overall risk rating for a model. In the spirit of lean MRM we would add a cap or overlay to this approach. It would cap models at a moderate risk level unless we have both: 1) The calculation yields a high result, and 2) the model’s misuse has the potential to impact a critical amount of company capital, invested assets, or revenue. In other words, if a model cannot create a loss above a pre-defined threshold level, it can never be rated as high risk. Such a convention limits the number of models rated high and also generally reduces the frequency of revalidation and resource usage.

Artificial Intelligence: New Problems and New Solutions

It goes without saying that many models are now based on artificial intelligence (AI) and machine learning approaches, and these models often necessitate specific model validation techniques.

A risk manager may very well not have the knowledge to personally validate such models or even outline the appropriate methods for such validations.

Key validation techniques for machine learning model validation include a variety of data partitioning and statistical methods to assess model performance, prevent overfitting, and ensure generalizability to unseen data and include:

• Train/Test Split: Dividing the dataset into separate training and testing sets to evaluate how well the model generalizes to new, unseen data.
• K-Fold Cross-Validation: The dataset is split into ksubsets (folds); the model is trained on k-1 folds and validated on the remaining fold, repeating this process ktimes and averaging the results for robust performance estimates.
• Stratified Cross-Validation: Ensures that each fold maintains the same class distribution as the original dataset, which is crucial for imbalanced datasets.
• Bootstrap Methods: Resample the dataset with replacement to generate multiple training samples, providing insights into model stability, especially with limited data.
• Holdout Validation: A reserved portion of data is used exclusively for model evaluation, separate from training and initial validation phases.
• Time-Series Cross-Validation: For temporal data, models are evaluated using splits that respect the sequential nature of time to avoid lookahead bias.
• Nested Cross-Validation: Combines an outer loop for robust model evaluation with an inner loop for model tuning, often used for hyperparameter optimization.
• Domain-Specific Validation: Incorporates industry-relevant metrics, external validation datasets, or subject matter expert review, particularly for specialized AI applications.

Additionally, AI itself may come to the rescue and there are many online tools available to actually carry out such validations, such as:

• Google Colab: Provides access to downloadable model risk management and validation scripts, allowing users to validate, monitor, and report on model performance in a collaborative and cloud-based environment.
• Databricks Solution Accelerator: Offers pre-built code samples, data, and step-by-step validation instructions tailored for financial services. Users can audit, document, visualize, and test models, including machine learning models relevant for model risk management.
• Galileo: Galileo is an advanced AI model validation platform designed to streamline and enhance the validation, monitoring, and analysis of machine learning and large language models (LLMs).

Conclusion

With the continued evolution of financial modeling, machine learning, and the nimble FinTech business model, challenges can be expected on the MRM front, both in terms of practical details and regulatory requirements. Lean MRM is an approach that provides high return on investment.

As AI continues to expand in terms of computational power, accuracy, and widespread industry acceptance, MRM’s importance can only be expected to grow.

This article is provided for informational and educational purposes only. Neither the Society of Actuaries nor the respective authors’ employers make any endorsement, representation or guarantee with regard to any content, and disclaim any liability in connection with the use or misuse of any information provided herein. This article should not be construed as professional or financial advice. Statements of fact and opinions expressed herein are those of the individual authors and are not necessarily those of the Society of Actuaries or the respective authors’ employers.

Damon Levine is an enterprise risk management practitioner and consultant. He can be reached at damonlevineCFA@gmail.com.