Validation of Predictive Models for Insurance Applications

Background and Purpose

Predictive models are becoming increasingly important in insurance applications such as accelerated underwriting, experience studies, and sales actuarial modeling of dynamic behaviors. Although the benefits of predictive models are apparent, the rapid adoption of these models present a number of challenges for insurers. For example:

  1. As predictive models are a relatively new tool for many users and other stakeholders:
    1. There may not be a complete understanding of the associated strengths and limitations of the models that are used, which may lead to unfamiliar issues such as difficulty in gaining acceptance of the models and communicating results to stakeholders, and understanding predictive model outputs.
    2. Evaluating the effectiveness of the many available predictive modeling forms and selecting the appropriate predictive model and data required for a particular objective may be a complex undertaking.
    3. There may be a lack of understanding of the maintenance, governance and controls that are needed around the data and models.
    4. Actuaries and other practitioners may not know how to validate model results.
  2. Additionally, predictive modeling practitioners have to deal with a lot of data issues. Often both art and science are involved when dealing with data outliers, missing data, miscoding and the use of qualitative data. As a core tenet of data science is reproducibility of research, these data issues create a new challenge for actuaries to develop appropriate methods for documentation and communication.

The purpose of this research is to develop a resource for actuaries and other practitioners to address the challenges insurers face with validating predictive models and also to enhance current practices in assessing, selecting, implementing, maintaining data and models and validating model outcomes.

Research Objective

The Modeling Section, Predictive Analytics and Futurism Section, and other sponsors (“Sponsors”) are seeking researcher(s) to identify and address the challenges insurers face with predictive models leading to, at a minimum, suggestions for enhancing practices around evaluating and selecting modeling approaches, data collection and maintenance, and independent reviews of predictive models.   The researcher will perform a survey of insurers, gathering information on the usage of predictive models across the company and current practices for setting up, maintaining and reviewing predictive models, data and output.

Utilizing survey results as well as other sources, the researcher(s) will evaluate the effectiveness of insurer practices and:

  1. Recommend leading practices regarding data issues, including but not limited to:
    1. Selection and validation of data sources
    2. Frequency of refreshing and rebuilding of models
    3. The application of ASOP 23, “Data Quality”, and ASOP 25, “Credibility”, and suggestions for how to consider the draft ASOPs “Modeling” and “Setting Assumptions”.
  2. Recommend leading practices for independent reviewers of predictive models to assess the models and their results.  Consider ways and means to verify model results, both prospectively and retrospectively. Consider the application of ASOP 21, “Assisting or Responding to Auditors or Examiners…”, as well as other applicable ASOPs, to the review of predictive models. Develop a life insurance case study illustrating these methods.
  3. Identify and outline approaches insurers are utilizing to develop margins for numeric predictive modeling results.

Results are to be summarized into a report suitable for publication on the SOA’s website.

Proposal

To facilitate the evaluation of proposals, the following information should be submitted:

  1. Resumes of the researcher(s), including any graduate student(s) expected to participate, indicating how their background, education and experience bear on their qualifications to undertake the research. If more than one researcher is involved, a single individual should be designated as the lead researcher and primary contact. The person submitting the proposal must be authorized to speak on behalf of all the researchers as well as for the firm or institution on whose behalf the proposal is submitted.
  2. An outline of the approach to be used (e.g. literature search, model, etc.), emphasizing issues that require special consideration. Details should be given regarding the techniques to be used, collateral material to be consulted, and possible limitations of the analysis.
  3. A description of the expected deliverables and any supporting data, tools or other resources.
  4. Cost estimates for the research, including computer time, salaries, report preparation, material costs, etc. Such estimates can be in the form of hourly rates, but in such cases, time estimates should also be included. Any guarantees as to total cost should be given and will be considered in the evaluation of the proposal. While cost will be a factor in the evaluation of the proposal, it will not necessarily be the decisive factor.
  5. A schedule for completion of the research, identifying key dates or time frames for research completion and report submissions.  The Sponsors are interested in completing this project in a timely manner.  Suggestions in the proposal for ensuring timely delivery, such as fee adjustments, are encouraged.
  6. Other related factors that give evidence of a proposer's capabilities to perform in a superior fashion should be detailed. 

Selection Process

The Sponsors will appoint a Project Oversight Group (POG) to oversee the project.  The POG is responsible for recommending the proposal to be funded.  Input from other knowledgeable individuals also may be sought, but the POG will make the final recommendation, subject to SOA leadership approval. The SOA's Research Actuary will provide staff actuarial support.

Questions

Any questions regarding this RFP should be directed to Ronora Stryker, SOA Research Actuary (phone: 847-706-3614; email:  rstryker@soa.org.    

Notification Of Intent To Submit ProposaL

If you intend to submit a proposal, please e-mail written notification by February 15, 2018 to Jan Schuh.

Submission of Proposal

Please e-mail a copy of the proposal to Jan Schuh.

Proposals must be received no later than February 23, 2018. It is anticipated that all proposers will be informed of the status of their proposal by the end of February 2018.

Note: Proposals are considered confidential and proprietary.

Conditions

The selection of a proposal is conditioned upon and not considered final until a Letter of Agreement is executed by both the Society of Actuaries and the researcher.

The Sponsors reserve the right to not award a contract for this research. Reasons for not awarding a contract could include, but are not limited to, a lack of acceptable proposals or a finding that insufficient funds are available. The Sponsors also reserve the right to redirect the project as is deemed advisable.

The Sponsors plan to hold the copyright to the research and to publish the results with appropriate credit given to the researcher(s).

The Sponsors may choose to seek public exposure or media attention for the research.  By submitting a proposal, you agree to cooperate with the Sponsors in publicizing or promoting the research and responding to media requests.

The Sponsors may also choose to market and promote the research to members, candidates and other interested parties.  You agree to perform promotional communication requested by the Sponsors, which may include, but is not limited to, leading a webcast on the research, presenting the research at an SOA meeting, and/or writing an article on the research for an SOA newsletter.