Objective

To validate a neural-network-based wire payment fraud detection model for regulatory submission and enhanced strategic decision-making. The goal was to validate the bank’s model for two large US portfolios and provide a decision on the model's use.

CRISIL's Solution

CRISIL GR&RS performed validation of the wire payment fraud detection model. This involved:

• Reviewing all documents submitted for validation and assessing whether more information was required from stakeholders to complete the thorough validation process;
  
• Holding regular discussions with stakeholders to ensure all ambiguities in their reports were removed before performing additional tests - at CRISIL’s suggestion - to evaluate the model's performance.
  

Validation Methodology

• Thorough study of the model development document and other documents and research articles relevant to understanding of the model;
• Review all tests done by the developers for evaluating the neural networks model performance;
• Prescribe and review results of performance tests other than those done by the model developer.
  

Validation Highlights

• The neural network model was found to be not performing well

Client Impact

• Model validation completed in short timeframe 
• Significant shortcomings in model performance identified
• CRISIL provided insights and suggestions, including new performance metrics to evaluate fraud model and enhancements to the quality of model documentation
• Created a thorough validation document to be used by the client for CCAR submission
• Provided specific recommendations about future use of the model in the two target portfolios
  

Questions

Looking for high-end research and risk services? Reach out to us at: