Q&A: SAS’ Naeem Siddiqi on ML for credit scoring

By Rebekah Tunstead | 20 January 2020

With banks under pressure to increase profit margins, more are turning to machine learning (ML) to make a significant difference in their middle and back offices.

One such area that is gaining greater attention from both the market and regulators is the use of ML and alternative data for credit underwriting.

On December 19 the Bank of International Settlement published a paper on its research on the impact of ML models and alternative data on credit scoring in a Chinese fintech firm. While overall the fintech’s ML models outperformed traditional empirical models at predicating losses and defaults, “the comparative advantage of the model that uses the fintech credit scoring technique based on machine learning tends to decline when the length of the relationship increases.”

bobsguide caught up with Naeem Siddiqi, a senior advisor in the Risk Research and Quantitative department at SAS to discuss how banks are implementing ML models, the use of alternative data in credit scoring, and tackling the black box issue.

At what stage is ML adoption in credit scoring at the moment?

The adoption is highly segmented.

In the regulated industry, the adoption rate is very low. Last year I did a survey with Canadian banks and none of them were using ML to lend money. That doesn’t mean that they are not looking at it, absolutely everybody is looking at it. Every single bank I’ve spoken to has done proof of concepts. 

Then there is segmentation by the usage. [Banks] are using ML for models that are not lending models. For example, one of my customers at TMB bank in Thailand uses ML models to predict income for small and medium enterprises for example. Scotiabank in Canada uses ML for some of their collections work. You’ve got Bank of Montreal in Canada using ML to predict whether you will run out of money by the end of the month, and then they use it to pitch small loans to carry you over.

[Banks] use ML for knowledge discovery – a pure ML approach, an unsupervised approach. It is where you are supposed to take data and let the algorithm run through it. You are not going to tell it anything, you don’t tell it what the target is, you don’t tell it what you are going to do, and the algorithm is supposed to find out all these relationships all by itself.

What is the ratio of adoption between supervised ML models to knowledge creation models?

In the banking sector it is mostly all supervised. But it depends on the problem.

In banking there are some complex problems, meaning the relationships are very non-linear and complex. Things like fraud and anti-money laundering for example. There are lots of transactions and fast changing behaviour and these are two areas where ML has shown a significant lift, meaning you get a good advantage using ML.

Then you have the simpler problems, like probability of default. A fairly simple problem, fairly simple data historically, and so the normal linear techniques like logistic regression have done very well in the past. In most of the proof of concepts that I’ve seen, ML did not provide a massive lift, it did not provide an increase in accuracy for simple default models. What did provide the lift was using additional data.

As all modellers know very well, a lift always comes from the data mostly, not from the actual math.

Where do see most of the banks’ proof of concepts being focused?

Almost everybody is trying the usual stuff meaning lending models - can I build a better model to lend money.

Then you have people building models to predict time to repayment; AML and know your customer type models; income estimation; asset predication models where ML should do well; cash short-fall models.

Then you’ve got things on the marketing side, cross sell, propensity to revolve, propensity to make use of a credit limit increase. This is one part of ML where it is the math. Banks would say, ‘let’s take the same data, apply the math and see what happens.’ Typically, not much happens. The better banks are looking at a two-pronged approach, ‘yes, let’s use ML, but let’s also get additional data’. If you look at ML and artificial intelligence, these algorithms are designed to take advantage of very large data sets, dynamic data sets.

How do you see the use of alternative data evolving in credit underwriting?

Alternative data is not a uniform thing, there are many different types. There is what I call the better alternative data which is data from utilities: electricity, gas, phone payments.Then there is the transactional data from your bank accounts - your savings and checking accounts. In some cases, banks have been using it for a while, but in some countries this is considered new alternative data.

Then you have some dubious stuff, and this starts with social media. This is where the New York governor signed a law, where in New York at least, your credit score cannot be impacted by any of your online activity or your friends and followers.

You can recognise a lot of problems with this for large institutions to be using this data. I don’t think regulators are going to allow this until there is a lot of evidence that this is actually useful, and it cannot be manipulated.

I don’t see it going anywhere because most of the online data for example the make and model of your cell phone, and its screen resolution, those have been shown to be predictive. It doesn’t take somebody too long to change that. I’ve looked at data where if you follow financial media such as the Wall Street Journal or Financial Times your scores tend to be much higher than others, which is an indicator of your social status as well as your income. However, when this information becomes widely known, it doesn’t stop anyone from going to LinkedIn or Twitter and liking Bloomberg and the Financial Times, and they get more points.

Where is ML explainability at currently?

True explainability is a while yet. Today there are a lot of methods out there, including a bunch that SAS are looking at. SAS in our ML software, we offer model explainability tools using things like Local Interpretable Model-Agnostic Explanation (Lime) and Individual Conditional Expectation (Ice). These are mathematical techniques that give you approximations for what might be in the model, but we don’t know if it’s 100 percent sure. There is explainability like that.

Whether it is enough for regulators and internal model governance people in banks, that is the big question.  

*This article was update on Januray 21, to reflect that logistical regression techniques are linear. 

Become a bobsguide member to access the following

1. Unrestricted access to bobsguide
2. Send a proposal request
3. Insights delivered daily to your inbox
4. Career development