Loan lending plays an important role in our everyday life and powerfully promotes the growth of consumption and the economy. Loan default has been unavoidable, which carries a great risk and may even end up in a financial crisis. Therefore, it is particularly important to identify whether a candidate is eligible for receiving a loan. In this paper, we apply Random Forest and XGBoost algorithms to train the prediction model and compare their performance in prediction accuracy. In the feature engineering part, we use the variance threshold method and Variance Inflation Factor method to filter out unimportant features, and then we input those selected features into Random Forest and XGBoost models. It turns out that Random Forest and XGBoost show little difference in the accuracy of their predictions since both get high accuracy of around 0.9 in the loan default cases.
Loan lending plays an important role in our everyday life and powerfully promotes the growth of consumption and the economy [
In the past, the evaluation primarily depended on manual review, which was time-consuming and labor-intensive [
In this paper, based on the loan default data provided by Imperial College London, we predict whether a loan will default, as well as the loss incurred if it does default. We choose Random Forest and XGBooost to build the prediction model and decide which one performs better.
The rest of this paper is organized as follows. Section 2 describes the characteristics of the raw data and shows the methodology of this paper, including feature engineering and introductions of Random Forest and XGBooost. In Section 3 we show the experiment process of applying models and evaluate the results. Finally, we draw our conclusion and discuss the potential applications of our outcomes in Section 4.
At present, researchers generally use machine learning methods to predict loan defaults, including Logistic Regression, Decision Trees, Random Forest, XGBoost, and other advanced techniques.
The main advantages of Logistic Regression lie in its simple understanding, sturdy performance, and easy implementation [
Decision Trees generates a structure like a tree by classifying the instances and using recursive portioning algorithm [
Random Forest runs by constructing multiple decision trees while training and outputting the class that is the mode of the classes output by individual trees, which outperformed single decision trees [
XGBoost has been shown to achieve state-of-art results on many machine learning tasks. It is an improvement of Gradient Boosting algorithm and a decision tree based on the gradient boosting algorithm. Li constructed an XGBoost-based model to predict peer-to-peer loan default and compare its outcome with Logistic Regression and Decision Trees. The results indicated that the accuracy of the XGBoost-based model achieved 97.705%, which fitted the actual results better [
The framework of our experiment process is shown in
First, we take a quick look at the whole data. The dataset provided by Imperial College London includes 105,471 records and 771 columns, containing customers’ ids, 778 features, and the loss of the record. The detailed description is shown in
Next, we clean the NAs in the dataset. There are 525 columns containing NAs. When dealing with the numeric columns, we fill those NAs with the average of the columns.
| Columns | Description |
|---|---|
| id | The ids of customers. Float type. |
| F1, F2, …, F778 | Different features of the customer behaviors. Numeric types. |
| Loss | The loss incurred. Float type. Note: If the record does default, the number shows the severity of the loss that result. If there is no default, the number is 0. |
Feature selection aims to drop those redundant columns which contain little useful information and reduce the number of input features when developing an effective model.
First, we use the variance threshold method to quickly filter out those columns whose variance equals 0, since those columns do not contain useful information for classification. After applying the variance threshold method, we get 760 columns left.
Second, we apply Variance Inflation Factor method to reduce multicollinearity. Multicollinearity inflates unnecessarily the standard errors of the coefficients, and increased standard errors indicates that the coefficients of some features might be close to 0, which will make some features insignificant when they should be significant. A useful way to measure multicollinearity is Variance Inflation Factor (VIF). If no features are correlated, the VIF will be 1. If the VIF is greater than 10, it shows that the regression coefficients are poorly estimated due to multicollinearity [
After filtering features by the variance threshold method and the VIF, we have 419 columns left.
In this paper, we apply Random Forest and XGBoost algorithms to train the model and compare their performance in prediction accuracy. Empirical studies show that the best results are obtained if we use 20% - 30% of the data for testing, and the remaining 70% - 80% of the data for training [
Random Forest is a decision tree based supervised learning algorithm, which implements the classification by constructing multiple decision trees. The metric we choose for splitting attributes in decision trees is the Gini index. For a candidate split attribute Xi, denote possible levels as L 1 , L 2 , ⋯ , L j , the Gini Index is calculated as [
G ( X i ) = ∑ j = 1 J Pr ( X i = L j ) ( 1 − Pr ( X i = L j ) ) = 1 − ∑ j = 1 J Pr ( X i = L j ) 2
The steps of Random Forest algorithm is explained as follows.
1) Start with the selection of random samples.
2) Construct a decision tree for every sample, and get the result from every decision tree.
3) Perform voting for every result.
4) Select the most voted result as the final prediction result.
Random Forest has its own predominance. On the one hand, it costs relatively little time on a large dataset. On the other hand, it performs well for estimating missing data, which makes the accuracy of the prediction model robust even when a large amount of data is missing.
In this paper, we use Random Forest Classifier from the sklearn package in python to build the model.
XGBoost is also a decision tree based algorithm and is an improvement of the gradient boost algorithm. The steps of Random Forest algorithm is explained as follows.
1) First, we input the training set { ( x i , y i ) } , a loss function L ( y , F ( x ) ) , weak learners M and a learning rate α.
2) Initialize model with a constant
f ^ ( 0 ) ( x ) = arg min θ ∑ i = 1 N L ( y i , θ )
3) For m = 1 to M:
a) Compute the gradient and the hessians:
g ^ m ( x i ) = [ ∂ L ( y i , f ( x i ) ) ∂ f ( x i ) ] f ( x ) = f ^ ( m − 1 ) ( x )
h ^ m ( x i ) = [ ∂ L ( y i , f ( x i ) ) ∂ f ( x i ) 2 ] f ( x ) = f ^ ( m − 1 ) ( x )
b) Fit a base learner using the training set { x i , − g ^ m ( x i ) h ^ m ( x i ) } i = 1 N by solving the optimization question below:
φ ^ m = arg min φ ∑ i = 1 N 1 2 h ^ m ( x i ) [ − g ^ m ( x i ) h ^ m ( x i ) − φ ( x i ) ] 2
f ^ m ( x ) = α φ ^ m ( x )
c) Update the model:
f ^ m ( x ) = f ^ ( m − 1 ) ( x ) + f ^ m ( x )
4) Output
f ^ ( x ) = f ^ ( X ) ( x ) ∑ m = 0 M f ^ m ( x )
In this paper, we use XGBClassifier from the xgboost package in python to build the model.
The prediction accuracy of the Random Forest model is 0.90657, while the prediction accuracy of the XGBoost model is 0.90635. The result indicates that Random Forest and XGBoost show little difference in the accuracy of their predictions, and both get high accuracy in the loan default cases. The prediction distributions of Random Forest and XGBoost are shown in
This paper verifies the ability of Random Forest as well as XGBoost to predict loan default. In the feature engineering part we use the variance threshold method and Variance Inflation Factor method to filter out unimportant features, and then we input those selected features into Random Forest and XGBoost models. It turns out that Random Forest and XGBoost show little difference in the accuracy of their predictions since both get high accuracy in the loan default cases. In future research, we will implement comparative studies using other advanced machine learning algorithms such as Neural Network, KNN, and MLP, or the mixed model of them, to find the most suitable model for loan default prediction.
The author declares no conflicts of interest regarding the publication of this paper.
Wu, W.J. (2022) Machine Learning Approaches to Predict Loan Default. Intelligent Information Management, 14, 157-164. https://doi.org/10.4236/iim.2022.145011