Every bank wants to hold there customers for sustaining their business so the ABC Multinational bank. The objective is to classify whether a Bank ABC customer is a high-risk of changing Bank. “Churn” is used as the target. 1 if the client has left the bank during some period or 0 if he/she has not using the predictors listed below (use “Bank Customer Churn Prediction.csv” file for your final project).

Response Variable| Description |
churn| 1 if the client has left the bank during some period or 0 if he/she has not. |
Variable Description
customer_id Account Number
credit_score credit score
country Name of the Country
gender Female or Male
age age
tenure How many years as client
balance Account Balance
products_number Number of products of the Bank
credit_card 0 – no, 1–yes
active_member 0 – no, 1–yes
estimated_salary Salary

Instructions:

  1. You may present the Project individually or join a team with your fellow students with appropriate size (Up to four Students total), if you need my help allocating you to a group let me know ()

  2. Load and Review the dataset named “Bank Customer Churn Prediction.csv”

  3. Create a model development document that describes the model using this template, input the name of the authors, Harvard IDs, the name of the Group, all of your code and calculations, etc..:

Notes No typographical errors, grammar mistakes, or misspelled words, use English language All tables need to be numbered and describe their content in the body of the document All figures/graphs need to be numbered and describe their content All results must be accurate and clearly explained for a casual reviewer to fully understand their purpose and impact Submit both the RMD markdown file and PDF with the sections with appropriate explanations. A more formal document in Word can be used in place of the pdf file but must include all appropriate explanations.

Executive Summary

This project developed a predictive model to identify customers at high risk of churning (i.e. changing banks) from ABC Multinational Bank, enabling proactive retention strategies such as personalized offers and improved customer engagement. Three classification models were trained and evaluated on a dataset of 10,000 customers: Logistic Regression, Boosted C5.0, and Random Forest. The Boosted C5.0 algorithm was selected as the champion model due to its superior performance, achieving an AUC of 0.863, F1 score of 0.689, and accuracy of 84.7% on the test set. Logistic Regression was retained as the benchmark for its interpretability: its odds ratios provide regulatory transparency. Key limitations include reduced explainability compared with Logistic Regression, sensitivity to class imbalance, and the model capturing patterns rather than causal relationships. Ongoing monthly monitoring using AUC and Population Stability Index is recommended to ensure continued effectiveness in deployment.

I. Introduction (5 points)

We at the ABC Multinational Bank are analyzing and interpreting a data set with information on 10,000 of our clients in hopes to better classify whether or not a customer is at high-risk of leaving the bank – and therefore losing us money. Through rigorous testing we have developed an optimized model to help apply targeted customer retention strategies before a loss of clientele. The data set we tested on contains information on demographics, account behavior, and product usage, including credit score, country (location of clientele), gender, age, tenure as a client, account balance, number of products of the regional Bank (products number), credit card ownership, active membership status, and estimated salary. Our model balances predictive accuracy, stability on the test sample (we used a 70%/30% training to testing sample split), and interpretability to output a “churn” value – a churn value of 1 indicates the client left ABC Multinational Bank during the observed period, while a value of 0 indicates successful clientele retention.

Our research team considered and compared multiple methods in order to find the most suitable model for predicting churn values. These included logistic regression, simple decision trees, and boosted decision trees.

II. Description of the data and quality (15 points)

The clientele information data set is very high quality and suitable for model training. There are no missing values nor any duplicated rows. The predictors include a mix of continuous and categorical variables. Credit score, age, tenure, products number, balance, and estimated salary are continuous, while country, gender, credit card ownership, and active member status are categorical. This means we had to construct dummy variables for the categorical variables in order to adequately build our model. However, prior to implementing our dummy variables, we performed diagnostic testing on the data set. Our findings were as follows:

The response variable churn is rather imbalanced, with 20.4% of clients leaving the bank, while 79.6% stayed (refer to Figure 1 below). This means when analyzing our model, accuracy alone may not be sufficient. In addition to accuracy, we evaluated model sensitivity and specificity, as well as the ROC curve and AUC. Several of the predictor variables show particularly meaningful differences when plotted against churn status. The distribution of age, as seen in Figure 2, is skewed right, with most clients being 25 to 40 years old. Figures 3 and 4 provide evidence that older clients, along with clients who have a higher account balance, are on average more likely to leave the ABC Multinational Bank. On top of age and account balance, gender has a noticeable impact on churn rates, with female clients having a higher probability of leaving the bank. France and Spain have virtually the same churn proportions (at 16% and 17% respectively), while Germany sees higher rates of clients leaving, with a 32% proportion. Figure 7 shows that the status of a member (either active or not) has a significant impact on churn rates. Active members are nearly half as likely to leave the bank (14% vs 27% for non-active members). Figure 8 informs us that estimated salary has a very minimal impact on churn probability. The final figure showing churn proportion by number of products reveals an interesting finding. All clients with a bank with four products churned, and a large majority (83%) of clients at a bank with three products churned as well. Banks with only two products seemed to perform the best, with a churn rate of only 7.6%. It is important to note, however, that only a small portion of the total data set represents banks with 3 or 4 products.

##   credit_score        age            tenure          balance      
##  Min.   :350.0   Min.   :18.00   Min.   : 0.000   Min.   :     0  
##  1st Qu.:584.0   1st Qu.:32.00   1st Qu.: 3.000   1st Qu.:     0  
##  Median :652.0   Median :37.00   Median : 5.000   Median : 97199  
##  Mean   :650.5   Mean   :38.92   Mean   : 5.013   Mean   : 76486  
##  3rd Qu.:718.0   3rd Qu.:44.00   3rd Qu.: 7.000   3rd Qu.:127644  
##  Max.   :850.0   Max.   :92.00   Max.   :10.000   Max.   :250898  
##  products_number estimated_salary    churn      credit_card_Yes 
##  Min.   :1.00    Min.   :    11.58   No :7963   Min.   :0.0000  
##  1st Qu.:1.00    1st Qu.: 51002.11   Yes:2037   1st Qu.:0.0000  
##  Median :1.00    Median :100193.91              Median :1.0000  
##  Mean   :1.53    Mean   :100090.24              Mean   :0.7055  
##  3rd Qu.:2.00    3rd Qu.:149388.25              3rd Qu.:1.0000  
##  Max.   :4.00    Max.   :199992.48              Max.   :1.0000  
##  active_member_Yes country_Spain    country_Germany   gender_Male    
##  Min.   :0.0000    Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
##  1st Qu.:0.0000    1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:0.0000  
##  Median :1.0000    Median :0.0000   Median :0.0000   Median :1.0000  
##  Mean   :0.5151    Mean   :0.2477   Mean   :0.2509   Mean   :0.5457  
##  3rd Qu.:1.0000    3rd Qu.:0.0000   3rd Qu.:1.0000   3rd Qu.:1.0000  
##  Max.   :1.0000    Max.   :1.0000   Max.   :1.0000   Max.   :1.0000
## [1] "Total Churn Values and Rates:"
## 
##   No  Yes 
## 7963 2037
## 
##     No    Yes 
## 0.7963 0.2037
## [1] "Churn Rates by Country:"
##          
##                  No       Yes
##   France  0.8384523 0.1615477
##   Spain   0.8332660 0.1667340
##   Germany 0.6755680 0.3244320
## [1] "Churn Rates by Active Member Status:"
##      
##              No       Yes
##   No  0.7314910 0.2685090
##   Yes 0.8573093 0.1426907
## [1] "Churn Rates and Total Values by Product Number:"
##    
##             No        Yes
##   1 0.72285602 0.27714398
##   2 0.92418301 0.07581699
##   3 0.17293233 0.82706767
##   4 0.00000000 1.00000000
##    
##       No  Yes
##   1 3675 1409
##   2 4242  348
##   3   46  220
##   4    0   60

III. Model Development Process (15 points)

## 'data.frame':    10000 obs. of  12 variables:
##  $ credit_score     : int  619 608 502 699 850 645 822 376 501 684 ...
##  $ age              : int  42 41 42 39 43 44 50 29 44 27 ...
##  $ tenure           : int  2 1 8 1 2 8 7 4 4 2 ...
##  $ balance          : num  0 83808 159661 0 125511 ...
##  $ products_number  : int  1 1 3 2 1 2 2 4 2 1 ...
##  $ estimated_salary : num  101349 112543 113932 93827 79084 ...
##  $ churn            : Factor w/ 2 levels "No","Yes": 2 1 2 1 1 2 1 2 1 1 ...
##  $ credit_card_Yes  : int  1 0 1 0 1 1 1 1 0 1 ...
##  $ active_member_Yes: int  1 1 0 0 1 0 1 0 1 1 ...
##  $ country_Spain    : int  0 1 0 0 1 1 0 0 0 0 ...
##  $ country_Germany  : int  0 0 0 0 0 0 0 1 0 0 ...
##  $ gender_Male      : int  0 0 0 0 0 1 1 0 1 1 ...
## Train split: 70.01 %
## Test split:  29.99 %
## Train churn rate: 20.37 %
## Test churn rate: 20.37 %

We chose a logistic regression model as our model due to the nature of our data. More specifically, we selected our model using stepwise AIC variable selection over the full logistic regression model due to better performance on certain metrics.

logit_full <- glm(churn ~ ., data=train, family="binomial")
summary(logit_full)
## 
## Call:
## glm(formula = churn ~ ., family = "binomial", data = train)
## 
## Coefficients:
##                     Estimate Std. Error z value Pr(>|z|)    
## (Intercept)       -3.281e+00  2.943e-01 -11.147  < 2e-16 ***
## credit_score      -9.176e-04  3.361e-04  -2.730  0.00633 ** 
## age                7.435e-02  3.110e-03  23.909  < 2e-16 ***
## tenure            -1.312e-02  1.118e-02  -1.173  0.24069    
## balance            3.446e-06  6.102e-07   5.647 1.63e-08 ***
## products_number   -6.968e-02  5.648e-02  -1.234  0.21726    
## estimated_salary  -1.638e-07  5.613e-07  -0.292  0.77045    
## credit_card_Yes   -7.038e-02  7.081e-02  -0.994  0.32025    
## active_member_Yes -1.090e+00  6.911e-02 -15.766  < 2e-16 ***
## country_Spain      1.527e-02  8.440e-02   0.181  0.85646    
## country_Germany    7.040e-01  8.094e-02   8.697  < 2e-16 ***
## gender_Male       -5.734e-01  6.534e-02  -8.775  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 7077.6  on 7000  degrees of freedom
## Residual deviance: 5969.4  on 6989  degrees of freedom
## AIC: 5993.4
## 
## Number of Fisher Scoring iterations: 5
logit_step <- stepAIC(logit_full, direction="both", trace=FALSE)
summary(logit_step)
## 
## Call:
## glm(formula = churn ~ credit_score + age + balance + active_member_Yes + 
##     country_Germany + gender_Male, family = "binomial", data = train)
## 
## Coefficients:
##                     Estimate Std. Error z value Pr(>|z|)    
## (Intercept)       -3.540e+00  2.586e-01 -13.692  < 2e-16 ***
## credit_score      -9.058e-04  3.359e-04  -2.697    0.007 ** 
## age                7.451e-02  3.107e-03  23.984  < 2e-16 ***
## balance            3.669e-06  5.871e-07   6.249 4.12e-10 ***
## active_member_Yes -1.088e+00  6.904e-02 -15.758  < 2e-16 ***
## country_Germany    6.846e-01  7.530e-02   9.092  < 2e-16 ***
## gender_Male       -5.733e-01  6.527e-02  -8.784  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 7077.6  on 7000  degrees of freedom
## Residual deviance: 5973.5  on 6994  degrees of freedom
## AIC: 5987.5
## 
## Number of Fisher Scoring iterations: 5

The final stepwise logistic regression model retained credit_score, age, balance, active_member_Yes, country_Germany, and gender as predictors of churn. Age, balance, active membership, being located in Germany, and gender were significant at the 5% level. Older customers and customers with higher account balances exhibited higher odds of churn, while active members and male customers exhibited lower odds of churn. Customers located in Germany were significantly more likely to churn than customers in the reference country. The final model reduced deviance substantially relative to the null model, indicating that the retained predictors provide meaningful explanatory power. The model substantially improved fit relative to the intercept-only model, reducing deviance from 7077.6 to 5973.5.

# Compute odds ratios
exp(coef(logit_step))
##       (Intercept)      credit_score               age           balance 
##        0.02900177        0.99909466        1.07735645        1.00000367 
## active_member_Yes   country_Germany       gender_Male 
##        0.33689204        1.98300666        0.56365751
# 95% CI for predictors
exp(confint(logit_step))
## Waiting for profiling to be done...
##                        2.5 %     97.5 %
## (Intercept)       0.01742474 0.04802264
## credit_score      0.99843671 0.99975225
## age               1.07085671 1.08398010
## balance           1.00000252 1.00000482
## active_member_Yes 0.29401278 0.38541689
## country_Germany   1.71097025 2.29855579
## gender_Male       0.49583848 0.64043400

Odds-ratio analysis confirmed that age, balance, active membership status, customer location in Germany, and gender were key drivers of churn in the final logistic regression model (James et al. 2014). Each additional year of age increased churn odds by about 7.7%, while active members had about 66% lower odds of churn than inactive members. Customers in Germany had approximately 1.98 times the odds of churn relative to the reference country, and male customers had lower odds of churn than female customers. Higher balances were also associated with increased churn risk, although the effect is best interpreted over larger balance changes because balance is measured in small units. Credit score had a statistically significant but modest protective effect, whereas tenure and number of products were only marginally significant.

Compute VIF to check for multicollinearity:

vif(logit_step)
##      credit_score               age           balance active_member_Yes 
##          1.000811          1.085746          1.186767          1.079263 
##   country_Germany       gender_Male 
##          1.184501          1.003899

All VIF values are close to 1, with the highest value equal to 1.187 for balance. These results indicate no evidence of problematic multicollinearity, suggesting that the coefficient estimates are stable and interpretable (James et al. 2014).

First we will perform any metric calculations that are not dependent on a prediction threshold cutoff. We check the Brier score (MSE), RMSE, SSE as follows:

##                                 Brier Score (MSE)      RMSE     SSE
## Probability Calibration Metrics         0.1350271 0.3674604 945.325

The logistic regression model had a Brier score of 0.135, which suggests that on average, the model’s predicted probabilities deviate from the true outcomes by a modest amount (n.d.). The corresponding RMSE of 0.367 also reflects moderate prediction error on the probability scale. The SSE was 945.325. However, this metric is scale-dependent and less informative than the two other metrics (James et al. 2014).

Compute the pseudo-\(R^2\):

## fitting null model for pseudo-r2
##           llh       llhNull            G2      McFadden          r2ML 
## -2986.7518470 -3538.7879986  1104.0723031     0.1559958     0.1458958 
##          r2CU 
##     0.2293518

The final logistic regression model achieved a McFadden pseudo-\(R^2\) of 0.156, indicating a meaningful improvement in fit relative to the intercept-only model. Although pseudo-\(R^2\) values in logistic regression are not interpreted the same way as ordinary pseudo-\(R^2\) in linear regression, this result suggests that the selected predictors provide a moderate and useful level of explanatory power for customer churn (Long 1997).

Next, we perform a Hosmer-Lemeshow goodness-of-fit test:

Hypotheses: \[ H_0: \text{model fits the data well}\\ \] \[ H_a: \text{model does not fit the data well} \]

Decision rule: At the 5% significance level, if p < 0.05, then reject \(H_0\). Otherwise, fail to reject \(H_0\).

## 
##  Hosmer and Lemeshow goodness of fit (GOF) test
## 
## data:  actual_numeric, probs_logit
## X-squared = 9.9436, df = 8, p-value = 0.269

Conclusion: At the 5% significance level, since the p-value is greater than 0.05, we fail to reject \(H_0\). There is sufficient evidence to suggest the model fits the data well (Hosmer and Lemeshow 2000).

We first check the ROC curve and AUC (Fawcett 2006) before deciding which cutoff to choose:

## Setting direction: controls < cases

## Area under the curve: 0.7688

The model achieved an AUC of 0.7624, meaning that in approximately 76.24% of randomly selected churner versus non-churner pairs, the model assigns a higher predicted probability to the churner. This indicates moderate discriminatory ability. While the model provides useful separation between customers who churn and those who do not, its ranking performance is only fair to acceptable rather than strong.

Finally, we actually try predicting churns two options by introducing a decision rule cutoff(Kennedy 2024). To determine what cutoff seems best - and if deviating from the trivial choice 0.5 is useful at all - one may build a function for simulation purposes.

Key Cutoff Selection Metrics for the Final Logistic Regression Model
Cutoff TN FP FN TP Accuracy Sensitivity Specificity Precision Recall F1 Balanced_Accuracy
0.50 5373 202 1115 311 0.8119 0.2181 0.9638 0.6062 0.2181 0.3208 0.5909
0.45 5276 299 1036 390 0.8093 0.2735 0.9464 0.5660 0.2735 0.3688 0.6099
0.40 5152 423 939 487 0.8055 0.3415 0.9241 0.5352 0.3415 0.4170 0.6328
0.35 4987 588 826 600 0.7980 0.4208 0.8945 0.5051 0.4208 0.4591 0.6576
0.30 4750 825 696 730 0.7827 0.5119 0.8520 0.4695 0.5119 0.4898 0.6820

Among the tested probability cutoffs, 0.35 provided the best overall balance between identifying churners and controlling false positives. While the 0.30 cutoff achieved the highest sensitivity, F1 score, and balanced accuracy, it also generated substantially more false positives and a lower precision rate. The 0.35 cutoff offered a stronger compromise, improving sensitivity meaningfully over 0.40 and 0.50 while maintaining better specificity and precision than 0.30. For this reason, 0.35 was selected as the final operational cutoff for the logistic regression model. Going forward, we will therefore only analyze that model:

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  4987  826
##        Yes  588  600
##                                           
##                Accuracy : 0.798           
##                  95% CI : (0.7884, 0.8074)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 0.3674          
##                                           
##                   Kappa : 0.3362          
##                                           
##  Mcnemar's Test P-Value : 2.926e-10       
##                                           
##               Precision : 0.5051          
##                  Recall : 0.4208          
##                      F1 : 0.4591          
##              Prevalence : 0.2037          
##          Detection Rate : 0.0857          
##    Detection Prevalence : 0.1697          
##       Balanced Accuracy : 0.6576          
##                                           
##        'Positive' Class : Yes             
##
Transposed Final Summary Metrics for the Logistic Regression Model on Train Data at a 0.35 Cutoff
Metric Logistic Regression (0.35 cutoff)
TN 4987
FP 588
FN 826
TP 600
Accuracy 0.7980
Kappa 0.3362
Sensitivity 0.4208
Specificity 0.8945
Precision 0.5051
Recall 0.4208
F1 0.4591
NPV 0.8579
Prevalence 0.2037
Detection Rate 0.0857
Detection Prevalence 0.1697
Balanced Accuracy 0.6576
AUC 0.7688
SSE 945.3250
RMSE 0.3675

On the train set, our stepwise selected logistic regression model achieved an accuracy of 79.4%, which is very close to the no information rate of 78.97%. This reflects the class imbalance. While precision (50.5%) and recall (42.1%) indicate moderate ability in identifying churners, the relatively low F1 score (45.9%) and balanced accuracy (65.8%) suggest the model performs better at identifying non-churners than it does for churners. Overall, our model demonstrates moderate in-sample performance.

IV. Model Performance Testing (15 points)

## Setting direction: controls < cases

## Area under the curve: 0.7624

The model achieved an AUC of 0.7624, meaning that in approximately 76.24% of randomly selected churner versus non-churner pairs, the model assigns a higher predicted probability to the churner (Fawcett 2006). This indicates moderate discriminatory ability. While the model provides useful separation between customers who churn and those who do not, its ranking performance is only fair to acceptable rather than strong.

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  2119  345
##        Yes  269  266
##                                           
##                Accuracy : 0.7953          
##                  95% CI : (0.7804, 0.8096)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 0.564755        
##                                           
##                   Kappa : 0.3384          
##                                           
##  Mcnemar's Test P-Value : 0.002472        
##                                           
##               Precision : 0.4972          
##                  Recall : 0.4354          
##                      F1 : 0.4642          
##              Prevalence : 0.2037          
##          Detection Rate : 0.0887          
##    Detection Prevalence : 0.1784          
##       Balanced Accuracy : 0.6614          
##                                           
##        'Positive' Class : Yes             
##
Transposed Final Summary Metrics for the Logistic Regression Model on Test Data at a 0.35 Cutoff
Metric Logistic Regression (0.35 cutoff)
TN 2119
FP 269
FN 345
TP 266
Accuracy 0.7953
Kappa 0.3384
Sensitivity 0.4354
Specificity 0.8874
Precision 0.4972
Recall 0.4354
F1 0.4642
NPV 0.8600
Prevalence 0.2037
Detection Rate 0.0887
Detection Prevalence 0.1784
Balanced Accuracy 0.6614
AUC 0.7624
SSE 413.8020
RMSE 0.3715

The logistic regression model showed similar performance on the train and test sets, which indicates the model generalizes well and that there is no evidence of overfitting. Brier score and RMSE increased only slightly from the train to test set, and classification metrics (accuracy, precision, recall, F1) were roughly the same. The model demonstrates moderate predictive performance on both sets, though it has limited ability to detect churners likely due to the class imbalance.

V. Challenger Models (15 points)

We use bank_factor, our factored dataset from earlier for these models.

## 'data.frame':    10000 obs. of  11 variables:
##  $ credit_score    : int  619 608 502 699 850 645 822 376 501 684 ...
##  $ country         : Factor w/ 3 levels "France","Spain",..: 1 2 1 1 2 2 1 3 1 1 ...
##  $ gender          : Factor w/ 2 levels "Female","Male": 1 1 1 1 1 2 2 1 2 2 ...
##  $ age             : int  42 41 42 39 43 44 50 29 44 27 ...
##  $ tenure          : int  2 1 8 1 2 8 7 4 4 2 ...
##  $ balance         : num  0 83808 159661 0 125511 ...
##  $ products_number : int  1 1 3 2 1 2 2 4 2 1 ...
##  $ credit_card     : Factor w/ 2 levels "No","Yes": 2 1 2 1 2 2 2 2 1 2 ...
##  $ active_member   : Factor w/ 2 levels "No","Yes": 2 2 1 1 2 1 2 1 2 2 ...
##  $ estimated_salary: num  101349 112543 113932 93827 79084 ...
##  $ churn           : Factor w/ 2 levels "No","Yes": 2 1 2 1 1 2 1 2 1 1 ...
## Train split: 70.01 %
## Test split:  29.99 %
## Train churn rate: 20.4 %
## Test churn rate: 20.4 %

Challenger 1: Boosted C5.0 Tree

The first challenger model we build is a Boosted C5.0 model (Perner 2018).

## 
## Call:
## C5.0.formula(formula = churn ~ ., data = train_factor, trials = 25, type
##  = "prob")
## 
## Classification Tree
## Number of samples: 7001 
## Number of predictors: 10 
## 
## Number of boosting iterations: 25 
## Average tree size: 22.2 
## 
## Non-standard options: attempt to group attributes

We test different probability cutoffs and analyze metrics for each (Kennedy 2024):

Testing Probability Cutoffs from 0.3 to 0.5 for C5.0 Boosted Model
Cutoff TN FP FN TP Accuracy Sensitivity Specificity Precision Recall F1 Balanced_Accuracy
0.50 5458 117 662 764 0.8887 0.5358 0.9790 0.8672 0.5358 0.6623 0.76
0.45 5390 185 596 830 0.8884 0.5820 0.9668 0.8177 0.5820 0.6800 0.77
0.40 5311 264 539 887 0.8853 0.6220 0.9526 0.7706 0.6220 0.6884 0.79
0.35 5197 378 478 948 0.8777 0.6648 0.9322 0.7149 0.6648 0.6890 0.80
0.30 5068 507 406 1020 0.8696 0.7153 0.9091 0.6680 0.7153 0.6908 0.81

While a cutoff of 0.30 achieves the highest recall it results in a large number of false positives and lower precision, which is less desirable. The 0.35 cutoff (consistent with the logistic regression model) maintains strong recall while significantly improving precision and reducing false positives. Strong recall is important when considering identifying customer churn for this bank. This cutoff also results in high overall performance metrics such as F1 score and balanced accuracy.

Evaluate model on train data:

## Setting levels: control = 0, case = 1
## Setting direction: controls < cases
##                   Boosted C5.0 Train
## Brier score (MSE)            0.08749
## RMSE                         0.29579
## SSE                        612.51083
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  5197  478
##        Yes  378  948
##                                           
##                Accuracy : 0.8777          
##                  95% CI : (0.8698, 0.8853)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.613           
##                                           
##  Mcnemar's Test P-Value : 0.000715        
##                                           
##               Precision : 0.7149          
##                  Recall : 0.6648          
##                      F1 : 0.6890          
##              Prevalence : 0.2037          
##          Detection Rate : 0.1354          
##    Detection Prevalence : 0.1894          
##       Balanced Accuracy : 0.7985          
##                                           
##        'Positive' Class : Yes             
##

The C5.0 boosted model showed strong in-sample performance with an accuracy of 87.77%, which is substantially higher than the no-information rate of 79.63%. Precision (71.49%), recall (66.48%), and F1-score (68.9%) showed good ability to identify churners, and the balanced accuracy of 79.85% suggests solid performance across both classes. In terms of probability metrics, the boosted C5.0 model also achieves a lower Brier score and RMSE, which indicates more accurate probability predictions than the logistic regression model (n.d.). The boosted C5.0 model substantially outperforms logistic regression on the training set, though we must assess test set performance in case there is overfitting.

Evaluate model on test data:

## Setting levels: control = 0, case = 1
## Setting direction: controls < cases
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  2157  228
##        Yes  231  383
##                                           
##                Accuracy : 0.8469          
##                  95% CI : (0.8336, 0.8597)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 6.052e-13       
##                                           
##                   Kappa : 0.5291          
##                                           
##  Mcnemar's Test P-Value : 0.9256          
##                                           
##               Precision : 0.6238          
##                  Recall : 0.6268          
##                      F1 : 0.6253          
##              Prevalence : 0.2037          
##          Detection Rate : 0.1277          
##    Detection Prevalence : 0.2047          
##       Balanced Accuracy : 0.7651          
##                                           
##        'Positive' Class : Yes             
##

The boosted C5.0 model demonstrated strong performance, though some overfitting is evident. The Brier score increased from the train to test set, meaning there was a decline in the probability prediction accuracy. Similarly, the accuracy decreased from 87.8% to 84.7% in the test set. The precision, recall, and F1 scores were also slightly lower when evaluated on the test set. Despite this performance drop, however, the model still achieves relatively strong test performance compared to logistic regression, so we can assume this model has good predictive ability and minimal overfitting.

## Area under the curve: 0.8629

The model had a small drop in AUC from 0.90 in the train set to 0.86 in the test set which indicates minimal overfitting and good generalization (Fawcett 2006).

Challenger 2: Random Forest

A random forest extends the decision tree by averaging predictions across 500 trees, each trained on a bootstrap sample with a random subset of predictors at each split. This reduces variance and produces a more stable, higher-performing classifier. OOB error provides an unbiased estimate of generalization error without needing a separate validation set (Perner 2018).

## 
## Call:
##  randomForest(formula = churn ~ credit_score + country + gender +      age + balance + products_number + credit_card + active_member,      data = train_factor, ntree = 500, mtry = 3, classwt = c(No = 1,          Yes = 4), importance = TRUE) 
##                Type of random forest: classification
##                      Number of trees: 500
## No. of variables tried at each split: 3
## 
##         OOB estimate of  error rate: 16.1%
## Confusion matrix:
##       No Yes class.error
## No  5066 509  0.09130045
## Yes  618 808  0.43338008

The Random Forest model achieved an out-of-bag (OOB) error rate of 16.1%, which indicates good overall predictive performance. However, the confusion matrix shows an imbalance in classification performance, with a lower error rate for non-churners (9.1%) and a high error rate for churners (43.3%).

We applied class weights (No = 1, Yes = 4) to penalize misclassification of churners more heavily. While this shifts the model’s focus toward the minority class, the relatively high error rate for churners suggests that imbalance remains an issue.

The variable importance plot shows that age and products_number are the most important based on Mean Decrease Accuracy, meaning they have a strong impact on model performance. Based on Mean Decrease Gini, age, balance, and credit_score are the most important, which suggests they are frequently used in tree splits.

Evaluate train set performance:

## Setting levels: control = 0, case = 1
## Setting direction: controls < cases
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  5444    0
##        Yes  131 1426
##                                           
##                Accuracy : 0.9813          
##                  95% CI : (0.9778, 0.9843)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9442          
##                                           
##  Mcnemar's Test P-Value : < 2.2e-16       
##                                           
##               Precision : 0.9159          
##                  Recall : 1.0000          
##                      F1 : 0.9561          
##              Prevalence : 0.2037          
##          Detection Rate : 0.2037          
##    Detection Prevalence : 0.2224          
##       Balanced Accuracy : 0.9883          
##                                           
##        'Positive' Class : Yes             
##

Evaluate test set performance:

## Setting levels: control = 0, case = 1
## Setting direction: controls < cases
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  2143  220
##        Yes  245  391
##                                           
##                Accuracy : 0.8449          
##                  95% CI : (0.8315, 0.8577)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 4.804e-12       
##                                           
##                   Kappa : 0.5293          
##                                           
##  Mcnemar's Test P-Value : 0.2657          
##                                           
##               Precision : 0.6148          
##                  Recall : 0.6399          
##                      F1 : 0.6271          
##              Prevalence : 0.2037          
##          Detection Rate : 0.1304          
##    Detection Prevalence : 0.2121          
##       Balanced Accuracy : 0.7687          
##                                           
##        'Positive' Class : Yes             
##

The random forest model showed strong performance with a test AUC of 0.858, the highest of the three models so far. We applied class weights of No = 1 and Yes = 4 to correct for the 80/20 class imbalance, which improved the sensitivity while minimizing AUC loss. Performance decreased from the train to test set (e.g., accuracy went down from 98.1% to 84.4%), which suggests moderate overfitting and poor generalization. The increase in Brier score and RMSE also indicates slightly less accurate probability predictions out-of-sample. Overall, the model had solid predictive performance on the test set, though its ability to detect churners (recall \(\approx\) 0.61) remains moderate.

ROC Curve Comparison on Test Data

The boosted C5.0 model had the most optimal ROC curve and the highest AUC. It had the best discrimination across all thresholds (Fawcett 2006).

Model Comparison Table

Out of Sample Model Performance Comparison
Model Accuracy Sensitivity Specificity Precision Recall F1.Score AUC MSE RMSE
Logistic Regression 0.7953 0.4354 0.8874 0.4972 0.4354 0.4642 0.7624 0.1380 0.3715
Boosted C5.0 0.8469 0.6268 0.9033 0.6238 0.6268 0.6253 0.8629 0.1033 0.3215
Random Forest 0.8449 0.6399 0.8974 0.6148 0.6399 0.6271 0.8578 0.1166 0.3415

Given the class imbalance in the dataset (~20% churn), we applied different strategies across the models to improve detection of the minority class. Logistic regression addresses class imbalance through threshold tuning, whereas random forest incorporates class weighting during training, leading to improved detection of churners. The Boosted C5.0 model is inherently less sensitive to class imbalance, but we also used threshold adjustments to improve this model (Kennedy 2024).

VI. Model Limitations and Assumptions (15 points)

6.1 Champion and Benchmark Selection

Based on Table 7, two models are designated for ongoing use:

  • Champion Model: Boosted C5.0 - Selected for its superior test set performance and stability when generalizing on unseen data. It achieved the highest AUC, accuracy, and lowest prediction error (Brier score, RMSE), meaning its probability estimates were the most well-calibrated. In addition, it had the highest F1 score, which indicates good balance between precision and recall.

  • Benchmark: Logistic Regression — While logistic regression offers greater interpretability through its odds ratios, it performed substantially worse than the boosted C5.0 and Random Forest models on key metrics such as AUC and F1 score. Given the objective of accurately identifying churners, the improvement in predictive performance provided by the boosted model outweighs the loss in interpretability in this context.

  • Random Forest — The Random Forest model performed strongly and achieved metrics similar to those of the Boosted C5.0 model on the test set. However, it was highly unstable when evaluated on unseen data and does not generalize well given the train set accuracy of 98.1% and test set accuracy of 84.5%.

6.2 Logistic Regression Assumptions

6.2.1 Assessment of Assumptions

Logistic Regression Assumption Assessment
Assumption Status
Binary outcome Met — churn is 0/1
Independence of observations Met — cross-sectional data, one row per customer
No severe multicollinearity Met — all VIF < 2 (Table 3)
Linear log-odds relationship Partially met — age may have non-linear effect
Adequate sample size Met — 7,001 training observations
No complete separation Met — No evidence of complete separation after proper variable encoding

4 of the 6 assumptions were fully met for the Logistic Regression model, though age non-linearity in particular is a concern (Bobbitt 2020).

6.2.2 Box-Tidwell Test

The Box-Tidwell test checks whether the log-odds relationship is truly linear in continuous predictors. A significant interaction term (variable x log(variable)) signals non-linearity.

## 
## Call:
## glm(formula = churn_num ~ age + age_log + balance + bal_log + 
##     credit_score + cs_log, family = binomial, data = train_bt)
## 
## Coefficients:
##                Estimate Std. Error z value Pr(>|z|)    
## (Intercept)  -1.149e+01  2.268e+00  -5.069    4e-07 ***
## age           1.395e+00  1.029e-01  13.553   <2e-16 ***
## age_log      -2.753e-01  2.123e-02 -12.964   <2e-16 ***
## balance       4.351e-05  1.694e-05   2.568   0.0102 *  
## bal_log      -3.218e-06  1.423e-06  -2.262   0.0237 *  
## credit_score -5.724e-02  2.434e-02  -2.352   0.0187 *  
## cs_log        7.560e-03  3.263e-03   2.317   0.0205 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 7077.6  on 7000  degrees of freedom
## Residual deviance: 6201.5  on 6994  degrees of freedom
## AIC: 6215.5
## 
## Number of Fisher Scoring iterations: 5

The Box-Tidwell test results indicate that the interaction terms for age, balance, and credit score are statistically significant, which suggests these predictors do not have a linear relationship with the log-odds of churn. This violates the linearity assumption of logistic regression. Transformations or polynomial terms are worth exploring in future models (Leung 2021).

6.3 Residuals and Normality

For logistic regression, residuals are not required to be normally distributed — this is an OLS assumption that does not apply here. For random forest there are no distributional assumptions at all (Perner 2018).

Random Forest OOB Error Rate

Random Forest OOB Error Rate

OOB error stabilizes ~50 trees. 500 trees are more than sufficient, so there is no benefit beyond this amount. While the overall OOB error rate is relatively low (~15%), the class-specific error rates show that the model performs much better for non-churners than for churners. In particular, the error rate for churners remains high at ~43%, which suggests the model struggles more to identify the minority class.

6.4 Tree-Based Ensemble Model Assumptions

The Boosted C5.0 and Random Forest are flexible and have no strict assumptions on linearity or functional form, but they require careful tuning and evaluation to avoid overfitting and handle class imbalance. These were addressed during our challenger model development through class weighting and threshold tuning (James et al. 2014; Kennedy 2024).

6.5 Pseudo R^2, SSE and RMSE Summary

Goodness-of-Fit Summary — All Models
Metric Logistic Boosted.C5.0 Random.Forest
McFadden Pseudo R^2 0.1560 NA NA
MSE (test) 0.1380 0.1033 0.1166
RMSE (test) 0.3715 0.3215 0.3415
AUC (test) 0.7624 0.8629 0.8578

Pseudo-\(R^2\) (only for logistic regression) estimates the goodness-of-fit by comparing the log-likelihood of a full model against an intercept-only model and typically ranges from 0 to 1. Higher values suggest better fit, with values greater than 0.20 considered as excellent (Long 1997). Lower MSE (Brier score) and RMSE indicate better probability calibration (n.d.).

6.6 Known Limitations

Logistic Regression: (Leung 2021)

  • Assumes linear log-odds relationship — Box-Tidwell suggests age may be non-linear
  • Prone to overfitting in higher-dimensional spaces where the number of observations is low
  • Sensitive to influential observations flagged in Cook’s Distance plot

Random Forest: (Perner 2018)

  • Black-box: cannot explain individual predictions to customers or regulators
  • Required class weight correction due to 80/20 class imbalance, without this model predicted almost all “No”
  • Cannot extend beyond the range of the training data

Boosted C5.0: (Perner 2018)

  • Boosted models may be susceptible to overfitting
  • Unlike logistic regression, boosted trees do not have simple, easily interpretable relationships (e.g., odds ratios), which makes them harder to explain in a business/regulatory context
  • Even if overall performance is strong, the model can still exhibit sensitivity to class imbalance

Dataset:

  • Static cross-sectional snapshot, no time-series features like balance trend or transaction frequency
  • No information on competitor offers or macroeconomic conditions that may independently drive churn

VII. Ongoing Model Monitoring Plan (5 points)

A deployed churn model operates in a live environment where customer behavior, economic conditions, and bank product offerings continuously evolve. Without active monitoring, a model that performs well today can silently degrade and produce misleading scores (Sobolik and Boudard 2024).

7.1 Quantitative Monitoring Thresholds

Model Monitoring Thresholds and Triggers
Metric Frequency Alert_Threshold Action
AUC (ROC) Monthly Drop > 0.03 from baseline (below 0.81) Re-validate, consider retraining
Population Stability Index (PSI) Monthly PSI > 0.20 Investigate data drift, retrain
Accuracy Monthly Drop > 3% from baseline (below 0.80) Flag for review
Actual vs Expected Churn Rate Monthly Divergence > 5 percentage points Recalibrate probability outputs
Hosmer-Lemeshow p-value Quarterly p < 0.05 Recalibrate model

Thresholds were set relative to our actual test results: AUC baseline 0.8629, accuracy baseline 0.8469.

7.2 Monitoring Assumptions

The model assumes the following conditions remain stable during deployment:

  1. Stable predictor distributions — age, balance, and other inputs do not shift materially. Monitored via PSI, where PSI > 0.20 signals significant population shift requiring retraining.

  2. Consistent data pipeline — input data continues to be formatted identically to training data. Any upstream schema changes must trigger an immediate model review.

  3. Stable churn definition — the business definition of churn remains unchanged. Redefining churn invalidates the model entirely.

  4. No structural breaks — major economic shocks, regulatory changes, or bank mergers may fundamentally alter customer behavior in ways the model cannot anticipate.

7.3 Model Replacement Triggers

The model should be retired and fully rebuilt when any of the following occur:

  • AUC falls below 0.75 on two consecutive monthly monitoring samples
  • PSI exceeds 0.25 for two consecutive periods
  • A regulatory requirement mandates a different modeling approach
  • A challenger model demonstrates materially superior performance in a scheduled annual evaluation

VIII. Conclusion (5 points)

We chose the Boosted C5.0 model based on its test performance and stability compared to its training performance. It achieved the highest AUC (0.8629), accuracy (0.8469), and the lowest prediction error (MSE and RMSE), which indicates strong discrimination and well-calibrated probabilities. While Random Forest showed comparable performance and slightly higher recall on the test set, it exhibited greater overfitting and weaker probability calibration, making it less robust overall. Logistic regression, although more interpretable, performed substantially worse across all metrics, so its predictive capability was not comparable to the other two models. Given the objective of accurately identifying customers at risk of churn, we select the Boosted model as the champion model since it provides the best balance of predictive performance and generalizability.

Bibliography (7 points)

n.d. Library.virginia.edu. https://library.virginia.edu/data/articles/a-brief-on-brier-scores.
Bobbitt, Zach. 2020. “The 6 Assumptions of Logistic Regression (with Examples).” Statology. https://www.statology.org/assumptions-of-logistic-regression/.
Fawcett, Tom. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8): 861–74. https://doi.org/https://doi.org/10.1016/j.patrec.2005.10.010.
Hosmer, David W., and Stanley. Lemeshow. 2000. Applied Logistic Regression David w. Hosmer, Stanley Lemeshow. 2nd ed. Wiley Series in Probability and Statistics. New York: Wiley.
James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2014. An Introduction to Statistical Learning: With Applications in r. Springer Publishing Company, Incorporated.
Kennedy, W Brett. 2024. “Achieve Better Classification Results with ClassificationThresholdTuner.” Medium. TDS Archive. https://medium.com/data-science/achieve-better-classification-results-with-classificationthresholdtuner-39c5d454637e.
Leung, Kenneth. 2021. “Assumptions of Logistic Regression, Clearly Explained.” Medium. TDS Archive. https://medium.com/data-science/assumptions-of-logistic-regression-clearly-explained-44d85a22b290.
Long, J. S. 1997. Regression Models for Categorical and Limited Dependent Variables. Thousand Oaks: Sage.
Perner, Petra, ed. 2018. Machine Learning and Data Mining in Pattern Recognition: 14th International Conference, MLDM 2018, New York, NY, USA, July 15-19, 2018, Proceedings, Part II. Berlin, Heidelberg: Springer-Verlag.
Sobolik, Thomas, and Léopold Boudard. 2024. “Machine Learning Model Monitoring: Best Practices.” Datadog. https://www.datadoghq.com/blog/ml-model-monitoring-in-production-best-practices/.

Appendix (3 points)

Confusion Matrices

## [1] "Logistic Regression Train"
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  4987  826
##        Yes  588  600
##                                           
##                Accuracy : 0.798           
##                  95% CI : (0.7884, 0.8074)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 0.3674          
##                                           
##                   Kappa : 0.3362          
##                                           
##  Mcnemar's Test P-Value : 2.926e-10       
##                                           
##               Precision : 0.5051          
##                  Recall : 0.4208          
##                      F1 : 0.4591          
##              Prevalence : 0.2037          
##          Detection Rate : 0.0857          
##    Detection Prevalence : 0.1697          
##       Balanced Accuracy : 0.6576          
##                                           
##        'Positive' Class : Yes             
## 
## [1] "Logistic Regression Test"
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  2119  345
##        Yes  269  266
##                                           
##                Accuracy : 0.7953          
##                  95% CI : (0.7804, 0.8096)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 0.564755        
##                                           
##                   Kappa : 0.3384          
##                                           
##  Mcnemar's Test P-Value : 0.002472        
##                                           
##               Precision : 0.4972          
##                  Recall : 0.4354          
##                      F1 : 0.4642          
##              Prevalence : 0.2037          
##          Detection Rate : 0.0887          
##    Detection Prevalence : 0.1784          
##       Balanced Accuracy : 0.6614          
##                                           
##        'Positive' Class : Yes             
## 
## [1] "Boosted C5.0 Train"
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  5197  478
##        Yes  378  948
##                                           
##                Accuracy : 0.8777          
##                  95% CI : (0.8698, 0.8853)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.613           
##                                           
##  Mcnemar's Test P-Value : 0.000715        
##                                           
##               Precision : 0.7149          
##                  Recall : 0.6648          
##                      F1 : 0.6890          
##              Prevalence : 0.2037          
##          Detection Rate : 0.1354          
##    Detection Prevalence : 0.1894          
##       Balanced Accuracy : 0.7985          
##                                           
##        'Positive' Class : Yes             
## 
## [1] "Boosted C5.0 Test"
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  2157  228
##        Yes  231  383
##                                           
##                Accuracy : 0.8469          
##                  95% CI : (0.8336, 0.8597)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 6.052e-13       
##                                           
##                   Kappa : 0.5291          
##                                           
##  Mcnemar's Test P-Value : 0.9256          
##                                           
##               Precision : 0.6238          
##                  Recall : 0.6268          
##                      F1 : 0.6253          
##              Prevalence : 0.2037          
##          Detection Rate : 0.1277          
##    Detection Prevalence : 0.2047          
##       Balanced Accuracy : 0.7651          
##                                           
##        'Positive' Class : Yes             
## 
## [1] "Random Forest Train"
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  5444    0
##        Yes  131 1426
##                                           
##                Accuracy : 0.9813          
##                  95% CI : (0.9778, 0.9843)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9442          
##                                           
##  Mcnemar's Test P-Value : < 2.2e-16       
##                                           
##               Precision : 0.9159          
##                  Recall : 1.0000          
##                      F1 : 0.9561          
##              Prevalence : 0.2037          
##          Detection Rate : 0.2037          
##    Detection Prevalence : 0.2224          
##       Balanced Accuracy : 0.9883          
##                                           
##        'Positive' Class : Yes             
## 
## [1] "Random Forest Test"
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  2143  220
##        Yes  245  391
##                                           
##                Accuracy : 0.8449          
##                  95% CI : (0.8315, 0.8577)
##     No Information Rate : 0.7963          
##     P-Value [Acc > NIR] : 4.804e-12       
##                                           
##                   Kappa : 0.5293          
##                                           
##  Mcnemar's Test P-Value : 0.2657          
##                                           
##               Precision : 0.6148          
##                  Recall : 0.6399          
##                      F1 : 0.6271          
##              Prevalence : 0.2037          
##          Detection Rate : 0.1304          
##    Detection Prevalence : 0.2121          
##       Balanced Accuracy : 0.7687          
##                                           
##        'Positive' Class : Yes             
##

The confusion matrices compare each model’s predicted vs. actual churn classifications across train and test sets, highlighting Random Forest’s overfitting (near-perfect training accuracy that drops significantly on test data) and Logistic Regression’s more consistent generalization.

ROC Curves

The six ROC curves display each model’s ability to distinguish between churned and non-churned customers across train and test sets, showing that Random Forest and Boosted C5.0 substantially outperform Logistic Regression, with the near-perfect Train Random Forest curve confirming overfitting to the training data.

Additional Data Analysis Plots

Tenure by Churn Status (box plot): Customers who churned (Yes) tend to have higher and more variable tenure than those who didn’t, suggesting that longer-tenured customers are not necessarily more loyal and may still leave the bank.

Churn Proportion by Credit Card Status (stacked bar): The churn rate is nearly identical between customers with and without a credit card (~20% each), indicating that credit card ownership has little to no predictive power for churn

Aggregated Model Metrics

Aggregated Model Metrics table: This table compares all three models (Logistic Regression, Boosted C5.0, Random Forest) across train and test sets on metrics like accuracy, AUC, and F1-score, revealing that Random Forest revealed high training performance which showed signs of overfitting, while Boosted C5.0 offered the strongest balance between train and test generalization.