E-commerce Customer Churn Classification

End-to-end churn classification pipeline demonstrating: rigorous 3-way data splits, probabilistic evaluation with proper scoring rules, post-hoc beta calibration, and SHAP-based model interpretability. Four models trained and compared; CatBoost achieves PR-AUC 0.983 and BSS 0.905 on the held-out test set.

---

Key Results

Model	PR-AUC ↑	ROC-AUC ↑	Brier Skill Score ↑	Brier Score ↓	ECE ↓
CatBoost	0.983	0.994	0.905	0.013	0.017
XGBoost	0.982	0.992	0.886	0.016	0.019
RandomForest	0.973	0.992	0.853	0.021	0.013
LogisticRegression	0.679	0.882	0.363	0.089	0.048

All metrics on the held-out test set. Models never saw this data during training, hyperparameter search, or calibration.

CatBoost at optimal threshold (0.374): precision 0.97, recall 0.96, F1 0.96 on the churn class.

---

Table of Contents

1. Dataset
2. Project Structure
3. Pipeline
4. EDA Findings
5. Modelling
6. SHAP Interpretability
7. Business Recommendations
8. Design Decisions
9. How to Run
10. References

---

Dataset

Source: E-Commerce Customer Churn Analysis and Prediction (Kaggle)

Property	Value
Rows	5,630 customers
Features	18 raw (27 post-transform)
Target	Churn (binary: 1 = churned)
Class balance	16.8% churn — ~5:1 imbalance
Split	70% train / 10% calibration / 20% test

Features

Numeric (12, including CityTier as ordinal)

Feature	Description
Tenure	Months as a customer
WarehouseToHome	Distance (km) from warehouse
HourSpendOnApp	Monthly app usage (hours)
NumberOfDeviceRegistered	Linked devices
SatisfactionScore	Satisfaction rating (1–5)
NumberOfAddress	Saved delivery addresses
OrderAmountHikeFromlastYear	% order value increase vs last year
CouponUsed	Coupons used last month
OrderCount	Orders placed last month
DaySinceLastOrder	Days since most recent order
CashbackAmount	Average cashback received
CityTier	City tier 1–3 (ordinal: metro → smaller city)

Categorical (5) — one-hot encoded

PreferredLoginDevice, PreferredPaymentMode, Gender, PreferedOrderCat, MaritalStatus

Binary (1) — passed through

Complain — customer raised a complaint in the last month

Missing Values

Seven numeric features have 4.5–5.5% missing values, all handled by median imputation fitted on the training set only.

Feature	Missing %
DaySinceLastOrder	5.5%
OrderAmountHikeFromlastYear	4.7%
Tenure	4.7%
OrderCount	4.6%
CouponUsed	4.5%
HourSpendOnApp	4.5%
WarehouseToHome	4.5%

---

Project Structure

ml_analysis_churn/
├── pyproject.toml
├── data/
│   ├── raw/ecommerce_churn.xlsx
│   └── processed/                  # train/cal/test parquet
├── models/                         # 8 pickles: {model}.pkl + {model}_cal.pkl
├── outputs/figures/                # All generated plots
├── src/churn/
│   ├── config.py                   # Single source of truth: paths, constants, column lists
│   ├── data_loader.py              # Download (kagglehub), load, normalise, 3-way split
│   ├── eda.py                      # Plot helpers
│   ├── preprocessing.py            # ColumnTransformer pipeline
│   ├── models.py                   # MODEL_REGISTRY, tuning, CalibratedPipeline, I/O
│   ├── evaluation.py               # Proper scoring rules, ECE, calibration + threshold plots
│   └── shap_analysis.py            # TreeExplainer/LinearExplainer, beeswarm/bar/dependence/waterfall
└── notebooks/
    ├── 01_eda.ipynb
    ├── 02_preprocessing.ipynb
    ├── 03_modeling.ipynb
    └── 04_evaluation_shap.ipynb

---

Pipeline

Raw XLSX  →  01_eda            Distribution audits, missing values, churn rate by feature
          →  02_preprocessing  3-way stratified split; ColumnTransformer fit on train only
          →  03_modeling        Baseline CV → RandomizedSearchCV (n_iter=50) → train on full
                                train set → beta/sigmoid calibration on held-out cal set
          →  04_evaluation      Proper scoring rules, ECE, reliability diagrams (pre/post),
                                ROC/PR curves, threshold analysis, SHAP (global + local)

---

EDA Findings

Tenure: Non-linear Churn Threshold

Tenure band	Churn rate
0–6 months	~33%
6–12 months	~6%
12–24 months	~6%
24+ months	~0%

New customers churn at six times the average rate. The drop near the 6-month mark is a sharp step, not a gradual curve — this is why linear models fail to capture it (LogReg PR-AUC 0.679 vs CatBoost 0.983). The SHAP dependence plot confirms this threshold effect.

Complain: Strongest Discrete Signal

Customers who raised a complaint in the past month churn at dramatically higher rates. This binary flag is among the top-3 SHAP features for all tree models and functions as a near-deterministic churn indicator in the short term.

CashbackAmount: Non-linear Effect

Both very low and very high cashback correlate with elevated churn. Moderate cashback recipients are the most retained — suggesting the current cashback policy may be either underpowered or attracting price-sensitive discount seekers with low loyalty.

Key Correlations

• Tenure ↔ Churn: −0.35 (strongest linear relationship)
• OrderCount ↔ CouponUsed: 0.75 (high collinearity — handled by regularisation and tree splits)
• CashbackAmount ↔ DaySinceLastOrder: 0.50

---

Modelling

Four Models, One Pipeline Interface

All models share the same sklearn Pipeline(preprocessor → classifier). CatBoost's native categorical handling is intentionally bypassed in favour of OHE for a uniform API across all models.

Model	Imbalance handling
LogisticRegression	class_weight='balanced'
RandomForestClassifier	class_weight='balanced'
XGBClassifier	scale_pos_weight computed from y_train (~4.95)
CatBoostClassifier	auto_class_weights='Balanced'

Three-Way Split

The dataset is split once at the start into three non-overlapping sets. No data crosses boundaries:

• Train (70%) — hyperparameter search with 5-fold StratifiedKFold
• Calibration (10%) — fits the calibration layer only; never seen during training
• Test (20%) — final evaluation; never touched until notebooks/04

Hyperparameter Tuning

RandomizedSearchCV with n_iter=50, scoring='average_precision' (PR-AUC). PR-AUC is chosen over ROC-AUC because it focuses on the minority class — ROC-AUC is misleadingly inflated under class imbalance.

Best cross-validation PR-AUC by model:

Model	Baseline CV	Tuned CV	Δ
CatBoost	0.893	0.928	+0.035
XGBoost	0.902	0.910	+0.008
RandomForest	0.901	0.909	+0.008
LogReg	0.703	0.706	+0.003

Post-Hoc Probability Calibration

After tuning, a calibration layer is fitted on the held-out calibration set using a custom CalibratedPipeline class (scikit-learn 1.6 removed CalibratedClassifierCV(cv='prefit')):

• Beta calibration [1] for tree models — logistic regression on [log(p), log(1−p)], allowing independent correction of both ends of the probability range. Unlike Platt scaling, which forces symmetric correction, beta calibration handles asymmetric miscalibration (the typical pattern for overconfident tree ensembles).
• Platt scaling (sigmoid) for LogisticRegression — already approximately calibrated; a single logistic layer is sufficient.

Calibration quality is validated via reliability diagrams (quantile binning) and Expected Calibration Error. All four calibrated models lie close to the diagonal; the tree models show meaningful improvement over their uncalibrated counterparts.

Evaluation Metrics

Proper scoring rules (both calibration and sharpness matter):

Metric	Description
Brier Score	MSE of predicted probabilities
Brier Skill Score	Normalised Brier vs naive baseline (predict the base rate for every sample)
Log Loss	Cross-entropy; heavily penalises confident wrong predictions

Calibration quality:

Metric	Description
ECE	Expected Calibration Error — bin-level weighted gap between predicted confidence and observed accuracy
Reliability Diagram	Visual check: predicted probability vs observed frequency, quantile bins

---

SHAP Interpretability

SHAP values for the best model (CatBoost) are computed using TreeExplainer on a 500-sample test subset. TreeExplainer gives exact Shapley values by exploiting the tree structure — no approximation.

Global Feature Importance

Top features by mean |SHAP|:

1. Tenure — dominant feature. New customers (Tenure < 6) have SHAP values of +3 to +4.5 (pushed strongly toward churn). Long-tenure customers cluster around −2.
2. Complain — discrete signal with a consistently large positive SHAP effect when present.
3. NumberOfAddress — more saved addresses correlates with higher churn, possibly indicating platform comparison-shopping behaviour.
4. DaySinceLastOrder — longer gaps increase predicted churn probability.
5. CashbackAmount — non-linear: both extremes increase churn risk.
6. SatisfactionScore — weaker than Tenure or Complain; dissatisfied customers who haven't complained yet.

Feature importance rankings are broadly consistent across all four models (visible in the per-model SHAP bar plots).

Tenure Dependence Plot

The SHAP dependence plot for Tenure confirms a sharp threshold near 6 months — SHAP drops from ~+4 (new customers) to ~−2 (established customers) over a narrow window. The interaction colour (SatisfactionScore) shows minimal interaction, indicating the tenure effect is largely orthogonal to satisfaction level.

Individual Predictions (Waterfall)

Waterfall plots decompose individual predictions into per-feature contributions starting from the 16.8% base rate. These are directly usable for explaining risk scores to business stakeholders or for audit purposes.

---

Business Recommendations

1. Protect new customers — Customers in their first 6 months are at 6× the average churn risk. Onboarding programmes, early check-ins, and first-purchase incentives have the highest expected retention value per pound spent.

2. Intercept complaints immediately — Complain = 1 is a near-deterministic short-term churn signal. A same-day escalation pathway for complaint-flagged accounts (compensation offer, account manager contact) can directly intercept this pathway before churn occurs.

3. Trigger re-engagement on inactivity — Rising DaySinceLastOrder among short-tenure customers is an early warning. A rule-based trigger (e.g. no order in 14 days for Tenure < 12 months) could surface at-risk customers for targeted re-engagement before they disengage fully.

4. Audit the cashback structure — The inverted-U SHAP relationship for CashbackAmount suggests the current policy is attracting a discount-seeking segment with low intrinsic loyalty at the high end, while under-incentivising another segment at the low end. A two-tier cashback design targeting moderate amounts for low-tenure customers may improve retention economics.

5. Deploy calibrated probability scores — CatBoost's calibrated probabilities are reliable (ECE = 0.017). They can be used as churn risk scores to rank customers for intervention, enabling efficient allocation of a finite retention budget rather than blanket outreach.

---

Design Decisions

Beta calibration over isotonic regression — Beta calibration [1] fits logistic regression on [log(p), log(1−p)], giving two free parameters that can independently correct the lower and upper tails. Isotonic regression is non-parametric and can overfit on small calibration sets; beta calibration's parameterisation provides more robust generalisation.

3-way split over cross-validated calibration — CalibratedClassifierCV with k-fold CV on the training set uses the same data for both tuning and calibration signal, making the final test evaluation slightly optimistic. The 3-way split makes each stage's data budget explicit and fully separates all three concerns.

PR-AUC as tuning metric — ROC-AUC is inflated under class imbalance because the large true-negative region dominates the curve. average_precision_score (PR-AUC) focuses exclusively on the positive class and correctly penalises a model that misses churners.

CityTier as ordinal numeric — CityTier takes values 1, 2, 3 with a natural ordering (metro → smaller city). One-hot encoding would destroy that ordinal relationship and add two spurious binary features. Treating it as numeric is the correct representation.

Uniform sklearn.Pipeline for all models — CatBoost's native categorical handling could marginally improve its performance. The pipeline consistency was preferred: all four models use identical preprocessing, making CV scores, tuning, and serialisation directly comparable.

Dynamic scale_pos_weight for XGBoost — Hardcoding scale_pos_weight=4 is incorrect because the actual ratio in the training split is ~4.95. The value is now computed from y_train at fit time.

---

How to Run

Prerequisites: Python 3.12+, uv, Kaggle API credentials (~/.kaggle/kaggle.json)

git clone <repo>
cd ml_analysis_churn
uv pip install -e .

Run notebooks in order (each depends on the previous):

jupyter nbconvert --to notebook --execute --inplace notebooks/01_eda.ipynb
jupyter nbconvert --to notebook --execute --inplace notebooks/02_preprocessing.ipynb
jupyter nbconvert --to notebook --execute --inplace notebooks/03_modeling.ipynb  # ~15 min
jupyter nbconvert --to notebook --execute --inplace notebooks/04_evaluation_shap.ipynb

All figures are saved automatically to outputs/figures/. Key outputs:

File	Contents
reliability_diagram.png	Calibration curves, all models
calibration_pre_post_*.png	Before/after calibration per model
roc_pr_curves.png	ROC and PR curves
threshold_curve_catboost.png	Precision/Recall/F1 vs threshold
shap_beeswarm_catboost.png	Global SHAP importance with direction
shap_dependence_Tenure_catboost.png	Tenure threshold effect
shap_waterfall_catboost_idx*.png	Individual prediction explanations

---

References

[1] Kull, M., Silva Filho, T. M., & Flach, P. (2017). Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 54:1826–1835. Available at: http://proceedings.mlr.press/v54/kull17a.html

---

License

This project is licensed under a custom Personal Use License.

You are free to:

• Use the code for personal or educational purposes
• Publish your own fork or modified version on GitHub with attribution

You are not allowed to:

• Use this code or its derivatives for commercial purposes
• Resell or redistribute the code as your own product
• Remove or change the license or attribution

For any use beyond personal or educational purposes, please contact the author for written permission.