Comparative Study of Classical ML Classifiers

A from-scratch comparison of 7 classification algorithms — Logistic Regression, SVM (Linear & RBF), Naive Bayes, Decision Tree, Random Forest, AdaBoost, and XGBoost — across two datasets with deliberately contrasting properties, to understand not just which model wins, but why.

Most tutorials stop at a metrics table. This project goes one level deeper: every result is traced back to the algorithm's underlying assumptions (linearity, feature independence, bias-variance trade-off) using VIF analysis, SHAP interpretability, and 2D decision boundary visualization.

---

Why Two Datasets?

A single dataset only tells you "X beat Y here." Two datasets with opposite characteristics let you build an actual theory of when each algorithm wins:

	Breast Cancer Wisconsin	Telco Customer Churn
Size	569 rows, 30 features	7,043 rows, 30 features (post-encoding)
Class balance	~63% / 37% (mild)	~74% / 26% (moderate imbalance)
Feature type	All continuous, numeric	Mixed — numeric + categorical (one-hot/label encoded)
Key challenge	Severe multicollinearity (VIF > 1000)	Class imbalance + a real-world data-cleaning trap
What it tests	Does added model complexity help when the boundary is already near-linear?	Does the "best" model change once imbalance and recall — not accuracy — become the priority?

---

Tech Stack

• Language: Python 3
• Environment: Google Colab (Google Drive-mounted datasets)
• Data handling: pandas, numpy
• Visualization: matplotlib, seaborn
• Modeling: scikit-learn (Logistic Regression, SVM, Naive Bayes, Decision Tree, Random Forest, AdaBoost), xgboost
• Diagnostics: statsmodels (VIF / multicollinearity), sklearn.decomposition.PCA (dimensionality reduction & 2D visualization)
• Interpretability: shap (TreeExplainer, LinearExplainer)

---

Project Pipeline

Each dataset follows the same structured pipeline, built cell-by-cell in a single notebook:

1. EDA — univariate/bivariate distributions, class balance, correlation heatmaps, PCA-projected class separability
2. Data Cleaning & Preprocessing — missing value diagnosis (not just imputation — understanding why values are missing), multicollinearity check (VIF), encoding, stratified train/val/test split, scaling (fit on train only)
3. Modeling — all 7 algorithms trained and evaluated on a held-out validation set, with class-imbalance handling explored per algorithm where relevant
4. Test Set Evaluation — a single, one-time evaluation on the untouched test set, after every model's configuration was already locked in from validation
5. Interpretability — SHAP value comparison across models, PCA-based 2D decision boundary visualization
6. Final Model Selection — justified by the metric that actually matters for each problem, not blanket accuracy

---

Dataset 1: Breast Cancer Wisconsin (Diagnostic)

Problem type: Binary classification (Malignant vs Benign), clean/balanced, near-linearly separable.

Key Findings

• Severe multicollinearity confirmed via VIF (radius_mean VIF ≈ 63,000) — radius, perimeter, and area are geometrically redundant, directly threatening Naive Bayes' independence assumption.
• Naive Bayes underperformed (95.3% test accuracy) relative to every other model — direct, measurable evidence of what happens when a model's core assumption is violated.
• A single Decision Tree plateaued around 93–94% regardless of pruning — pruning fixed overfitting (train/val gap shrank) but did not raise the accuracy ceiling, showing the real limitation was the model's rigid, axis-aligned splitting, not overfitting.
• Logistic Regression matched the best ensemble methods without any added complexity — strong evidence the true decision boundary is close to linear.

Final Test Set Results

Model	Accuracy	Precision	Recall	F1	ROC-AUC
XGBoost	1.000	1.000	1.000	1.000	1.000
Logistic Regression	0.988	1.000	0.969	0.984	0.998
SVM (RBF)	0.988	1.000	0.969	0.984	0.999
SVM (Linear)	0.977	1.000	0.938	0.968	0.993
AdaBoost	0.977	0.969	0.969	0.969	0.999
Random Forest	0.965	1.000	0.906	0.951	0.999
Naive Bayes	0.953	0.967	0.906	0.935	0.995
Decision Tree (Pruned)	0.942	0.966	0.875	0.918	0.936

(XGBoost's perfect score is flagged as likely a small-test-set artifact — 86 samples — rather than genuine superiority, since it doesn't hold a consistent edge over LR/SVM-RBF across validation and test.)

✅ Chosen Model: Logistic Regression

Matches the top ensemble methods on every key metric while remaining fully interpretable (clear feature coefficients), fast to train, and simple to deploy — ideal properties for a clinical decision-support context.

---

Dataset 2: Telco Customer Churn

Problem type: Binary classification (Churn vs No Churn), moderately imbalanced, mixed categorical/numeric features.

Key Findings

• Diagnosed a real-world data-cleaning trap: TotalCharges was stored as a string with 11 blank entries invisible to a plain .isnull() check — traced to tenure = 0 (new customers with no billing history yet), and imputed with 0 based on that logical understanding rather than blind mean/median imputation.
• Contract type was the single strongest churn predictor (month-to-month: 42.7% churn vs two-year: 2.8%) — confirmed independently through EDA crosstabs, Logistic Regression coefficients, and Random Forest feature importances.
• Recall — not accuracy — was identified as the business-critical metric: missing an actual churner (false negative) costs far more than a wasted retention offer (false positive). Every model was therefore re-evaluated with class_weight='balanced' (or its algorithm-specific equivalent).
• Imbalance-handling is not one-size-fits-all — several non-obvious failures were diagnosed and fixed:
  • class_weight='balanced' failed silently on unrestricted Random Forest (recall barely moved) until tree depth was also restricted — overfitting was masking the reweighting's effect.
  • Setting class_weight='balanced' directly on AdaBoost's base estimator broke training entirely (ValueError) — fixed by computing sample_weight upfront via compute_sample_weight instead, since AdaBoost's own reweighting loop conflicts with a pre-weighted base learner.
  • XGBoost's scale_pos_weight lost effectiveness as n_estimators increased (recall dropped from 80% at 10 trees to ~60% at 100 trees) — more boosting rounds let the model re-fit majority-class patterns, gradually eroding the reweighting's intended effect.

Final Test Set Results (sorted by Recall)

Model	Accuracy	Precision	Recall	F1	ROC-AUC
Naive Bayes (Balanced Priors)	0.639	0.416	0.900	0.569	0.814
SVM Linear (Balanced)	0.688	0.452	0.832	0.585	0.825
AdaBoost (Balanced)	0.757	0.526	0.829	0.644	0.832
XGBoost (Balanced, n_estimators=10)	0.747	0.515	0.818	0.632	0.844
Logistic Regression (Balanced)	0.743	0.509	0.811	0.625	0.849
Decision Tree (Balanced)	0.756	0.526	0.807	0.637	0.837
SVM RBF (Balanced)	0.742	0.508	0.800	0.621	0.827
Random Forest (Balanced + Restricted Depth)	0.747	0.515	0.800	0.627	0.850

✅ Chosen Model: AdaBoost (Balanced)

Catches ~83% of actual churners while remaining precise enough to be operationally useful, and posts one of the highest ROC-AUC scores in the comparison — the best practical balance between "catch enough churners to act on" and "don't drown the retention team in false alarms."

---

The Core Takeaway

Dataset	Winning Model	Why
Breast Cancer (clean, balanced, linear)	Logistic Regression	Simple model was sufficient — added complexity bought nothing
Telco Churn (imbalanced, mixed-type, non-linear)	AdaBoost	Bias-reducing boosting + explicit imbalance handling was necessary

There is no universally "best" classical ML algorithm. The right model depends entirely on the data's underlying structure — linearity, feature independence, class balance — and this project set out to demonstrate that with evidence, not just assert it.

---

Repository Structure

├── Classification_algorithms_comparison_breast_cancer_dataset.ipynb
├── Classification_algorithms_comparison_telco_churn_dataset.ipynb
└── README.md

How to Run

Both notebooks were built for Google Colab with datasets loaded from Google Drive:

1. Download Breast Cancer Wisconsin (Diagnostic) and Telco Customer Churn datasets.
2. Upload them to your Google Drive and update the file paths in the first few cells.
3. Run all cells top to bottom — each notebook is self-contained (EDA → preprocessing → modeling → interpretability → final model selection).