Methodological Case Study: Causal Impact of Feature Adoption on Retention

Executive Summary

This project is a technical case study using the public Telco Churn dataset (Kaggle) to demonstrate a rigorous causal inference pipeline. The analysis aims to isolate the impact of feature adoption (Tech Support) on customer churn by neutralizing a massive "Super-Customer" selection bias.

By pivoting from crude correlations to a non-parametric Causal ML framework (XGBoost + ATO Weighting), this study achieved nearly perfect group balance and successfully identified the true causal signal previously masked by customer seniority.

1. The Challenge: "Too Good to Be True" Correlations

A Naive comparison of crude churn rates suggested an overwhelming 27% performance gap between groups:

• Tech Support Group: ~15.2% Churn
• No Tech Support Group: ~41.7% Churn

The Problem: This raw difference is heavily confounded by a "Tenure Trap." Diagnostic analysis revealed that the Treated group (Tech Support users) was composed of "veteran" customers with more tenure and higher rates of long-term contracts than the control group.

2. Methodology: Bias Mitigation Pipeline

To find the true ROI of the feature, the following Causal ML pipeline was implemented:

A. Non-Parametric Propensity Scoring

Utilized an XGBoost Classifier to estimate propensity scores ($PS$). A tree-based model was chosen to automatically capture the non-linear interactions between tenure, monthly charges, and contract structure without manual feature engineering (e.g., squared terms).

B. Overlap Weighting (ATO)

Applied Average Treatment Effect on the Overlap (ATO) weights. This method was selected over standard IPW to ensure high stability and avoid "exploding weights" at the extremes of the distribution.

• Positivity: Verified a broad region of common support between groups.
• Balance: Achieved a Standardized Mean Difference (SMD) < 0.1 for all primary drivers, successfully neutralizing the "Super-Customer" advantage.

C. Robust Inference

Conducted a 1,000-iteration Bootstrap to generate 95% Confidence Intervals, ensuring the final causal lift was not a byproduct of sampling variance or outlier weights.

3. Results & Findings

The analysis demonstrates that while the "crude" 27% gap was an exaggeration driven by tenure, the true causal impact of Tech Support is both real and statistically significant.

Key Technical Achievements:

• Neutralized Tenure Bias: Reduced tenure SMD from 0.69 (unweighted) to -0.03 (weighted).
• Rigorous Validation: Confirmed that the feature adoption itself provides a meaningful lift in retention, even when comparing users with identical seniority and contract types.

4. Tech Stack & Environment

• Language: Python 3.x (Pandas, NumPy)
• ML Libraries: XGBoost, Scikit-Learn
• Visualization: Matplotlib, Seaborn
• Environment: VS Code / Jupyter Notebook