Car Weight Prediction using OLS, Ridge & Lasso Regression Project Overview
This project predicts car weight using multiple linear regression techniques and compares the performance of:
Ordinary Least Squares (OLS)
Linear Regression (Scikit-Learn)
Ridge Regression
Lasso Regression
The objective is to analyze multicollinearity, reduce overfitting, and improve model generalization using regularization techniques.
Dataset
Dataset: Cars93 Target Variable: Weight
Features Used: Price details, MPG, Engine specifications, Vehicle dimensions, Type, Airbags, DriveTrain, Cylinders, Origin, and other relevant attributes.
Tech Stack
Python
Pandas
NumPy
Matplotlib
Scikit-Learn
Statsmodels
Workflow
- Data Preprocessing
Removed irrelevant columns (id, Make, Model, Manufacturer)
Separated categorical and numerical features
Applied One-Hot Encoding to categorical variables
Applied StandardScaler to continuous variables
Performed Train-Test Split (80:20)
- OLS Regression (Statsmodels)
Added intercept using add_constant()
Evaluated:
Adjusted R²
p-values
Multicollinearity warning (Condition Number)
Performed Backward Elimination
Removed highest p-value features iteratively
Adjusted R² achieved: 0.9665
- Baseline: Linear Regression Metric Value Train MSE 8356 Test MSE 16155
Clear overfitting was observed in the baseline model.
- Ridge Regression (L2 Regularization)
Used GridSearchCV for hyperparameter tuning
Best Alpha from cross-validation: 0.9
Performed manual alpha tuning using visualization
Best Performance:
Metric Value Train MSE 10209 Test MSE 10210
Significant reduction in overfitting compared to standard Linear Regression.
- Lasso Regression (L1 Regularization)
Performed manual alpha tuning
Observed feature shrinkage behavior
Best Performance:
Metric Value Train MSE 9898 Test MSE 9897
Lasso provided the best generalization performance.
Model Comparison Model Train MSE Test MSE Overfitting Level Linear Regression 8356 16155 High Ridge Regression 10209 10210 Low Lasso Regression 9898 9897 Very Low Key Insights
Linear Regression suffered from overfitting due to multicollinearity.
A high condition number indicated strong multicollinearity in the dataset.
Ridge regression reduced coefficient magnitudes and improved stability.
Lasso regression performed implicit feature selection and achieved better generalization.
Regularization significantly improved model robustness and reduced variance.
Concepts Demonstrated
Multiple Linear Regression
Backward Elimination
Multicollinearity Analysis
Bias–Variance Tradeoff
Ridge and Lasso Regularization
Hyperparameter Tuning
Model Evaluation using Mean Squared Error
Future Improvements
Add cross-validation comparison for all models
Implement ElasticNet regression
Perform detailed residual analysis
Compare R² scores across models
Deploy the model using Streamlit
Author
Divya Jagtap Computer Engineering Student Interested in Machine Learning and Data Science