An end-to-end MLOps pipeline that predicts whether a customer is likely to respond positively to a vehicle insurance cross-sell offer — from raw data in MongoDB to a fully automated, containerized deployment on AWS.

🚀 Overview · 🏗️ Architecture · ⚙️ Installation · ☁️ AWS Setup · 🔁 CI/CD · 📡 App Routes

---

🚀 Project Overview

This project goes beyond a one-off notebook model and simulates a production-grade MLOps workflow end to end:

• 📦 Modular, installable Python package (src) via setup.py / pyproject.toml
• 🍃 MongoDB Atlas as the source data store
• 🪵 Custom logging and exception-handling modules
• 🔄 A fully modular training pipeline — Data Ingestion → Data Validation → Data Transformation → Model Trainer → Model Evaluation → Model Pusher
• 🪣 AWS S3 used as a model registry (push/pull trained models)
• ⚡ A FastAPI web app to trigger training and serve real-time predictions
• 🔁 Fully automated CI/CD with GitHub Actions, Docker, Amazon ECR, and a self-hosted EC2 runner

---

✨ Features

	Feature	Description
🍃	Cloud Data Source	Raw data ingested directly from MongoDB Atlas
🧪	Schema-Based Validation	Automated data validation against config/schema.yaml
🛠️	Feature Engineering Pipeline	Reusable preprocessing objects persisted as .pkl artifacts
🤖	Model Training & Evaluation	Scikit-learn based training with threshold-based model acceptance
🪣	S3 Model Registry	Trained models versioned and pushed/pulled from AWS S3
⚡	FastAPI Backend	REST endpoints to trigger training and serve predictions
🐳	Containerized Deployment	Dockerized app shipped via Amazon ECR
🔁	Full CI/CD Automation	GitHub Actions + self-hosted EC2 runner auto-deploys on every push
🗂️	Timestamped Artifacts	Every pipeline run stores versioned artifacts for full traceability

---

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                     MongoDB Atlas (Raw Data)                 │
└───────────────────────────┬───────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                     TRAINING PIPELINE (src/)                 │
│                                                               │
│   Data Ingestion → Data Validation → Data Transformation     │
│                         │                                    │
│                         ▼                                    │
│                    Model Trainer                             │
│                         │                                    │
│                         ▼                                    │
│                  Model Evaluation  ───────► AWS S3            │
│                         │                (Model Registry)     │
│                         ▼                                    │
│                    Model Pusher                               │
└───────────────────────────┬───────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                 FASTAPI APPLICATION (main.py)                │
│                                                               │
│   ┌────────────┐   ┌────────────────┐   ┌─────────────────┐  │
│   │  pages.py  │   │ prediction.py  │   │  training.py    │  │
│   └────────────┘   └────────────────┘   └─────────────────┘  │
└───────────────────────────┬───────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│         Docker Image ─► Amazon ECR ─► EC2 (self-hosted        │
│                runner) ─► Live App (port 5000)                │
└─────────────────────────────────────────────────────────────┘

Training Pipeline Flow

MongoDB Collection
       │
       ▼
Data Ingestion  ──►  feature_store/data.csv  ──►  ingested/train.csv, test.csv
       │
       ▼
Data Validation  ──►  report.yaml  (schema drift & completeness checks)
       │
       ▼
Data Transformation  ──►  transformed/train.npy, test.npy
       │                  transformed_object/preprocessing.pkl
       ▼
Model Trainer  ──►  trained_model/model.pkl
       │
       ▼
Model Evaluation  ──►  compares against current production model in S3
       │
       ▼
Model Pusher  ──►  pushes accepted model to S3 (my-model-mlopsproj)

---

🗂️ Project Structure

vehicle-insurance-mlops/
│
├── .github/workflows/
│   └── aws.yaml                     # CI/CD pipeline (build → push → deploy)
│
├── app/                              # FastAPI application layer
│   ├── config.py                     # App settings (host, port, CORS, log level)
│   ├── schemas.py                    # Pydantic request/response models
│   └── routers/
│       ├── pages.py                  # Home / UI routes
│       ├── prediction.py             # /predict route
│       ├── training.py               # /training route
│       └── __init__.py
│
├── artifact/                         # Auto-generated, timestamped pipeline runs
│   └── <run_timestamp>/
│       ├── data_ingestion/
│       │   ├── feature_store/data.csv
│       │   └── ingested/{train,test}.csv
│       ├── data_validation/report.yaml
│       ├── data_transformation/
│       │   ├── transformed/{train,test}.npy
│       │   └── transformed_object/preprocessing.pkl
│       └── model_trainer/trained_model/model.pkl
│
├── config/
│   ├── model.yaml                    # Model hyperparameters / selection config
│   └── schema.yaml                   # Expected dataset schema for validation
│
├── logs/                              # Auto-generated log files
│
├── notebook/
│   ├── data.csv
│   ├── exp-notebook.ipynb            # EDA & feature engineering
│   └── mongo_DB_demo.ipynb           # Push raw dataset to MongoDB
│
├── src/
│   ├── cloud_storage/                # AWS S3 read/write utilities
│   ├── components/                   # Ingestion, validation, transformation, trainer, evaluation, pusher
│   ├── configuration/                # MongoDB & AWS connection setup
│   ├── constants/                    # Project-wide constants
│   ├── data_access/                  # MongoDB → DataFrame access layer
│   ├── entity/                       # Config / artifact / estimator entities
│   ├── exception/                    # Custom exception handling
│   ├── logger/                       # Custom logging setup
│   ├── pipeline/                     # Training & prediction pipelines
│   └── utils/                        # Shared utility functions
│
├── static/
│   ├── css/style.css
│   └── js/script.js
│
├── templates/
│   └── index.html                    # Jinja2 UI template
│
├── demo.py                            # Quick script to test logger/exception/pipeline
├── Dockerfile
├── main.py                            # FastAPI app entry point
├── requirements.txt
├── setup.py
├── pyproject.toml
└── template.py                        # Scaffolds the initial project structure

---

⚙️ Installation

Prerequisites

• Python 3.10+
• A free MongoDB Atlas account
• An AWS account (for S3 model registry & deployment)

1. Clone the repository

git clone https://github.com/sharif-abusad/vehicle-insurance-mlops.git
cd vehicle-insurance-mlops

2. Scaffold & set up the environment

# (Optional) Regenerate the project skeleton
python template.py

# Create and activate a virtual environment
conda create -n vehicle python=3.10 -y
conda activate vehicle

# Install dependencies (installs local "src" package via -e .)
pip install -r requirements.txt

# Verify local package installation
pip list

3. MongoDB Atlas Setup

1. Sign up on MongoDB Atlas and create a new project.
2. Create a cluster → select the M0 (free tier) → keep defaults → Create Deployment.
3. Create a database user with a username and password.
4. Under Network Access, add 0.0.0.0/0 to allow access from anywhere.
5. Go to Get Connection String → Drivers → select Python, version 3.6+ → copy the connection string and replace <password>.
6. Add your dataset to notebook/ and push it to MongoDB via mongo_DB_demo.ipynb.
7. Verify the upload under Atlas → Database → Browse Collections.

4. Configure environment variables

cp .env.example .env

Bash

export MONGODB_URL="mongodb+srv://<username>:<password>@<cluster-url>/?appName=Cluster0"
export AWS_ACCESS_KEY_ID="<your-aws-access-key-id>"
export AWS_SECRET_ACCESS_KEY="<your-aws-secret-access-key>"
export AWS_DEFAULT_REGION="us-east-1"

PowerShell

$env:MONGODB_URL = "mongodb+srv://<username>:<password>@<cluster-url>/?appName=Cluster0"
$env:AWS_ACCESS_KEY_ID = "<your-aws-access-key-id>"
$env:AWS_SECRET_ACCESS_KEY = "<your-aws-secret-access-key>"
$env:AWS_DEFAULT_REGION = "us-east-1"

⚠️ Security note: Never commit .env files or real credentials. Ensure .env, _env, and artifact/ are listed in .gitignore, and rotate any keys that may have been exposed.

---

🖥️ Running Locally

Sanity-check logger, exception handling & pipeline

python demo.py

Run the training pipeline

python demo.py

Each run creates a timestamped folder under artifact/ containing intermediate outputs (ingested data, transformed arrays, preprocessing object, trained model, validation report).

Start the web app

python main.py

Resource	URL
App	http://localhost:5000
Trigger Training	http://localhost:5000/training
Prediction UI	http://localhost:5000/predict

---

☁️ AWS Setup (S3 Model Registry)

1. Log in to the AWS Console, keep the region set to us-east-1.
2. Go to IAM → Create User (e.g., firstproj) → attach the AdministratorAccess policy.
3. Under the user's Security Credentials → Access Keys, create a new CLI access key and download the CSV.
4. Export the access key/secret as environment variables (see above).
5. Add the following to src/constants/__init__.py:

   MODEL_EVALUATION_CHANGED_THRESHOLD_SCORE: float = 0.02
   MODEL_BUCKET_NAME = "my-model-mlopsproj"
   MODEL_PUSHER_S3_KEY = "model-registry"

6. Create an S3 bucket (my-model-mlopsproj) in us-east-1, unchecking "Block all public access" (acknowledge the warning).
7. src/configuration/aws_connection.py and src/cloud_storage/aws_storage.py handle the AWS session and S3 push/pull logic, while src/entity/s3_estimator.py wraps the pull/push operations used by Model Evaluation and Model Pusher.

---

🔁 CI/CD Deployment (GitHub Actions + EC2 + ECR)

The pipeline is defined in .github/workflows/aws.yaml and runs in two stages:

┌─────────────────────────────┐        ┌──────────────────────────────────┐
│  Continuous Integration     │        │   Continuous Deployment           │
│  (runs-on: ubuntu-latest)   │  ───►  │   (runs-on: self-hosted / EC2)    │
│                              │        │                                    │
│  • Checkout code            │        │  • Pull latest image from ECR     │
│  • Configure AWS creds      │        │  • Stop & remove old container    │
│  • Login to Amazon ECR      │        │  • Run new container with env     │
│  • Build & push Docker image│        │    vars injected as secrets       │
└─────────────────────────────┘        └──────────────────────────────────┘

Setup Steps

1. ECR Repository — AWS Console → ECR → create repository vehicleproj in us-east-1.

2. EC2 Instance — Launch an Ubuntu Server 24.04 (free-tier eligible) instance (e.g., vehicledata-machine), allow HTTP/HTTPS traffic, attach a new key pair, and connect via EC2 Instance Connect.

3. Install Docker on EC2:

   sudo apt-get update -y
   sudo apt-get upgrade
   curl -fsSL https://get.docker.com -o get-docker.sh
   sudo sh get-docker.sh
   sudo usermod -aG docker ubuntu
   newgrp docker

4. Self-hosted GitHub Runner — In your GitHub repo, go to Settings → Actions → Runners → New self-hosted runner, select Linux, and run the provided download/configure commands on the EC2 instance. Start it with ./run.sh.

5. GitHub Secrets — Under Settings → Secrets and variables → Actions, add:

   Secret	Description
   AWS_ACCESS_KEY_ID	IAM user access key
   AWS_SECRET_ACCESS_KEY	IAM user secret key
   AWS_DEFAULT_REGION	e.g. us-east-1
   ECR_REPO	ECR repository name (vehicleproj)
   MONGODB_URL	MongoDB Atlas connection string

6. Open the App Port — In the EC2 instance's Security Group, add an inbound rule for the app port (e.g., 5000) with source 0.0.0.0/0.

7. Push to main — the workflow triggers automatically: builds, pushes to ECR, and deploys the container on EC2.

8. Access the app at http://<EC2-public-ip>:<port>.

---

📡 Application Routes

Method	Route	Description
GET	/	Home page (served via Jinja2 templates)
POST/GET	/training	Triggers the model training pipeline
POST	/predict	Serves predictions from the trained model

---

🧩 Pipeline Components

Component	Responsibility
Data Ingestion	Pulls data from MongoDB, converts to DataFrame, splits into train/test
Data Validation	Validates schema/data drift against config/schema.yaml
Data Transformation	Feature engineering, preprocessing pipeline, saves transformed arrays
Model Trainer	Trains the ML model and saves the trained estimator
Model Evaluation	Compares the new model against the production model in S3
Model Pusher	Pushes the accepted model to the S3 model registry

Each component is driven by strongly-typed config and artifact entities defined in src/entity/config_entity.py and src/entity/artifact_entity.py, keeping the pipeline modular and easy to extend.

---

🛠️ Tech Stack

Data & ML

• Pandas / NumPy — data manipulation
• Scikit-learn — modeling & preprocessing
• imbalanced-learn — handling class imbalance
• Matplotlib / Seaborn / Plotly — EDA & visualization

Backend & Serving

• FastAPI — REST API framework
• Uvicorn — ASGI server
• Jinja2 — HTML templating

Data & Cloud

• PyMongo — MongoDB Atlas connectivity
• boto3 — AWS S3 integration
• AWS S3 / IAM / ECR / EC2 — model registry & deployment infrastructure

DevOps

• Docker — containerization
• GitHub Actions — CI/CD automation (self-hosted EC2 runner)

---

🗺️ Roadmap

• Add automated data drift monitoring
• Add unit and integration tests (pytest)
• Add model explainability (SHAP)
• Add experiment tracking (MLflow / DVC)
• Add batch prediction endpoint
• Add structured logging and monitoring (Sentry / CloudWatch)
• Blue-green or canary deployment strategy on EC2

---

🤝 Contributing

Contributions are welcome and appreciated.

# 1. Fork the repository
# 2. Create a feature branch
git checkout -b feature/your-feature-name

# 3. Commit your changes
git commit -m "feat: add your feature"

# 4. Push and open a Pull Request
git push origin feature/your-feature-name

Please follow Conventional Commits for commit messages and open an issue before starting large changes.

---

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

👤 Author

Sharif Abusad 📧 abusharif866@gmail.com

If you found this project useful, consider giving it a ⭐ on GitHub — it helps a lot!

---

_{Built with ❤️ using MongoDB, FastAPI, Scikit-learn, Docker & AWS}