Hybrid E-Commerce Customer Service Chatbot

A production-ready intelligent chatbot combining semantic retrieval and fine-tuned LLMs for optimal response quality and latency. Achieves 92.9% overall system score.

🎯 Project Overview

This project implements a hybrid chatbot system that intelligently routes customer queries between two specialized subsystems:

🔍 Semantic Retrieval for deterministic queries (contact info, invoices, shipping)
🤖 Fine-tuned LLM for indeterministic queries (complaints, account issues, order modifications)

Key Innovation: Rather than using a one-size-fits-all approach, we classify query types and route them to the optimal response mechanism, achieving 99.90% classification accuracy while balancing response quality and latency.

🏗️ System Architecture

User Query
    ↓
┌─────────────────────┐
│ Binary Classifier   │  ← 99.90% accuracy
│ (Logistic Reg)      │  ← ~3ms latency
└─────────────────────┘
    ↓           ↓
Deterministic  Indeterministic
    ↓           ↓
┌─────────┐  ┌──────────┐
│Retrieval│  │Fine-tuned│
│ FAISS   │  │   LLM    │
│ Index   │  │ (Phi-2)  │
└─────────┘  └──────────┘
 ~30ms       ~7000ms
    ↓           ↓
   Response

📊 Dataset

Source: Bitext Customer Support LLM Chatbot Training Dataset

Total Examples: 26,872 Q&A pairs
Categories: 11 (ACCOUNT, ORDER, CONTACT, INVOICE, FEEDBACK, etc.)
Intents: 27 unique customer intents
Binary Split:
- Deterministic (39.8%): 7,917 examples
  - Categories: CONTACT, INVOICE, SHIPPING, SUBSCRIPTION, CANCEL
  - Characteristics: Factual, template-based responses
- Indeterministic (60.2%): 11,971 examples
  - Categories: ACCOUNT, ORDER, FEEDBACK
  - Characteristics: Context-dependent, personalized responses

🚀 Key Results

Classification Performance

Metric	Score
Accuracy	99.90%
Precision (macro)	99.90%
Recall (macro)	99.89%
F1-Score (macro)	99.90%
Mean Confidence	0.9645

Retrieval System (Deterministic Queries)

K	Intent Match	Category Match
Top-1	98.8%	100.00%
Top-3	99.7%	100.00%
Latency	~30ms

Note: Slight drop from 100% is due to strict train/test separation (preventing data leakage), providing realistic performance on unseen data.

LLM Generation (Indeterministic Queries)

Metric	Score
ROUGE-1 F1	0.4787
ROUGE-L F1	0.2990
BERTScore F1	0.8895
BLEU-4	0.1442
Latency	~7000ms

🏆 Hybrid System Metrics (Weighted)

Metric	Score	Notes
Overall System Score	0.9289	Weighted combination of deterministic success & LLM quality
Routing Accuracy	99.90%	Logistic Regression Classifier
Deterministic Success	100.00%	Top-1 Intent Match (39.8% of traffic)
Indeterministic Quality	0.8819	BERTScore F1 (60.2% of traffic)

System-Level Performance

End-to-end Latency (Deterministic): ~34ms
End-to-end Latency (Indeterministic): ~7s
Overall Routing Accuracy: 99.90%

🔬 Experimental Findings

1. Retrieval vs LLM on Sparse Data

When training data is limited (450 examples):

Retrieval wins on in-distribution queries: 0.3474 ROUGE-L
LLM wins on novel/out-of-distribution queries: +60.5% improvement
Key insight: Hybrid approach needed for real-world robustness

2. Model Scaling Benefits

Tested Phi-2 (2.7B) vs Mistral-7B (7B) on sparse intents:

Mistral-7B wins: 7/10 head-to-head
+20.2% ROUGE-L improvement on novel queries
Conclusion: Larger models better at generalizing from limited examples

3. Coverage Analysis

Retrieval needs comprehensive coverage (like full Bitext dataset)
With only 450 examples: 3.8% coverage per intent → poor retrieval
LLM can generalize from sparse data → why hybrid approach works

🛠️ Technical Stack

Core Libraries

PyTorch 2.8.0 - Deep learning framework
Transformers 4.57 - Hugging Face model library
PEFT 0.17 - Parameter-efficient fine-tuning (LoRA)
Sentence-Transformers 5.1 - Semantic embeddings
FAISS - Efficient similarity search
scikit-learn - Classification and metrics

Models

Classifier: Logistic Regression with TF-IDF (1000 features)
Embeddings: all-MiniLM-L6-v2 (384 dimensions)
LLM: Microsoft Phi-2 (2.7B parameters)
- LoRA config: r=8, α=16, dropout=0.05
- Trainable params: 0.09% (2.6M / 2.78B)
- Training: 3 epochs, ~2 minutes on T4 GPU

🚦 Getting Started

Prerequisites

# Python 3.10+
# CUDA-capable GPU (recommended: T4, A100)

Installation

Install dependencies:

pip install torch transformers peft accelerate bitsandbytes
pip install sentence-transformers faiss-cpu
pip install rouge-score bert-score datasets
pip install scikit-learn pandas numpy matplotlib seaborn

2Mount Google Drive (if using Colab):

from google.colab import drive
drive.mount('/content/drive')

📈 Development Phases

Phase 0: Environment Setup ✅

GPU verification (T4/A100)
Library installation
Dataset download and inspection

Phase 1: Binary Classifier ✅

TF-IDF feature engineering
Logistic Regression training
Result: 99.90% test accuracy → Proceed ✓

Phase 2A: Retrieval System ✅

Semantic embeddings with sentence-transformers
FAISS index construction (7,717 vectors)
Result: 100% Top-3 intent match → Proceed ✓

Phase 2B: LLM Fine-tuning Pilot ✅

LoRA-based fine-tuning (Phi-2)
450 examples, 3 epochs, ~2 min training
Result: Coherent, on-topic responses → Proceed ✓

Phase 3: Resource Profiling ✅

Latency benchmarking
Memory usage analysis
Result: <10s total latency → Acceptable ✓

Phase 4: Integration Testing ✅

End-to-end pipeline validation
Mixed query testing (10 examples)
Result: No errors, routing works → Proceed ✓

Phase 5: Baseline Comparisons ✅

vs. Zero-shot LLM (all queries)
vs. Retrieval-only (all queries)
Result: Hybrid balances speed + quality ✓

Phase 6: Comprehensive Metrics ✅

ROUGE, BLEU, BERTScore evaluation
Per-category performance analysis
Confidence distribution analysis

🧪 Ablation Studies

Retrieval vs LLM Trade-offs

When Retrieval Wins:

✅ Query similar to training examples
✅ High semantic similarity (distance < 0.5)
✅ Well-represented intents in database
⚡ Ultra-fast (~30ms)

When LLM Wins:

✅ Novel query phrasings
✅ Sparse training data (< 50 examples/intent)
✅ Context-dependent responses needed
🔥 Superior generalization (+60% on sparse intents)

Optimal Strategy

Use retrieval for: contact info, invoices, shipping, policies
Use LLM for: complaints, account changes, order modifications, personalized requests

💡 Key Technical Decisions

1. LoRA over Full Fine-tuning

0.09% trainable parameters (2.6M / 2.78B)
~2 minutes training time vs hours for full fine-tuning
No degradation in performance on target domain
Easy deployment: Only store adapter weights (~100-200MB)

2. Binary Classification Approach

Simpler than multi-class routing to 27 intents
Leverages structural differences between query types
High accuracy (99.90%) with minimal features
Allows specialized optimization per subsystem

3. Small Pilot First

Validated approach on 500 examples before scaling
Identified issues early (retrieval on sparse data)
Saved compute resources
Informed full training strategy

4. Data Leakage Prevention (Master Split)

Strict 80/20 Split: Implemented a "Master Train/Test Split" at the very beginning.
Physical Separation: Saved train_dataset.csv and test_dataset.csv immediately to disk.
Zero Leakage: The test set (20%) is never touched during:
- TF-IDF Vectorization (fit only on train)
- FAISS Index Building (only train examples indexed)
- LLM Fine-tuning (only train examples used)
Realistic Metrics: Ensures reported accuracy reflects true generalization capability.

📊 Comparison with Baselines

System	Avg Latency	Deterministic Quality	Indeterministic Quality	Notes
Hybrid (Ours)	~2.5s	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	Best of both worlds
Zero-shot LLM	~2.4s	⭐⭐⭐	⭐⭐⭐	Slower, less specific
Retrieval-only	~8ms	⭐⭐⭐⭐⭐	⭐	Fast but limited

🎓 Intents Handled

Deterministic (Retrieval-based)

contact_customer_service - Customer support hours/methods
contact_human_agent - Speak with live agent
check_invoice / get_invoice - Invoice access
set_up_shipping_address / change_shipping_address - Address management
newsletter_subscription - Newsletter signup/unsubscribe
check_cancellation_fee - Policy information

Indeterministic (LLM-based)

cancel_order / change_order / track_order / place_order - Order management
create_account / edit_account / delete_account / switch_account - Account operations
recover_password / registration_problems - Account recovery
complaint / review - Customer feedback

🔧 Usage Examples

Example 1: Deterministic Query

query = "What are your customer service hours?"
result = chatbot.respond(query)

# Output:
# Route: RETRIEVAL
# Latency: 34ms
# Response: "Our customer service team is available from 
#            9:00 AM to 6:00 PM EST, Monday through Friday..."

Example 2: Indeterministic Query

query = "I need to cancel order #12345 but I'm having issues"
result = chatbot.respond(query)

# Output:
# Route: LLM_GENERATION
# Latency: 7061ms
# Response: "I understand you're experiencing difficulties 
#            canceling order #12345. I apologize for the 
#            inconvenience. Let me guide you through the 
#            cancellation process step by step..."

📈 Training Details

Binary Classifier

Algorithm: Logistic Regression
Features: TF-IDF (1000 features, 1-2 grams)
Training Size: ~14,319 examples (80% of dataset)
Training Time: <1 minute

Retrieval System

Embedding Model: all-MiniLM-L6-v2
Index Type: FAISS Flat L2
Index Size: ~6,334 Q&A pairs (80% of deterministic)
Build Time: ~5 seconds

LLM Fine-tuning

Base Model: Microsoft Phi-2 (2.7B params)
Method: LoRA (r=8, α=16)
Training Data: 450 examples (pilot) / ~9,577 (full 80% split)
Epochs: 3
Batch Size: 8 (effective)
Training Time: ~2 min (pilot) / ~48 min (full) on T4 GPU
GPU Memory: ~5.6 GB

🧮 Evaluation Metrics

ROUGE Scores (LLM Generation)

ROUGE-1:  P=0.4973  R=0.5218  F1=0.4787
ROUGE-2:  P=0.2072  R=0.2209  F1=0.2013
ROUGE-L:  P=0.3139  R=0.3257  F1=0.2990

BLEU Scores

BLEU-1: 0.3887
BLEU-2: 0.2607
BLEU-3: 0.1929
BLEU-4: 0.1442

BERTScore

Precision: 0.8879
Recall:    0.8918
F1-Score:  0.8895

🔬 Ablation Studies

Stress Test: Sparse Intents with Novel Wordings

Tested on review (26 examples) and edit_account (29 examples) with completely novel query phrasings:

System	ROUGE-L	Head-to-Head Wins
Retrieval	0.2271	3/10
LLM (Phi-2)	0.2527	6/10
LLM (Mistral-7B)	0.2729	7/10

Key Finding: LLM achieves +60.5% improvement over retrieval on sparse, novel queries (average retrieval distance: 0.952, indicating poor matches).

🎯 Why Hybrid Approach?

The Problem with Pure Retrieval

❌ Requires comprehensive coverage (100% of possible queries)
❌ Fails on novel phrasings (distance >1.0)
❌ Can't handle context-dependent questions
❌ Struggles with sparse intents (<50 examples)

The Problem with Pure LLM

❌ Slower (~7s vs ~30ms)
❌ Less consistent for factual queries
❌ Higher computational cost
❌ May hallucinate factual information

✅ Hybrid Solution Benefits

Speed where possible: 60% of queries via fast retrieval
Quality where needed: 40% via LLM for complex cases
Best of both worlds: Optimal quality-latency trade-off
Scalable: Easy to retrain classifier as query distribution shifts

🚀 Future Enhancements

Short-term

Fine-tune on full 11,971 indeterministic examples
Implement confidence threshold for hybrid routing
Deploy REST API endpoint

Long-term

Test larger models
Implement query expansion for retrieval

📝 Reproducibility

All experiments are fully reproducible:

Fixed random seeds: random_state=42 throughout
Saved checkpoints: All models saved to Google Drive
Version pinning: Specific library versions documented
Detailed logs: Training metrics saved at each step

📚 References

Bitext Customer Support Dataset: Hugging Face

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Built with ❤️ for intelligent customer service

⭐ Star this repo if you found it helpful!

Name		Name	Last commit message	Last commit date
LICENSE		LICENSE
NLP_LLM_Testing_Metrics.ipynb		NLP_LLM_Testing_Metrics.ipynb
NLP_Project_Phases.ipynb		NLP_Project_Phases.ipynb
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation