Reproducibility Guide - AI Fashion Assistant v2.5

Complete guide to reproducing all results from v2.0 through v2.5

This document provides step-by-step instructions to reproduce:

Search performance metrics (NDCG, Recall, MRR)
RAG pipeline results
AI agent evaluations
Personalization metrics
User study results (SUS 84.50)
Full-stack application deployment

Environment Setup

Prerequisites

# Python version
Python 3.10 or higher

# Check environment.json for exact versions
cat environment.json

Installation

# Clone repository
git clone https://github.com/haticebaydemir/ai-fashion-assistant-v2.git
cd ai-fashion-assistant-v2

# Install core dependencies
pip install -r requirements.txt

Configuration

All experiments use configuration files in configs/.

Default configuration: configs/default.yaml

To run with a specific config:

import yaml
with open('configs/default.yaml') as f:
    config = yaml.safe_load(f)

Random Seeds

Set random seed for reproducibility:

import numpy as np
import torch

seed = config.get('random_seed', 42)
np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

🎯 Version-Specific Reproduction

v2.0 Baseline Results

Target Metrics:

NDCG@10: 97.43%
MRR: 100%
Recall@10: 51.11%
Response time: <50ms

Steps:

Navigate to baseline directory:

cd v2.0-baseline/

Follow setup instructions:

# See v2.0-baseline/README.md for details

Run evaluation notebooks:

# Open Jupyter
jupyter notebook

# Navigate to:
research/notebooks/phase9_evaluation/

# Run all evaluation notebooks sequentially

Check results:

# Results saved in:
evaluation/results/

# Key files:
# - baseline_metrics.csv
# - fusion_metrics.csv
# - performance_comparison.png

Expected Time: 2-3 hours (includes model downloads)

Variance: ±0.5% on metrics due to FAISS approximation

v2.1 Visual Attributes

Target Results:

307,720 attributes extracted
95.4% product coverage
10 semantic categories
Avg 6.93 attributes per product

Steps:

Navigate to v2.1 directory:

cd v2.1-core-ml-plus/

Run attribute extraction:

jupyter notebook notebooks/01_visual_attributes_extraction.ipynb

Check results:

# Output file:
evaluation/results/product_attributes.csv

# Verify:
# - Total rows: 307,720
# - Unique products with attributes: ~42,388 (95.4%)
# - Categories: 10 (pattern, fit, length, etc.)

Expected Time: 1-2 hours

Note: CLIP inference requires GPU for faster processing (CPU: 4-6 hours)

v2.2 RAG Pipeline

Target Metrics:

Average RAG score: 0.714
Response time: 0.89s
30 test queries evaluated

Steps:

Set up environment:

cd v2.2-rag-langchain/

# Set GROQ API key
export GROQ_API_KEY="your_api_key_here"
# Or create .env file

Run evaluation:

jupyter notebook notebooks/03_evaluation.ipynb

Check results:

evaluation/results/
├── evaluation_results.csv      # Per-query scores
├── evaluation_stats.json       # Summary statistics
├── category_performance.csv    # By category
└── score_distribution.png      # Visualization

Expected Time: 30 minutes

API Requirements:

GROQ API key (free tier: 14,400 requests/day)
Results may vary ±5% due to LLM non-determinism

Troubleshooting:

Rate limit: Wait or use free tier limit
API errors: Check GROQ_API_KEY environment variable

v2.3 AI Agents

Target Metrics:

Success rate: 100%
Tool usage rate: 100%
Avg response time: 2.6s
Memory efficiency: 92%

Steps:

Setup:

cd v2.3-ai-agents-langchain/

# Ensure GROQ_API_KEY is set
echo $GROQ_API_KEY

Run comprehensive evaluation:

jupyter notebook notebooks/04_final_evaluation.ipynb

Verify results:

evaluation/results/
├── final_evaluation_results.csv
├── final_evaluation_complete.png
├── v2.3_final_summary.json
└── conversation_memory_results.json

# Check metrics match:
# - success_rate: 1.0 (100%)
# - tool_usage_rate: 1.0 (100%)
# - avg_response_time: ~2.6s

Expected Time: 1 hour

Note:

LangChain 0.1.20 required
Results consistent across runs for same inputs

v2.4 Personalization

Target Metrics:

Preference match: 76.7%
Personalization latency: 11.92ms
Coverage: 100%
Personalization rate: 83.3%

Steps:

Navigate and run:

cd v2.4-complete/

jupyter notebook notebooks/02_personalization_engine.ipynb

Check outputs:

evaluation/results/
├── personalization_metrics.csv     # Per-user metrics
├── recommendations_U001.csv        # User 1 recommendations
├── recommendations_U002.csv        # User 2 recommendations
└── recommendations_U003.csv        # User 3 recommendations

# Verify:
# - Avg preference_match: 76.7%
# - Avg latency: 11.92ms
# - Coverage: 100%

Expected Time: 30 minutes

Data Requirements:

Test user profiles in data/users/
Pre-computed embeddings from v2.0

v2.4.5 Multimodal RAG

Target Metrics:

Response time: 0.64s (28% faster than v2.2)
Visual keywords: 7.6 per response
Visual keyword rate: 100%
Multimodal unique products: 6.0

Steps:

Run all notebooks sequentially:

cd v2.4.5-multimodal-rag/

# Run notebooks in order:
jupyter notebook notebooks/01_multimodal_rag_architecture.ipynb
jupyter notebook notebooks/02_image_query_processing.ipynb
jupyter notebook notebooks/03_multimodal_retrieval.ipynb
jupyter notebook notebooks/04_visual_aware_rag.ipynb
jupyter notebook notebooks/05_evaluation_metrics.ipynb
jupyter notebook notebooks/06_final_documentation.ipynb

Verify results:

evaluation/results/
├── rag_quality_metrics.json
├── performance_visualization.png
├── v2.4.5_comprehensive_results.xlsx
└── final_summary.json

# Check key metrics:
# - avg_response_time: 0.642s
# - avg_visual_keywords: 7.6
# - visual_keyword_rate: 1.0 (100%)

Expected Time: 2 hours

Note: Requires CLIP model and GROQ API

v2.5 User Study

Target Results:

SUS Score: 84.50 (Grade A)
92% usage intent
n=25 participants

Reproduction Options:

Option A: Analyze Existing Data (Recommended)

Read comprehensive report:

cat USER_STUDY_RESULTS.md

Verify calculations:

# SUS Calculation
# Odd questions (1,3,5,7,9): score = answer - 1
# Even questions (2,4,6,8,10): score = 5 - answer
# Total SUS = sum(all_scores) * 2.5

# Example for one participant:
scores = [
    (4 - 1),  # Q1
    (5 - 2),  # Q2
    (5 - 1),  # Q3
    (5 - 1),  # Q4
    (4 - 1),  # Q5
    (5 - 2),  # Q6
    (5 - 1),  # Q7
    (5 - 2),  # Q8
    (4 - 1),  # Q9
    (5 - 2),  # Q10
]
sus_score = sum(scores) * 2.5
print(f"SUS Score: {sus_score}")  # Should be 80-100 for good UX

Check raw data (if provided):

# If you have access to:
data/user_study/responses.csv

Expected Time: 1 hour (data analysis)

Option B: Run New User Study

Deploy full-stack application (see next section)
Recruit 20-30 participants
Create Google Forms with SUS questions:

Scale: 1 (Strongly Disagree) to 5 (Strongly Agree)

Q1: I think I would like to use this system frequently
Q2: I found the system unnecessarily complex
Q3: I thought the system was easy to use
Q4: I think I would need technical support to use this
Q5: I found the various functions well integrated
Q6: I thought there was too much inconsistency
Q7: I imagine most people would learn quickly
Q8: I found the system very cumbersome to use
Q9: I felt very confident using the system
Q10: I needed to learn a lot before I could use this

Calculate SUS:

def calculate_sus(responses):
    """
    responses: list of 10 answers (1-5)
    """
    odd_sum = sum(responses[i] - 1 for i in [0,2,4,6,8])
    even_sum = sum(5 - responses[i] for i in [1,3,5,7,9])
    sus = (odd_sum + even_sum) * 2.5
    return sus

# Example:
participant_responses = [4, 2, 5, 1, 4, 2, 5, 1, 4, 1]
print(calculate_sus(participant_responses))  # Output: 85.0

Analyze results:

import pandas as pd
import numpy as np

# Load responses
df = pd.read_csv('responses.csv')

# Calculate SUS for each participant
df['sus_score'] = df.apply(lambda row: calculate_sus(row[['Q1','Q2',...,'Q10']].values), axis=1)

# Statistics
print(f"Mean SUS: {df['sus_score'].mean():.2f}")
print(f"Median SUS: {df['sus_score'].median():.2f}")
print(f"Std Dev: {df['sus_score'].std():.2f}")

# Grade distribution
def sus_grade(score):
    if score >= 80.3: return 'A'
    if score >= 68: return 'B'
    if score >= 52: return 'C'
    if score >= 25: return 'D'
    return 'F'

df['grade'] = df['sus_score'].apply(sus_grade)
print(df['grade'].value_counts())

Expected Time: 1-2 weeks (full study)

Variance: ±5 points normal for SUS

🚀 Full-Stack Application

Target: Deploy working application matching screenshots

Prerequisites

Python 3.10+
Node.js 18+
MongoDB (local or Atlas)

Backend Setup

cd AI-Fashion-fullstack/backend

# Create virtual environment
python -m venv venv

# Activate
# Windows:
venv\Scripts\activate
# Linux/Mac:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Copy data files
# Option 1: Use batch script (Windows)
copy_data.bat

# Option 2: Manual copy (Linux/Mac)
# Copy embeddings/ and models/ from v2.0-baseline/
cp -r ../../v2.0-baseline/embeddings/ ./data/
cp -r ../../v2.0-baseline/models/ ./data/

# Set environment variables
cp .env.example .env

# Edit .env file:
# MONGODB_URI=mongodb://localhost:27017/  # or MongoDB Atlas URI
# GROQ_API_KEY=your_groq_key
# JWT_SECRET=your_secret_key

# Start backend
uvicorn main:app --reload

Backend should be running on: http://localhost:8000

Frontend Setup

cd AI-Fashion-fullstack/frontend

# Install dependencies
npm install

# Set environment
cp .env.example .env

# Edit .env:
# VITE_API_URL=http://localhost:8000

# Start development server
npm run dev

Frontend should be running on: http://localhost:5173

Verification Checklist

✅ Backend Health Check:

curl http://localhost:8000/health
# Expected: {"status": "healthy"}

✅ Test Search:

curl -X POST http://localhost:8000/api/search \
  -H "Content-Type: application/json" \
  -d '{"query": "blue dress", "limit": 5}'
# Expected: JSON with 5 products

✅ Frontend Tests:

Open http://localhost:5173
Register new user → Should work
Login → Should redirect to home
Search "blue dress" → Should return results in <1s
Try image search → Should work
Use chat → Should get AI responses
Add favorites → Should persist
Update profile → Should save

Common Issues

Issue: MongoDB connection failed

ServerSelectionTimeoutError

Solution:

Check MongoDB is running: mongod --version
Verify MONGODB_URI in .env
For Atlas: Check IP whitelist

Issue: FAISS dimension mismatch

AssertionError: Dimension mismatch

Solution:

Ensure embeddings are from correct version
Check search_engine.py padding logic

Issue: GROQ API errors

Rate limit exceeded

Solution:

Check GROQ_API_KEY is valid
Free tier: 14,400 requests/day
Wait or upgrade plan

Expected Time: 1-2 hours (first-time setup)

📊 Expected Results Summary

Version	Key Metric	Expected Result	Variance
v2.0	NDCG@10	97.43%	±0.5%
v2.1	Attributes	307K	±100
v2.2	RAG Score	0.714	±0.05
v2.3	Success Rate	100%	0%
v2.4	Preference Match	76.7%	±5%
v2.4.5	Response Time	0.64s	±0.1s
v2.5	SUS Score	84.50	±5

Variance Notes

Search metrics (v2.0, v2.1): Very stable, ±0.5% expected
LLM-based metrics (v2.2, v2.3): ±5% due to non-determinism
User study (v2.5): ±5 points normal for SUS
Timing metrics: ±10% depending on hardware

🔬 Advanced Reproduction

Running Experiments

Start experiment:

from src.utils import ExperimentTracker

tracker = ExperimentTracker()
tracker.start_experiment('my_experiment', config)

Run your code
Log metrics:

tracker.log_metric('recall@10', value)
tracker.log_metric('ndcg@10', value)

End experiment:

tracker.end_experiment()

Experiment Logs

All experiments saved in experiments/ directory.

Each folder contains:

config.yaml: Configuration used
experiment.json: Metrics and artifacts
logs/: Detailed logs

Data Versions

Data files tracked in config under data section.

Ensure correct versions:

# Check data hash
import hashlib

def hash_file(filepath):
    sha256 = hashlib.sha256()
    with open(filepath, 'rb') as f:
        for chunk in iter(lambda: f.read(4096), b''):
            sha256.update(chunk)
    return sha256.hexdigest()

# Verify
expected_hash = config['data']['hash']
actual_hash = hash_file('data/meta_ssot.csv')
assert expected_hash == actual_hash, "Data file mismatch!"

🐛 Troubleshooting

Python Version Mismatch

Symptom: Package installation errors

Solution:

python --version  # Check version
# Should be 3.10+

# If different, use pyenv or conda
conda create -n fashion python=3.10
conda activate fashion

Package Version Conflicts

Symptom: Import errors

Solution:

# Use exact versions
pip install -r requirements.txt --force-reinstall

# Check installed versions
pip list

GPU vs CPU Differences

Symptom: Slightly different results

Solution:

Expected: Some operations differ between GPU/CPU
Variance: Usually <1%
For exact reproduction: Use same hardware type

Results Don't Match Exactly

Checklist:

✅ Python version matches?
✅ Package versions match requirements.txt?
✅ Random seed set correctly?
✅ Data file versions correct?
✅ GPU vs CPU noted?
✅ LLM variance expected (<5%)?

If still different:

Document variance in issues
Provide system info (Python, OS, hardware)
Share config and logs

📖 Additional Resources

Documentation

Main README: Comprehensive overview
Version READMEs: Specific to each version
CHANGELOG: Version history
USER_STUDY_RESULTS: Detailed SUS analysis

Code Examples

v2.0-baseline/research/notebooks/: 30+ examples
v2.1-core-ml-plus/notebooks/: Visual attributes
v2.2-rag-langchain/notebooks/: RAG pipeline
All notebooks fully documented

Support

GitHub Issues: Report problems
README FAQ: Common questions
Code comments: Inline documentation

🎯 Reproducibility Checklist

Before starting:

Python 3.10+ installed
Git repository cloned
Dependencies installed (pip install -r requirements.txt)
MongoDB running (if testing full-stack)
GROQ API key set (if testing LLM features)

For each version:

After completion:

All metrics within expected variance
Full-stack application deployed
Screenshots match documentation
Issues reported (if any)

✅ Success Criteria

Reproduction successful if:

✅ All metrics within expected variance
✅ Full-stack application runs
✅ No critical errors
✅ Documentation clear and helpful

Last Updated: January 17, 2026
Version: 2.5
Maintainer: Hatice Baydemir
Contact: GitHub Issues

For detailed API documentation, see individual version READMEs.

FilesExpand file tree

REPRODUCIBILITY.md

Latest commit

History

REPRODUCIBILITY.md

File metadata and controls

Reproducibility Guide - AI Fashion Assistant v2.5

📋 Quick Navigation

Environment Setup

Prerequisites

Installation

Configuration

Random Seeds

🎯 Version-Specific Reproduction

v2.0 Baseline Results

v2.1 Visual Attributes

v2.2 RAG Pipeline

v2.3 AI Agents

v2.4 Personalization

v2.4.5 Multimodal RAG

v2.5 User Study

Option A: Analyze Existing Data (Recommended)

Option B: Run New User Study

🚀 Full-Stack Application

Prerequisites

Backend Setup

Frontend Setup

Verification Checklist

Common Issues

📊 Expected Results Summary

Variance Notes

🔬 Advanced Reproduction

Running Experiments

Experiment Logs

Data Versions

🐛 Troubleshooting

Python Version Mismatch

Package Version Conflicts

GPU vs CPU Differences

Results Don't Match Exactly

📖 Additional Resources

Documentation

Code Examples

Support

🎯 Reproducibility Checklist

✅ Success Criteria