🧬 Scientific Mission

Achieving high generalization on Scaffold Splits is the gold standard for drug discovery. Standard random splits leak structural information — the model memorizes scaffolds rather than learning chemistry. This study optimizes Mistral-7B for BACE-1 inhibitor prediction using two distinct training strategies, isolating the exact conditions that produce peak generalization on unseen molecular scaffolds.

---

📊 Results — Both Experiments

Experiment	Strategy	ROC-AUC	Split
Experiment 1 (bace_mistral_benchmark.py)	Step-based, 2500 steps, peak at Step 500	0.8034	Scaffold
Experiment 2 (mistral-bace-epoch5.ipynb)	Epoch-based, 5 epochs, peak at Epoch 4	0.8371 ✅	Scaffold

Final Best: 0.8371 ROC-AUC — Epoch 4, Scaffold Split, Kaggle T4

Key finding: Epoch-based training with explicit overfitting detection outperforms fixed step-based training. The model peaks at Epoch 4 (0.8371) and begins overfitting at Epoch 5 (0.8042) — confirming the importance of early stopping tied to scaffold-split evaluation rather than training loss.

---

🔬 Experiment Details

Experiment 1 — Step-Based Peak Detection

• File: bace_mistral_benchmark.py
• Strategy: Evaluate every 250 steps, capture peak before overfitting
• LoRA: r=8, lora_alpha=16, dropout=0.2
• Balancing: 5x oversampling of positive class
• Peak: Step 500 → ROC-AUC 0.8034

Experiment 2 — Epoch-Based with Overfitting Guard ✅ Final

• File: mistral-bace-epoch5.ipynb
• Strategy: 5 epochs, save checkpoint on every improvement
• LoRA: r=8, lora_alpha=16, target q_proj + v_proj
• Training curve:

Epoch	ROC-AUC	Status
1	0.6933	Learning
2	0.7908	Improving ↑
3	0.8167	New record ↑
4	0.8371	BEST ✅
5	0.8042	Overfitting ↓

Scientific insight: Overfitting begins exactly at Epoch 5 — this validates that scaffold-split evaluation must be used as the early-stopping criterion, not training loss.

---

🛠️ Tech Stack

---

📂 File Index

File	Description
bace_mistral_benchmark.py	Experiment 1 — step-based training, ROC-AUC 0.8034
mistral-bace-epoch5.ipynb	Experiment 2 — epoch-based, best ROC-AUC 0.8371
requirements.txt	Environment dependencies

---

💻 Execution

pip install -r requirements.txt

# Experiment 1 — step based
python bace_mistral_benchmark.py

# Experiment 2 — open in Kaggle
# kaggle.com/sameernadeem66/mistral-bace-epoch5

---

🔗 Part of DeepChem GSoC 2026 Research

This benchmark is part of a complete pre-proposal research suite:

Task	Model	Result	Link
BACE Classification	Mistral-7B QLoRA	0.8371 ROC-AUC	This repo
ClinTox Classification	Mistral-7B QLoRA	0.9913 ROC-AUC	[ClinTox Repo]
BBBP Classification	OLMo-7B QLoRA	0.7100 ROC-AUC	[BBBP Repo]
ESOL Regression	OLMo-7B + Reg Head	0.8582 RMSE	[ESOL Repo]
SMILES Generation	OLMo-7B + RDKit TSM	20/20 = 100% valid	[Generation Repo]