Synthetic dataset generation platform for training and evaluating Retrieval-Augmented Generation (RAG) systems.
🔴 Try it live: apc15-rag-spectrum.hf.space
This repo is the public overview of a private codebase. For source access or collaboration: ayushchhoker15@gmail.com
Most RAG systems are evaluated on whatever documents happen to be lying around. RAG Spectrum replaces that with systematic, controlled test data: it generates synthetic document + question/answer datasets across 25 distinct challenge types (D01–D25), each targeting a specific failure mode of RAG pipelines:
| Category | Example types |
|---|---|
| Basic retrieval | D01 atomic facts |
| Complex reasoning | D04 cross-document synthesis, D15 coreference resolution |
| Robustness | D08 OCR noise & typos, D23 noisy retrieval |
| Domain & language shift | D16 domain shift, D17 multilingual |
| Adversarial | D25 fabrication / hallucination detection |
Select any mix of the 25 types and generate them in one run, with live progress streamed to the UI.
Topic + D-type selection
│
▼
┌─────────────────────────────┐
│ 6-layer reliability stack │
│ ─ model router │
│ ─ provider orchestrator │
│ ─ circuit breaker │
│ ─ adaptive manager │
│ ─ exponential backoff retry │
│ ─ connection pooling │
└──────────────┬──────────────┘
▼
Quality assurance pipeline
content validation · triple-layer
dedup (MD5 exact + fuzzy 85% +
BERTScore semantic) · Bloom filter
▼
Documents + Q/A pairs → export
JSON · CSV · JSONL · Parquet ·
HuggingFace · Markdown · HTML
- Multi-provider LLM support — OpenRouter (cloud, bring-your-own-key) and Ollama (local, free), with automatic failure tracking and provider switching
- Per-dataset adaptive temperature — D01 factual (0.1) through D25 creative (0.85)
- React + FastAPI web UI — session auth, live token/latency tracking, batch processing with checkpoint recovery, per-call analytics
- Performance — async generation engine; connection pooling cuts SQLite operations from ~10ms to ~0.1ms
RAG Spectrum is the data engine behind Manthan, a RAG evaluation platform that stress-tests live RAG endpoints and recommends pipeline configurations using Design of Experiments. Together they answer the question every LLM team faces: "did this change actually make our RAG better?"
Python · FastAPI · React · asyncio · SQLite · BERTScore · RapidFuzz · Docker · deployed on Hugging Face Spaces