RAG intelligence platform — generate challenge datasets, stress-test live RAG endpoints, and design optimal RAG pipelines with statistically rigorous experiments.
This repo is the public overview of a private codebase. For source access or collaboration: ayushchhoker15@gmail.com
Every team shipping a RAG product faces the same question: "did this change actually make our RAG better?" Most answer it by eyeballing a few queries. Manthan answers it with controlled test data and designed experiments.
┌───────────────────────────────────┐
│ RAG SPECTRUM │
│ (core synthetic-data engine) │
│ topic + challenge type → │
│ documents + ground-truth Q/A │
│ 25-type taxonomy (D01–D25) │
└───────────────┬─────────────────────┘
│ "give me a dataset"
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ 1. GENERATE │ │ 2. TEST A RAG │ │ 3. BUILD A RAG │
│ synthetic datasets│ │ stress-test your │ │ find the best │
│ in JSON, CSV, │ │ live endpoint → │ │ pipeline config │
│ JSONL, Parquet, │ │ ranked weakness │ │ via designed │
│ HuggingFace… │ │ profile │ │ experiments (DoE) │
└──────────────────┘ └──────────────────┘ └──────────────────┘
1 — Generate. Pick challenge types from a 25-type taxonomy (atomic facts → cross-document reasoning → adversarial hallucination bait), get documents plus ground-truth Q/A pairs, download in multiple formats. Powered by the RAG Spectrum engine.
2 — Test a RAG. Point Manthan at your RAG endpoint. It ingests generated documents into your system first (testing on unseen documents produces meaningless results), fires questions across all 25 challenge types, and uses LLM-as-judge scoring against ground truth. Output: a ranked weakness profile with severity bands and remediation hints, streamed live to the UI.
3 — Build a RAG. Bring a document or a goal. Manthan runs candidate pipeline configurations — chunking strategy, embedding model, retrieval method, top_k, hybrid alpha — and recommends the best one. With a sweep configured, it uses a statistically designed experiment (fractional factorial, blocked ANOVA, Tukey HSD, bootstrap confidence intervals) and reports which factors actually move the score, not just which config won.
Comparing two RAG configs on 20 questions and picking the higher number is noise, not evidence. Manthan treats RAG tuning as an experiment-design problem: factor screening separates what matters (usually chunking and retrieval method) from what doesn't, and effect sizes with confidence intervals tell you whether a difference is real before you ship it.
Python · FastAPI · React · asyncio · PostgreSQL · Qdrant · Design of Experiments · LLM-as-judge