LLM-Bench is a research-grade framework for systematically evaluating and comparing Large Language Models across multiple cognitive task categories. As LLMs proliferate across applications, principled evaluation becomes essential for model selection and understanding capability boundaries.
This framework provides:
- Multi-category benchmarking — reasoning, factuality, and mathematical tasks
- Multi-metric scoring — semantic similarity, ROUGE-L, exact match, composite score
- Latency profiling — response time comparison across models
- Visual reporting — automated chart generation for result analysis
- REST API — FastAPI backend for programmatic access
LLM-Bench/
├── src/
│ ├── evaluator/ # LLM response generation
│ │ └── llm_evaluator.py
│ ├── metrics/ # Multi-metric scoring
│ │ └── benchmark_scorer.py
│ ├── visualizer/ # Chart generation
│ │ └── benchmark_visualizer.py
│ └── api/ # FastAPI REST API
│ └── main.py
├── data/
│ ├── prompts/ # Benchmark datasets
│ └── results/ # CSV reports + charts
├── benchmark.py # Main runner script
├── Dockerfile
└── requirements.txt
| Category | Tasks | Description |
|---|---|---|
| Reasoning | 5 | Logical deduction, multi-step problem solving |
| Factuality | 5 | World knowledge, factual accuracy |
| Math | 5 | Arithmetic, algebra, probability |
| Metric | Description |
|---|---|
| Semantic Similarity | TF-IDF cosine similarity vs ground truth |
| ROUGE-L | Longest common subsequence F1 score |
| Exact Match | Normalized string matching |
| Composite Score | Weighted: 60% semantic + 40% ROUGE-L |
| Latency (ms) | End-to-end response time |
git clone https://github.com/niharikabanothu/LLM-Bench.git
cd LLM-Bench
python -m venv venv
venv\Scripts\activate # Windows
pip install -r requirements.txt
cp .env.example .env
# Add your OPENAI_API_KEY to .envpython benchmark.py
python benchmark.py --models gpt-3.5-turbo,gpt-4o-miniuvicorn src.api.main:app --reload --port 8000docker build -t llm-bench .
docker run -p 8000:8000 --env-file .env llm-bench| Model | Reasoning | Factuality | Math | Overall |
|---|---|---|---|---|
| gpt-3.5-turbo | 0.61 | 0.78 | 0.72 | 0.70 |
| gpt-4o-mini | 0.74 | 0.85 | 0.81 | 0.80 |
Results vary — run benchmark.py to generate results for your evaluation.
Systematic LLM evaluation is an open research challenge. This project investigates:
- Do larger models consistently outperform smaller ones across all task types?
- Which task categories expose the largest capability gaps between models?
- Is there a latency-accuracy tradeoff worth quantifying for production use?
Niharika Banothu M.Tech Artificial Intelligence | MANIT Bhopal (NIT Bhopal)
- GitHub: github.com/niharikabanothu
- LinkedIn: linkedin.com/in/niharikabanothu
MIT License