Dankjewel — Online Evaluation System for LLM-Generated Text

A modular, multi-dimensional evaluation framework based on the TrustLLM project protocol (D7.5) for assessing the quality of text generated by Large Language Models. All four evaluation prongs are fully implemented.

License: Apache 2.0 — see LICENSE
Project: TrustLLM (EU Grant #101135671)
Lead: Lars Bungum, NTNU

---

Architecture Overview

The system is built on a four-pronged evaluation approach, each analyzing a different dimension family of generated text:

Prong	Name	Method	Status
1	LLM-as-Judge	Sentence embeddings (MiniLM) + toxicity classifier (toxic-bert)	✅ Complete
2	Rule-Based	Lexicon/heuristic scoring, language-aware (9 languages)	✅ Complete
3	Extraction-Based	spaCy triple extraction + offline Wikidata fact-checking	✅ Complete
4	Intrinsic	Model log-probabilities from vLLM (perplexity, uncertainty, entropy)	✅ Complete

A FastAPI server (src/api.py) with a single-page HTML5 frontend (frontend/index.html) supports interactive generation (vLLM or any OpenAI-compatible remote API) and real-time evaluation with streaming output.

---

Directory Structure

online-eval-base/
├── src/
│   ├── base.py                     # Base classes (DimensionScore, Evaluator, ProngResult)
│   ├── config.py                   # Language codes, Flesch coefficients, constants
│   ├── pipeline.py                 # EvaluationPipeline — orchestrates all prongs
│   ├── api.py                      # FastAPI server (generation, evaluation, streaming)
│   ├── vllm_client.py              # Async vLLM/OpenAI-compatible HTTP client
│   ├── vllm_adapter.py             # GenerationResult → Prong 4 IntrinsicData
│   ├── prongs/
│   │   ├── prong1_llm_as_judge.py  # Prong 1 orchestrator
│   │   ├── prong1_dimensions.py    # Relevance, Coherence, Reliability, Toxicity
│   │   ├── prong2_rule_based.py    # Prong 2 orchestrator
│   │   ├── prong3_extraction.py    # Prong 3 orchestrator
│   │   └── prong4_intrinsic.py     # Prong 4 orchestrator + IntrinsicData
│   ├── dimensions/
│   │   ├── base_dimension.py       # ScorableDimension base class
│   │   ├── prong2_dimensions.py    # Fluency, Politeness, Emotional Intensity, Formality, Hedging
│   │   └── prong3_dimensions.py    # Factual Accuracy, Consistency, Temporal, Comparative, Modality
│   ├── extraction/
│   │   └── claim_extractor.py      # spaCy dependency-parsing claim extractor
│   ├── aggregation/
│   │   └── aggregator.py           # Weighted prong/dimension aggregation
│   ├── models/
│   │   ├── embeddings.py           # sentence-transformers wrapper (LRU cached)
│   │   └── toxicity_model.py       # unitary/toxic-bert wrapper
│   ├── llm_backends/               # Pluggable backend abstraction (local vLLM / remote)
│   └── resources/
│       ├── wikidata_offline.py     # Offline Wikidata interface (SQLite)
│       ├── lexicon_manager.py      # Per-language YAML lexicon loader
│       └── lexicons/               # da, de, en, fo, is, nb, nn, nl, sv
├── scripts/
│   ├── build_wikidata_db.py        # Build local Wikidata seed DB
│   ├── build_fever_wikidata.py     # Build FEVER-coverage Wikidata DB
│   ├── extend_wikidata_db.py       # Extend DB with entities from claim text
│   ├── eval_fever2.py              # Prong 3 benchmark on FEVER 2.0
│   ├── filter_oracle_claims.py     # Filter oracle claims for benchmarking
│   └── run_benchmark.py            # Multi-model benchmarking script
├── tests/
│   ├── test_prong1.py
│   ├── test_prong2.py
│   ├── test_prong3.py
│   ├── test_prong4.py
│   ├── test_api.py
│   └── test_llm_backends.py
├── frontend/
│   ├── index.html                  # Single-page control panel
│   └── locales/                    # UI translations (9 languages)
├── data/
│   └── wikidata_seed.db            # Pre-built offline Wikidata seed
├── evaluation_results/             # FEVER 2.0 benchmark output
├── requirements.txt
├── main.py                         # CLI example — evaluates 4 sample texts
└── LICENSE                         # Apache 2.0

---

Installation

Standard install (CPU only — no GPU required)

pip install -r requirements-minimal.txt
python3 -c "import nltk; nltk.download('vader_lexicon')"
python3 -m spacy download en_core_web_sm
python3 scripts/build_wikidata_db.py
uvicorn src.api:app --host 0.0.0.0 --port 8080

Open http://localhost:8080. All four prongs are available:

• Prongs 1–3 run locally on CPU with no external dependencies.
• Prong 4 works via any remote OpenAI-compatible backend that returns per-token logprobs (OpenAI, Groq, Fireworks, etc.) — supply a Base URL and API key in the UI's Backend Selector. If the provider does not return logprobs, Prong 4 shows neutral placeholder scores; the other three prongs are unaffected.

Local vLLM server (requirements.txt, CUDA GPU required) is only needed if you want to run generation on your own machine rather than via a remote API key.

Python 3.12+ required.

---

Quick Start

CLI Example

python3 main.py

Evaluates 4 sample texts across all prongs and saves results to evaluation_results/evaluation_results.json.

Web UI (generate + evaluate)

# Terminal 1: start a vLLM server (or skip if using a remote API key)
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-1.5B-Instruct --port 8000

# Terminal 2: start the IE server
uvicorn src.api:app --host 0.0.0.0 --port 8080

Open http://localhost:8080 in your browser. From there you can:

• Pick a backend: Local (vLLM) or Remote (OpenAI-compatible, bring-your-own-key)
• Configure generation parameters (temperature, top-p, penalties, max tokens)
• Generate text with real-time SSE token streaming
• View all 4 prong scores with expandable per-dimension breakdowns
• Export results as JSON / browse evaluation history (localStorage)

Evaluate-only (no generation, no vLLM)

curl -X POST http://localhost:8080/evaluate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is the capital of France?",
       "text": "The capital of France is Paris."}'

Python API

from src.pipeline import EvaluationPipeline

pipeline = EvaluationPipeline(enable_prongs=[1, 2, 3, 4])
result = pipeline.evaluate(
    text="The capital of France is Paris.",
    prompt="What is the capital of France?",
    lang="en",
)
print(result["aggregated_scores"]["total_score"])

---

Evaluation Dimensions

Prong 1 — LLM-as-Judge

Dimension	Method	Notes
Relevance	Cosine similarity (MiniLM embeddings) to prompt	Requires prompt
Semantic Coherence	Sliding-window sentence embedding similarity
Reliability	Topic-stability (embedding variance)
Toxicity	unitary/toxic-bert classifier	agg_value = 1 − toxicity

Prong 2 — Rule-Based (9 languages: da, de, en, fo, is, nb, nn, nl, sv)

Dimension	Method	Notes
Fluency	Flesch readability (language-adapted formula)
Politeness	Lexicon cue count (+0.1/cue, −0.1/rude cue, clipped to [0,1])
Emotional Intensity	VADER sentiment + punctuation features	aggregate=False
Formality	Formal/informal lexicon markers
Hedging	Modal verbs & uncertainty cues	aggregate=False

Prong 3 — Extraction-Based

Dimension	Method	Notes
Factual Accuracy	spaCy triple extraction → offline Wikidata verification
Factual Consistency	Cross-claim contradiction detection via Wikidata QIDs
Temporal Accuracy	Date anchor extraction → Wikidata P585/P580/P582/P569/P570/P571
Comparative Factuality	JJR/JJS comparative claims → Wikidata
Modality Awareness	Hedged vs. asserted claim fraction	aggregate=False

Prong 4 — Intrinsic (requires vLLM logprobs)

Dimension	Method	Notes
Uncertainty	1 − mean token confidence	agg_value = confidence
Perplexity	exp(mean NLL)	agg_value = 1/(1+ppl), displayed raw
Token Entropy	Shannon entropy of token distributions (normalized)	aggregate=False
Semantic Entropy	Entropy of top-k cluster (normalized)	aggregate=False

Without a vLLM backend, Prong 4 returns neutral placeholder scores (0.5).

---

Score Aggregation

Each enabled prong produces a score in [0, 1] as the mean of its aggregate=True dimensions. Dimensions marked aggregate=False (Emotional Intensity, Hedging, Token Entropy, Semantic Entropy, Modality Awareness) are reported but excluded from the score — they have no universal "better" direction.

For dimensions whose raw scale is not [0, 1] / higher-is-better (Toxicity, Uncertainty, Perplexity), an agg_value supplies an orientation-corrected bounded value used during aggregation.

The total score is a weighted average of prong scores (default: 0.25 each), renormalized if any prong is missing or errors out.

---

Benchmarking (FEVER 2.0)

Prong 3 has been benchmarked on the FEVER 2.0 fixers development set (1,174 claims). Key results:

Condition	Coverage	Binary AUC	Conditional AUC
Seed DB (~86 entities)	9.0%	0.494	0.500
Oracle DB (FEVER-targeted)	8.7%	0.538	0.682

Coverage (fraction of claims where Wikidata returned CONFIRMED or DENIED) is the primary bottleneck — uncovered claims fall back to the neutral 0.5. See DB_EXTENSION_GUIDE.md and scripts/eval_fever2.py for details.

---

Testing

pytest tests/ -v            # all tests
pytest tests/test_prong2.py -v
pytest tests/ --cov=src

---

Planned Work

See TODO.md. Key next steps:

• LLM-based claim extraction for Prong 3 (vLLM-prompted, à la FActScore/Claimify)
• LLM-based readability for Prong 2 (replacing Flesch baseline)
• Dual-output benchmarking: baseline vs. LLM score per dimension
• Expand Wikidata DB toward 200k entities via SPARQL dump

---

References

• Protocol: TrustLLM Deliverable 7.5 — Multi-Dimensional Evaluation Metric
• Danescu-Niculescu-Mizil et al. (2013) — Politeness cue lists
• Aly et al. (2021) — FEVER 2.0 benchmark
• Flesch (1948), Amstad (1978), Douma (1960), Matricciani (2023) — Readability formulas