Try this pipeline in action at Vitreon!
A three-stage RAG pipeline for the ARLC 2026 Agentic RAG Legal Challenge — question answering over DIFC (Dubai International Financial Centre) legal documents.
Click the banner to explore our interactive experiment tree — 150+ experiments, color-coded by outcome
Question
|
v
Router (regex) -- deterministic doc routing: case IDs, law names, article numbers
| oracle fast-path for metadata (dates, judges, parties)
v
Retriever -- hybrid BM25 + dense embeddings + cross-encoder reranking
| adaptive page selection (max 1 page/doc, max 3 total)
v
Answerer -- single Claude Sonnet call per question
| type-specific prompts, confidence calibration
v
Post-processing -- absence detection, format validation, telemetry
|
v
Submission JSON
# 1. Clone and install
git clone https://github.com/neonsecret/ai-challenge-legal.git
cd ai-challenge-legal
cp .env.example .env # set ANTHROPIC_API_KEY and EVAL_API_KEY
uv sync # or: pip install -e .
# 2. Prepare corpus (downloads docs from platform API)
make prepare # or: python -m arlc.indexing.prepare_corpus
# 3. Run pipeline
make run # or: python run.py --workers 5 --output output/run1| Module | Purpose |
|---|---|
arlc/pipeline.py |
Pipeline orchestrator — routing, retrieval, answering, post-processing |
arlc/router.py |
Deterministic document routing via regex (no LLM) |
arlc/retriever.py |
Hybrid BM25 + dense vector + cross-encoder reranking |
arlc/answerer.py |
Answer generation via Anthropic SDK (Claude Sonnet) |
arlc/indexing/indexer.py |
Vector + BM25 index builder |
arlc/indexing/prepare_corpus.py |
End-to-end corpus download, indexing, and smoke test |
arlc/llm/anthropic_backend.py |
Anthropic SDK backend |
arlc/llm/reranker.py |
LLM-based page reranking |
arlc/page_verifier.py |
Page citation verification |
arlc/format_guardian.py |
Answer format validation and fixing |
speed_agent/ |
Speed pipeline — PyPy + oracle metadata, 152ms avg TTFT |
The data/ directory is not included in the repo (copyrighted legal documents). To reproduce:
# 1. Get the ARLC starter kit
git clone https://github.com/agentic-challenge/arlc-starter-kit.git starter_kit
# 2. Download corpus and build all indexes
make prepare # or: python -m arlc.indexing.prepare_corpus
# This creates:
# data/documents/ — PDF corpus (303 documents)
# data/questions.json — question set (900 questions)
# data/case_metadata_index.json — case metadata (auto-built, then manually corrected)
# data/law_name_index.json — law name variants to doc IDs
# data/article_page_index.json — article number to page number
# data/appeal_index.json — SCT appeal classifications (manual)
# data/consultation_paper_index.json — consultation paper metadata (manual)
# data/court_order_index.json — court order metadata (manual)
# data/latest_edition_index.json — latest edition of each law (manual)Note: Some indexes (appeal, consultation paper, court order) were manually curated during the competition and are generated by
prepare_corpus.pywith LLM-assisted extraction. Results may vary slightly with different API keys/models.
| Variable | Required | Description |
|---|---|---|
ANTHROPIC_API_KEY |
Yes | Anthropic API key for Claude models |
EVAL_API_KEY |
Yes | ARLC platform API key (corpus download) |
| Model | Provider | License | Role |
|---|---|---|---|
| Claude Sonnet 4.6 | Anthropic | API ToS | Answer generation (deterministic types) |
| Claude Opus 4.6 | Anthropic | API ToS | Answer generation (free-text), SAC summaries |
| Claude Haiku 4.5 | Anthropic | API ToS | Question decomposition, LLM reranking |
| Snowflake Arctic Embed L v2.0 | Snowflake | Apache 2.0 | Dense embeddings (1024d) |
| BGE Reranker v2 M3 | BAAI | MIT | Cross-encoder reranking |
| FlashRank MiniLM L-12 | Prithivi Da | Apache 2.0 | Fast initial reranking |
| FAISS | Meta Research | MIT | Vector similarity search |
| Docling | IBM Research | MIT | Structural PDF extraction |
| PyMuPDF | Artifex | AGPL-3.0 | PDF text extraction (fallback) |
Built with the Anthropic API via Vertex AI.
make setup # install dependencies
make lint # run ruff linter
make test # run testsA separate speed-optimized pipeline lives in speed_agent/. It achieves 152ms average TTFT by:
- Answering 44% of questions via oracle metadata lookup (~1ms, no LLM)
- Using PyPy for maximum stdlib throughput (no C dependencies)
- Streaming via Gemini Flash Lite for remaining questions (~280ms TTFT)
- Pre-extracting all PDF text to JSON (zero I/O at inference)
See the speed agent benchmarks for detailed performance data.
The ARLC 2026 Agentic RAG Legal Challenge tests RAG systems on real DIFC legal documents — court cases, laws, regulations, and practice directions. The scoring formula is:
Total = S_det x S_asst x G x F
Where:
- S_det — deterministic answer accuracy (boolean, date, number, name)
- S_asst — LLM-judged free-text quality
- G — grounding (citation accuracy)
- F — speed bonus (F >= 1.0 for fast responses)
After the competition ended, we studied other participants' published approaches and integrated their best ideas into our pipeline. These upgrades were not used in our competition submissions — they represent what we learned from the community afterward.
| Technique | Inspired By | Description |
|---|---|---|
| Docling PDF extraction | Alexander Ivanov / IAS Partners, Maksim Metelskii | Structural PDF parsing replacing raw PyMuPDF |
| Multi-signal document fusion | Alexander Ivanov / IAS Partners | 5-weight doc fusion, dense-only page ranking |
| Custom legal tokenizer | Alexander Ivanov / IAS Partners | Compound legal reference expansion for BM25 |
| IndexRAG (AKU extraction, bridging facts) | Bao & Shi, 2026 (Continuum AI) | QA-structured facts + cross-reference graph at index time |
| Typed document ontology | Dmitry Savostyanov (DotaGPT agent approach) | Structural navigation instead of embedding search |
| Embedding decontamination | Maksim Metelskii (structure-first methodology) | Strip boilerplate before embedding, preserve for BM25 |
| Structured reasoning with grounding | Maksim Metelskii | Force LLM to cite page evidence per claim |
| Gemini PDF preprocessing | Dmitry Savostyanov | Vision models for image-embedded structural elements |
| Evaluation tooling insights | Dmitry Donchenko / mlboost | Visual review UI + LLM-as-judge for batch evaluation |
| Scoring methodology insights | Dmitry Stepanov | Structured output via tool_use, reranker tradeoffs |
| Small-doc full inclusion | Vitaliy Pokrovskiy (3rd place) | Skip reranking for docs ≤8 pages |
| Recall-biased LLM reranker | Azamat Yelmagambetov / CPBD (1st place, G=0.990) | "Round UP when uncertain" — F-beta(2.5) aware scoring |
| Per-type retrieval depth | Azamat Yelmagambetov / CPBD (1st place) | 22 depth values swept per question type |
| Entity indexing at build time | Azamat Yelmagambetov / CPBD (1st place) | Entities as separate BM25 column |
| Component-level metrics | Azamat Yelmagambetov / CPBD (1st place) | Per-stage precision/recall instrumentation |
Thank you to all participants who shared their approaches — the open exchange of ideas after the competition made everyone's systems better.
| Phase | S_det | S_asst | G | T | F | Total |
|---|---|---|---|---|---|---|
| Warmup (best, v14) | 0.986 | 0.820 | 0.957 | 0.995 | 1.033 | 0.920 |
| Finals (v2) | 0.939 | 0.761 | 0.797 | 1.000 | 1.018 | 0.719 |
- Oracle coverage: 37.3% of questions answered deterministically (zero LLM calls)
- Page verifier correction rate: 26.3% (pages corrected after answer generation)
- E2E pipeline test: 20/20 on real corpus (FAISS + Arctic Embed + Vertex AI)
| Benchmark | Metric | Our Score | SOTA | Dataset Size |
|---|---|---|---|---|
| GaRAGe (ACL 2025) | RAF | 0.826 | 0.607 (Nova Pro) | 2,366 questions |
| ContractNLI | Accuracy / Macro F1 | 0.763 / 0.725 | 0.875 / 0.855 (fine-tuned BERT) | 2,091 pairs (123 NDAs) |
| Legal RAG Bench | Retrieval Acc | 0.690 | ~0.80 (Kanon 2) | 100 questions |
GaRAGe — Full-dataset evaluation on all 2,366 items. RAF (Retrieval-Augmented Factuality) measures combined answer eligibility, citation attribution, and deflection quality. Our pipeline scores 0.826 vs. SOTA 0.607 (Nova Pro, GaRAGe paper Table 3). No other published system exceeds 0.607. See benchmarks/garage/ for reproduction steps.
ContractNLI — Zero-shot 3-way NLI (entailment/contradiction/not_mentioned) on 123 NDAs × 17 hypotheses. SOTA (0.875) is a fine-tuned Span NLI BERT_large trained on the ContractNLI training set (Koreeda & Manning, 2021). Our zero-shot approach closes two-thirds of the gap without any in-domain training. Notably, our contradiction F1 (0.611) exceeds the fine-tuned baseline (0.357). See benchmarks/contractnli/ for reproduction steps.
Legal RAG Bench — 100 expert-written criminal law questions over 4,876 passages from the Victorian Judicial College Criminal Charge Book (arxiv 2603.01710). SOTA uses the legal-domain Kanon 2 Embedder (~0.80+ retrieval accuracy); we use general-purpose Arctic Embed. See benchmarks/legal-rag-bench/ for reproduction steps.
Read the full story of building this system — from first submission (0.401) to peak warmup (0.920) to finals (0.719) — in JOURNEY.md.
Explore our experiments interactively in the Experiment Tree (open in browser).
AGPL-3.0 — Free for academic and open-source use. Commercial use requires sharing modifications under the same license. For commercial licensing inquiries, contact the author.