📊 KappaBench

KappaBench is a premium, zero-dependency consensus evaluation framework designed to measure, benchmark, and analyze inter-agent reliability in multi-agent classification tasks. It provides standalone Python implementations of standard consensus metrics and incorporates robust perturbation testing (e.g., noise injection, model dropout) to quantify the reliability and resilience of LLM-based agent networks.

---

🚀 Key Features

• Standardized Inter-Rater Reliability Metrics:
  • Cohen's Kappa ($\kappa$): Pairwise agreement between two classifiers.
  • Fleiss' Kappa ($\kappa_F$): Generalized multi-rater agreement for fixed raters.
  • Krippendorff's Alpha ($\alpha$): Premium consensus coefficient supporting missing ratings and multiple measurement scales (Nominal, Ordinal, Interval, Ratio).
• Robustness Perturbation Engine: Test consensus resilience by simulating agent/model dropouts or injecting classification noise.
• Unified CLI Interface: Run benchmarks directly on JSON/CSV datasets and output report summaries.
• Developer Friendly: Fully type-hinted, packaged with standard packaging config (pyproject.toml), and easy to integrate.

---

📐 Mathematical Formulation

1. Cohen's Kappa ($\kappa$)

Cohen's Kappa measures the agreement between two raters who each classify $N$ items into $C$ mutually exclusive categories:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where:

• $p_o$ is the relative observed agreement among raters (equivalent to accuracy).
• $p_e$ is the hypothetical probability of chance agreement.

---

2. Fleiss' Kappa ($\kappa_F$)

Fleiss' Kappa extends Scott's $\pi$ statistic to multiple raters. For $N$ subjects, $R$ raters, and $C$ categories:

$$\kappa_F = \frac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e}$$

The mean of the subjects' agreement $P_i$ is defined as:

$$P_i = \frac{1}{R(R-1)} \left( \sum_{j=1}^{C} n_{ij}^2 - R \right)$$

$$\bar{P} = \frac{1}{N} \sum_{i=1}^{N} P_i$$

The expected chance agreement $\bar{P}_e$ is:

$$\bar{P}_e = \sum_{j=1}^{C} p_j^2 \quad \text{where} \quad p_j = \frac{1}{NR} \sum_{i=1}^{N} n_{ij}$$

---

3. Krippendorff's Alpha ($\alpha$)

Krippendorff's Alpha is a general coefficient that measures the agreement among any number of observers, with any level of measurement (nominal, ordinal, interval, ratio) and handling missing values natively:

$$\alpha = 1 - \frac{D_o}{D_e}$$

where:

• $D_o$ is the observed disagreement:

  $$D_o = \frac{1}{n} \sum_{i} \frac{1}{m_i - 1} \sum_{c} \sum_{c'} n_{ic} n_{ic'} \delta^2(c, c')$$

• $D_e$ is the expected disagreement:

  $$D_e = \frac{1}{n(n-1)} \sum_{c} \sum_{c'} n_{\cdot c} n_{\cdot c'} \delta^2(c, c')$$

• $\delta^2(c, c')$ is the metric difference function depending on the level of measurement (e.g., Nominal, Ordinal, Interval).

---

🏗️ Architecture

The system consists of three main decoupled modules:

graph TD
    CLI[cli.py: argparse interface] -->|Invokes| Benchmark[benchmark.py: ConsensusBenchmark]
    Benchmark -->|Invokes| Metrics[metrics.py: Math Computations]
    
    subgraph metrics.py
        CK[cohen_kappa]
        FK[fleiss_kappa]
        KA[krippendorff_alpha]
    end
    
    subgraph benchmark.py
        LD[load_dataset]
        CC[compute_consensus]
        PT[run_perturbation_tests]
    end

Loading

---

📦 Installation

To install KappaBench in editable development mode:

git clone https://github.com/Rituparno-Majumdar/kappabench.git
cd kappabench
make install

---

🛠️ Quick Start

Python Library Usage

from kappabench.metrics import cohen_kappa, fleiss_kappa, krippendorff_alpha
from kappabench.benchmark import ConsensusBenchmark

# 1. Direct Metric Calculations
rater1 = [1, 2, 1, 2, 1]
rater2 = [1, 2, 2, 2, 1]

ck = cohen_kappa(rater1, rater2)
print(f"Cohen's Kappa: {ck:.4f}")

# 2. Benchmarking Multi-Agent Data
benchmark = ConsensusBenchmark()
benchmark.load_dataset("tests/mock_data.csv")

# Compute baseline metrics
summary = benchmark.compute_consensus(level_of_measurement='nominal')
print("Fleiss' Kappa:", summary['fleiss_kappa'])
print("Krippendorff's Alpha:", summary['krippendorff_alpha'])

CLI Interface Usage

KappaBench provides a CLI to run consensus analyses and perturbation tests:

# Run baseline evaluation
kappabench run --dataset my_agents_output.csv

# Run evaluation and noise/dropout robustness perturbation test suite
kappabench run --dataset my_agents_output.json --level nominal --perturb --output results.json

---

📈 CLI Execution Report Preview

When running kappabench run, you'll receive a professional summary:

==================================================
 KAPPABENCH EVALUATION REPORT 
==================================================
Dataset:       my_agents_output.csv
Total Items:   100
Total Models:  3
Models:        agent_a, agent_b, agent_c
Categories:    class_1, class_2
Metric Level:  nominal
--------------------------------------------------
Fleiss' Kappa:         0.7812
Krippendorff's Alpha:  0.7820
Mean Cohen's Kappa:    0.7818 (std: 0.0125)
==================================================

---

🧪 Testing and Quality Control

All code is linted and verified via unit tests. Run the test suite:

make test

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.