🧠 Proof-of-Deliberation (PoD)

Process-based supervision certification for streaming Human-in-the-Loop active learning

---

PoD attaches a credibility signal to every human-generated label using observable interaction traces, deliberation time and its coupling with task difficulty, rather than the label content itself, and gates model updates accordingly.

This repository is the official, reproducible companion to the paper. It packages the protocol as an installable Python library (pod), exposes console-script entry points, and ships frozen configurations, unit tests, and per-dataset reproduction recipes.

---

📋 Contents

• ⚡ Quickstart
• 🗂️ Repository layout
• 🔬 The PoD protocol at a glance
• 📊 Reproducing the paper
• ✈️ Hick-Hyman validation on ATC logs
• 👥 Real-user labeling study
• 🛠️ Development
• 🐳 Docker
• 🧾 Artifact evaluation
• 📡 Real-time PoD dashboard
• 📖 Citation
• 📜 License

---

⚡ Quickstart

Step 1 — Clone and install

git clone https://github.com/jorge-martinez-gil/proof-of-deliberation.git
cd proof-of-deliberation
pip install -e .

Step 2 — Reproduce the F1 trajectory figures from Section 6

pod-experiments --datasets synth,elec2,gas

Step 3 (optional) — Hick-Hyman validation (Section 6.5)

pip install -e ".[validation]"
pod-validate --out hick_hyman_pod_validation_v2.png

For developers (tests, lint, type-check):

pip install -e ".[dev]"
pytest
ruff check src tests
mypy src/pod

---

🗂️ Repository layout

proof-of-deliberation/
│
├── src/pod/                # Installable package
│   ├── config.py           # Frozen dataclasses (Section 4 notation)
│   ├── core.py             # Gate, coupling, vigilance checks
│   ├── operator.py         # Stochastic operator simulator
│   ├── learner.py          # SGD wrapper + decision-function softmax
│   ├── baselines.py        # AL, StaticGating, AdaptiveGating
│   ├── experiment.py       # Closed-loop runner and aggregator
│   ├── viz.py              # Publication-grade plotting
│   ├── presets.py          # Frozen per-dataset hyperparameters
│   ├── streams/            # Synth-Boundary, OpenML, UCI 224 loaders
│   ├── validation/         # Hick-Hyman pipeline on ATC corpora
│   └── cli/                # `pod-experiments`, `pod-validate`
│
├── configs/                # JSON snapshots matching presets.py
├── experiments/            # Extended-regime, ablation, and robustness runners
├── scripts/                # Stats-report and claim-checking utilities
├── tests/                  # pytest suite (unit + integration)
├── docs/                   # Notation, reproducibility, architecture
├── PoD_code/               # Browser-based MQTT dashboard + labeling app
└── archive/                # Pre-refactor scripts (do NOT use)
│
├── Makefile                # Reproducibility entry points (`make paper`)
├── pyproject.toml          # PEP 621 packaging + tooling config
├── requirements*.txt       # Pinned dependency manifests
├── CITATION.cff            # Software citation metadata
└── CHANGELOG.md            # Project changelog

Generated experiment outputs (out_*/) and downloaded datasets (data_cache_uci224/, etc.) are produced on demand and intentionally not tracked.

---

🔬 The PoD protocol at a glance

PoD checks three independent signals for each queried label. A label is incorporated into the learner only when all three checks agree:

V(π_t) = (V_t = 1)  AND  (ρ_cog > ε)  AND  (S_vig = 1)

flowchart LR
    A([Human Label]) --> B{Deliberation Gate\nV_t}
    B -->|pass| C{Cognitive Coupling\nρ_cog}
    B -->|fail| R1([❌ Rejected])
    C -->|pass| D{Multi-scale Vigilance\nS_vig}
    C -->|fail| R2([❌ Rejected])
    D -->|pass| E([✅ Accepted → Model Update])
    D -->|fail| R3([❌ Rejected])

Loading

Layer	Symbol	Implementation
Deliberation gate	V_t	pod.core.gate_check
Cognitive coupling	ρ_cog	pod.core.coupling_check
Multi-scale vigilance	S_vig	pod.core.gaming_detector, pod.core.fatigue_detector

See docs/NOTATION.md for the complete symbol-to-code mapping.

---

📊 Reproducing the paper

The reference experiments compare PoD against 4 baselines and 3 ablation variants on 5 data streams under 3 operator regimes (Baseline, Gaming, Fatigue):

Family	Method	Decision rule
Baselines	AL	Accept every queried label
	StaticGating	Accept iff delib_ms >= 510
	AdaptiveGating	Difficulty-scaled deliberation window only
	WorkerQuality	Online Dawid-Skene single-annotator posterior
PoD	PoD	Composite verification (gate + coupling + S_vig)
Ablations	PoD-NoGate	Drop V_t; keep coupling and vigilance
	PoD-NoCoupling	Drop ρ_cog; keep V_t and vigilance
	PoD-NoVigilance	Drop gaming + fatigue detectors; keep V_t, ρ

Full reproduction (20 runs per dataset)

pod-experiments --datasets synth,elec2,gas,covertype,airlines

📁 Output structure

out_pod_unified/
├── synth_boundary/
│   ├── figs/Synth-Boundary_methods_f1.{pdf,png}     # Figure in Sec. 6
│   ├── runs/Synth-Boundary_<method>_run<N>.csv      # Per-run F1 trace
│   ├── diagnostics.csv                              # Query/accept rates
│   └── config.json                                  # Frozen config snapshot
├── elec2/
└── uci224_gas_drift/

Quick sanity check (~30 seconds)

pod-experiments --datasets synth --runs 3 --baseline 500 --gaming 500 \
                --fatigue 500 --synth_pool 5000 --holdout 1000

Seeding and determinism

All randomness is centralised through numpy.random.default_rng(seed). The per-run seed is derived as 1000 * run_idx + 7 (see pod.experiment.run_suite_generic). Identical hardware + Python + NumPy versions + pinned dependencies reproduce F1 trajectories exactly.

Configuration provenance

Every run writes a config.json next to the figures. Verify against the paper's frozen configs:

diff -u configs/synth.json out_pod_unified/synth_boundary/config.json

See docs/REPRODUCIBILITY.md for the full recipe.

Statistical analysis

pod-stats --in out_pod_unified --reference PoD

This writes the following artefacts under out_pod_unified/stats/:

Artefact	Contents
per_run_scores.csv	One row per (dataset, method, run) with scalar score
average_ranks.csv	Mean rank of each method across datasets (lower is better)
friedman.json	Friedman chi-square and Iman-Davenport F omnibus test
nemenyi_cd.json	Critical-difference value CD at chosen alpha
cd_diagram.pdf/.png	Demsar-style critical-difference diagram with NSD cliques
wilcoxon_holm.csv	Pairwise Wilcoxon signed-rank vs. reference, Holm-corrected
bootstrap_ci.csv	Percentile bootstrap CI (default 10K resamples) of mean F1
per_regime_anova.csv	One-way ANOVA over regimes from per-phase accept rates
summary.json	Aggregated summary of the full suite

---

✈️ Hick-Hyman validation on ATC logs

Section 6.5 of the paper validates that PoD's coupling signal reflects real deliberation rather than a simulator artefact. The validation uses two HuggingFace corpora: atco2_corpus_1h and uwb_atcc.

pip install -e ".[validation]"
pod-validate --out hick_hyman_pod_validation_v2.png

The pipeline:

1. Loads ~3,400 pilot-controller pairs
2. Computes out-of-fold predictive entropies with a TF-IDF + isotonic-calibrated logistic regression (averaged over 3 seeds)
3. Applies post-hoc temperature scaling
4. Runs Spearman tests against controller response delay with linguistic-proxy contrasts
5. Renders the summary figure used in the paper

---

👥 Real-user labeling study

Participants label elec2 trials in a browser-based app across three induced regimes (baseline, speed-bonus, long-block) while millisecond-precision response times are recorded. Collected CSVs are analysed offline through the same PoD verification primitives.

# 1) Build the frozen task pool (once per study)
python -m pod.realdata.build_pool --out PoD_code/labeling/tasks.json

# 2) Deploy the labeling app
cd PoD_code/labeling && python -m http.server 8000

# 3) Collect participant CSVs, then analyse
pod-realdata --in data_real/ --out out_real/

The labeling app at PoD_code/labeling/app.html is a single-file, offline page — no analytics, no telemetry, no network calls.

The full protocol (consent text, exclusion criteria, statistical tests, and limitations) is documented in docs/REAL_DATA_PROTOCOL.md.

---

🛠️ Development

pip install -e ".[dev]"

pytest                                    # Unit + integration smoke tests (~10 s)
pytest --cov=pod --cov-report=term-missing  # With coverage
ruff check src tests                      # Lint and format checks
mypy src/pod                              # Static type checking

The tests/ directory covers PoD primitives, the operator simulator, stream generation, the comparison baselines, and a tiny end-to-end run for every method.

---

🐳 Docker

A locked-environment image reproduces the exact toolchain recorded in requirements-lock.txt (Python 3.10, numpy 2.2.6, scikit-learn 1.7.2, ...):

docker build -t pod .
docker run --rm pod                                   # test suite
docker run --rm -v "$PWD/out_pod_unified:/work/out_pod_unified" pod \
    pod-experiments --datasets synth,elec2,gas,covertype,airlines

---

🧾 Artifact evaluation

Reviewers can follow docs/ARTIFACT_EVALUATION.md, which maps every experimental section of the paper to the exact command that regenerates it, states expected runtimes, and documents the claim-consistency gate (scripts/check_claims.py) that fails CI when a reported number drifts from the regenerated statistics.

---

📡 Real-time PoD dashboard

The PoD_code/ directory contains a single-page, browser-based MQTT dashboard that monitors the PoD gate live on a deployed sensor stream.

1. Configure your broker in PoD_code/config.json
2. Open PoD_code/app.html directly in a browser

The default broker (broker.emqx.io) is a public test broker — replace it with your own for production deployments.

---

📖 Citation

If you use this software, please cite the paper and the software artefact:

@software{MartinezGil2026PoDSoftware,
    author  = {Jorge Martinez-Gil},
    title   = {Proof-of-Deliberation (PoD): reference implementation},
    year    = {2026},
    version = {1.3.0},
    url     = {https://github.com/jorge-martinez-gil/proof-of-deliberation}
}

The full citation metadata in CITATION.cff is GitHub-renderable — click "Cite this repository" on the project page to copy a BibTeX or APA entry.

---

📜 License

This project is released under the MIT License.

---

🙏 Acknowledgements

Source	Contribution
OpenML	Electricity (elec2, id 44156), Forest CoverType (id 1596), and Airlines (id 1169) benchmarks
UCI ML Repository	Gas Sensor Array Drift dataset (#224)
HuggingFace / Zuluaga-Gomez et al.	atco2_corpus_1h and uwb_atcc ATC corpora
scikit-learn, NumPy, pandas, Matplotlib, SciPy	Scientific computing stack