Skip to content

ROADMAP.md

Latest commit

 

History

History
164 lines (119 loc) · 5.69 KB

ROADMAP.md

File metadata and controls

164 lines (119 loc) · 5.69 KB
Raw

TrustLens Roadmap

Last updated: May 2026 This roadmap reflects our current priorities. Community feedback shapes every phase.

Current State (v0.5.0)

  • ✅ Stable ML evaluation pipeline (calibration, failure, bias, representation)
  • ✅ Head-to-head model comparison API (trustlens.compare)
  • ✅ Decision-ready Trust Score with penalty reasoning
  • ✅ Framework-agnostic prediction resolver architecture (XGBoost, LightGBM, CatBoost)
  • ✅ Scientific Validation: Model Zoo Benchmark for accuracy/trust decoupling
  • ✅ Professional contributor infrastructure and documentation

Status Legend

  • Completed (production-ready)
  • [~] In progress / Experimental (limited availability)
  • Planned

Active Work (v0.6.0)

These are high-priority items currently being developed or targeted for the next release.

  • Policy Profiles (Issue #56) — Context-aware scoring (Strict/Lenient)
  • TrustComparison (Issue #57) — Differential reliability audits
  • XGBoost Support — Native prediction resolver architecture
  • Model Zoo Benchmark — Scientific validation of trust diagnostics
  • [~] Deep Learning Backends — Experimental Keras & TensorFlow integration
  • [~] HTML Report Export (Issue #19) — [OPEN]
  • Maximum Calibration Error (MCE) (Issue #1)

Phase 1: MVP — The Foundation

Status: COMPLETE

The minimal set of features required to be genuinely useful to practitioners.

Deliverables

  • Core analyze() API with module dispatch
  • TrustReport result container with show(), plot(), save()
  • Calibration: Brier Score, ECE, reliability curve + reliability diagram
  • Failure Analysis: misclassification summary, confidence gap histogram
  • Bias Detection: class imbalance report, subgroup accuracy/F1
  • Representation Analysis: silhouette separability, CKA metric
  • [~] Explainability: Grad-CAM with PyTorch support — [EXPERIMENTAL]
  • [~] Faithfulness: pixel deletion + insertion tests (AUPC) — [EXPERIMENTAL]
  • Plugin system (BasePlugin + PluginRegistry)
  • Full test suite (>230 tests)
  • Professional README, CONTRIBUTING, quickstart examples
  • PyPI package & GitHub Actions CI (lint/test/format)

Phase 2: Core Expansion — Going Deeper

Status: COMPLETE (v0.5.0)

Focus: High-impact ML features that integrate directly into the analyze() pipeline.

High Priority

  • Equalized Odds (Issue #25)
  • UMAP/t-SNE Visualization (Issue #22)
  • Jupyter Rich Display (_repr_html_) (Issue #24)
  • Progress Bars via tqdm (Issue #28)
  • Subgroup ECE (calibration per demographic group)
  • [~] HTML Report Export (Issue #19) — [OPEN]
  • Temperature Scaling (Issue #18)
  • Maximum Calibration Error (MCE) (Issue #1)

Nice to Have

  • Multi-class ECE (label-wise decomposition)
  • Critical Failures table for TrustReport
  • Per-class PR curves and optimal threshold analysis
  • Prediction Flip Analysis (robustness check)
  • Eigen-CAM (gradient-free explainability)
  • Integrated Gradients (IG) for tabular models
  • SHAP Wrapper (optional dependency)
  • Text-based Sensitive Feature Parsing
  • Linear Probing accuracy per layer
  • Intrinsic Dimensionality estimation for embeddings

Framework Support

  • XGBoost Integration — Native analyze() support for XGBClassifier and raw Booster
  • LightGBM Integration — Native analyze() support for LGBMClassifier and raw Booster
  • CatBoost Integration — Native analyze() support for CatBoostClassifier
  • [~] Keras Experimental — Sequential and functional model support
  • [~] TensorFlow Experimental — SavedModel loading and lazy import hygiene

Phase 3: Comparative & Research — Frontier Methods

Target: v0.5.x

Methods primarily of interest to ML researchers pushing state-of-the-art and production diffing.

Comparative Audits

  • Model recommendation engine (trustlens.compare)
  • Score comparison tables
  • Policy-aware deltas (Issue #57)

Advanced Representation

  • Layer-wise CKA heatmap (n_layers × n_layers)
  • Neuron activation statistics per class
  • Representation fragility score (adversarial perturbation)

Benchmarking

  • benchmark() function — run TrustLens on standard datasets (CIFAR-10, etc.)
  • Baseline scores for common architectures

Phase 4: Community Growth — Scaling Impact

Target: v0.5.0

Making TrustLens a community standard.

Contribution & Documentation

  • Contributor hall of fame in README
  • Video walkthrough series
  • Interactive Jupyter notebooks (Colab-ready)
  • Plugin submission process (community plugin registry)
  • trustlens-contrib companion repository

Integrations

  • MLflow: log TrustLens metrics as experiment artifacts
  • Weights & Biases: log reliability diagrams as charts
  • Hugging Face: evaluate-compatible metric modules
  • DVC: TrustLens report as a DVC stage output

Phase 5: Platform — The Full Stack

Target: v1.0.0

TrustLens as a complete model analysis platform.

Web Dashboard

  • Zero-config web UI (trustlens serve) using FastAPI + React
  • Interactive reliability diagrams (Plotly)
  • Model comparison views

Enterprise Features

  • Scheduled monitoring reports (model drift detection)
  • Slack/Teams alerting on metric regression
  • Role-based access for team reports

Feedback

Have thoughts on the roadmap? Open a discussion on GitHub — we read everything.