Skip to content

ROADMAP.md

Latest commit

 

History

History
214 lines (159 loc) · 9.89 KB

ROADMAP.md

File metadata and controls

214 lines (159 loc) · 9.89 KB
Raw

Fair Code - Public Roadmap

Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Phase 6

This is the public roadmap for Fair Code. It tracks what has been built, what is actively in progress, and what comes next.

Last updated: September 2026

Where We Are · Phase 1 · Phase 2 · Phase 3 · Phase 4 · Phase 5 · Phase 6 · Content Schedule · How to Contribute

Where We Are

Fair Code is an open-source responsible AI platform explaining algorithmic bias, fairness, and AI accountability through code audits, explainers, healthcare-bias case studies, and contributor-led GitHub documentation.

Current traction (September 2026):

Stars Contributors Forks Watching Social Reach Countries Audits Explainers CI
46 29 33 8 30K+ 20 7 60 ✅ every push/PR

The earlier paper freeze has lifted - the real paper, with fresh results, is now planned for next year. paper/results-frozen/ (tag v1.0-paper, commit bbef2ba) is kept as a reference snapshot. See CLAUDE.md for the current policy.

Version & release gate:

  • Current release: v2.2.0
  • The next major version can now bundle a re-run benchmark and new audits without waiting on a publication gate - Phase 6 below reflects the earlier plan and will be revisited alongside next year's real paper submission.

Phase 1 - Bias Glossary and Beginner Explainers ✅

Status: Foundational library complete - 61 explainers published, expanding toward a 60+ library

Build the foundational vocabulary and explain core fairness concepts clearly enough for a non-technical reader.

  • Proxy Variables
  • Equalized Odds
  • Sampling Bias
  • SHAP Values
  • Disparate Impact (The 80% Rule)
  • Disparate Treatment
  • Why Fairness Metrics Conflict
  • Calibration
  • Demographic Parity
  • Feedback Loop Bias
  • Label Bias
  • Individual Fairness
  • Counterfactual Fairness
  • What Happens Inside a Neural Network
  • Why AI Hallucinates
  • What Is Reinforcement Learning
  • Proxy Entanglement
  • What Is Machine Learning Bias
  • What Is Data Leakage
  • How AI Detects Patterns
  • What Is Distribution Shift
  • The Biggest Myth About AI Objectivity
  • What Is a Confounding Variable?
  • What Is Predictive Parity?
  • False Positives vs. False Negatives in Medical Risk Models
  • What Is Supervised Learning?
  • What Is Unsupervised Learning?
  • What Is Model Drift?
  • What Is Selection Bias?
  • What Is Automation Bias?
  • What Is a ROC Curve and AUC?
  • What Is a Protected Attribute?
  • What Is a Confusion Matrix?
  • What Is Class Imbalance?
  • What Is the Bias-Variance Trade-off?
  • What Is the Base Rate Fallacy?
  • What Is Reject Inference?
  • What Is a Precision-Recall Curve?

Phase 2 - Healthcare AI Bias Examples ✅ / 🔄 In Progress

Status: Audits complete - healthcare explainers shipped alongside them

Publish healthcare-specific bias audits and explainers that show how AI discrimination shows up in clinical and insurance contexts.

  • Insurance Denial bias audit
  • Benefits Denial bias audit
  • Healthcare Readmission bias audit
  • Jupyter notebooks for all three healthcare audits
  • Explainer: Why Accuracy Is Not Enough in Healthcare AI
  • Explainer: False Positives and False Negatives in Medical Risk Models
  • Explainer: Miscalibration in Clinical Risk Scores Across Groups - when the same risk score means a different real-world risk depending on the patient's group
  • Explainer: Missing Data as Bias in Electronic Health Records - how unequal access to care turns into unequal missingness, and how models misread it
  • Explainer: Why Medical Imaging Models Fail on Underrepresented Groups - representation gaps in imaging datasets and the skin-tone / equipment confounders they hide
  • Case study write-up: Insurance Denial Bias
  • Case study write-up: Benefits Denial Bias (standalone
  • Case study write-up: Healthcare Readmission Bias
  • Explainer: Race Correction in Clinical Algorithms - why "race-adjusted" formulas (eGFR kidney function, spirometry, VBAC calculators) bake bias directly into the math
  • Explainer: The Obermeyer Case - When Cost Becomes a Proxy for Health Need - a dedicated case study of the 2019 algorithm that under-referred sicker Black patients
  • Explainer: Underdiagnosis Bias - When the Label Itself Is Sicker for One Group - why historical care gaps make the training target unequal before modeling starts

No further healthcare explainers are currently planned - the backlog from this phase is now fully shipped.

Phase 3 - Code Audits 🔄 In Progress

Status: 7 of 9 planned audits published - the remaining two are open to contribute

Each audit follows the same pipeline: train a biased model → measure the fairness gap → remove proxies → retrain → measure again. New audits now merge to main as usual (see CLAUDE.md).

  • COMPAS - Criminal Justice Bias
  • AI Fair Recruitment - Hiring Bias
  • German Credit Lending - Lending Bias
  • Insurance Denial - Healthcare Bias
  • Benefits Denial - Welfare Eligibility Bias
  • Healthcare Readmission - Clinical Bias
  • Tenant Screening - Rental Application Bias
  • LLM bias audit
  • HMDA Mortgage Lending Bias
  • Facial Recognition Accuracy Gaps (MIT Gender Shades methodology)

Phase 4 - Contributor Expansion 🔄 In Progress

Status: Goal exceeded - 29 external contributors, past the original 15+ target

Goal: grow to 15+ contributors with quality-controlled contributions.

  • CONTRIBUTING.md
  • Issue templates (bug report, new audit, new explainer)
  • PR template
  • CODE_OF_CONDUCT.md
  • CI pipeline (all audit scripts run on push/PR)
  • Good-first-issue and help-wanted labels
  • First-interaction workflow (greets new contributors)
  • 10–15 labelled issues open at all times
  • Contributor list in README
  • METRICS.md tracking contributor growth weekly

Phase 5 - Fairness Metrics and Notebooks 🔄 In Progress

Status: Cross-domain benchmark harness shipped - dashboards and notebooks continuing

Go deeper on measurement - fairness dashboards, interactive notebooks, and statistical tools for auditors.

  • Fairness audit web dashboard - Open Dataset Profiler (profiler.html)
  • Bias detection utility library (faircode/ module) - diagnostic representation profiler + CLI
  • Profiler: two-dataset comparison for representation drift (faircode compare, PSI)
  • Profiler: manual column mapping, reference-population baseline, choosable intersection pair, tunable thresholds, and chi-squared proxy hints
  • Fairlearn integration: ExponentiatedGradient in-processing + ThresholdOptimizer post-processing, as two rungs of a five-strategy mitigation ladder (S0-S4) run uniformly across every audit
  • Cross-domain benchmark harness - declarative audit.yaml manifests (faircode/MANIFEST_SPEC.md) + faircode benchmark: 5 strategies x 3 model families x 6 fairness metrics (bootstrap CI + permutation p-value) + accuracy/AUC/F1, written to results/
  • Intersectional bias notebook (auditing across multiple protected attributes simultaneously)
  • Statistical significance testing for fairness gaps
  • Fairness dashboard for the benchmark harness results (interactive results/ explorer, mirroring the Open Dataset Profiler's web/CLI split)

Phase 6 - Research Paper and Publication - Paused, Retargeted for Next Year

Status: Freeze lifted - the manuscript was never actually submitted this cycle

The original plan was to publish a peer-reviewed paper on the cross-domain fairness benchmark and freeze the repo's results against it. The manuscript submission step never actually happened, so freezing development ahead of it was premature - the freeze has been lifted (see CLAUDE.md), and this phase is retargeted for a real submission next year, built on a fresh run of results.

  • Freeze benchmark results at tag v1.0-paper (commit bbef2ba) - kept as a reference snapshot
  • CLAUDE.md paper-freeze policy for the benchmark and audits - lifted; will be re-established for next year's real submission
  • Submit manuscript to peer review (next year, with fresh results)
  • Address reviewer feedback
  • Paper accepted and published
  • Add citation and DOI to README.md and CITATION.cff
  • Development reopened: results/, audits, and the analysis core are unfrozen; paper/results-frozen/ stays untouched as the historical reference for this earlier snapshot

Content Schedule

During school:

  • Monday: AI bias explainer
  • Wednesday: Healthcare AI / fairness example
  • Friday: Code audit or project update

During holidays:

  • Monday–Friday posting acceptable if sustainable

How to Contribute

See CONTRIBUTING.md to claim an open issue or propose a new audit or explainer.

New audits are welcome and merge into main like any other contribution - the earlier freeze on new audits has lifted. Explainers, docs, and website content merge as usual too. See CLAUDE.md before opening a PR.

Fair Code is maintained by Yash Kewlani. Follow the project at @thefaircodeproject.