Skip to content

Commit 3c984a6

Browse files
docs: fix two Results-at-a-Glance inaccuracies (COMPAS reduction %, Benefits Denial proxy count)
#545: COMPAS's "Reduction" figure used the raw percentage-point gap (86.77 - 15.69 = 71.08 -> "71%") instead of the relative reduction (86.77 - 15.69) / 86.77 = 81.9% -> "82%") that every other row in the same table uses. Fixed in COMPAS/README.md, the root README's Results-at-a-Glance row, and the "Result: N% reduction" line in the root README's COMPAS section. This understated COMPAS's actual mitigation effectiveness. #546: Benefits Denial's audit.yaml lists five proxy_features (marital.status, occupation, relationship, hours.per.week, fnlwgt) and fair.py removes all five with documented reasoning, but both READMEs listed only four, omitting fnlwgt. Added fnlwgt to the proxy list in Benefits Denial/README.md and the root README (summary-table row, "five proxy variables" counts, and the fair.py code-block comment list). Closes #545 Closes #546
1 parent 9cd70fc commit 3c984a6

3 files changed

Lines changed: 8 additions & 7 deletions

File tree

Benefits Denial/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ python3 "Benefits Denial/fair.py" # mitigated (protected attribute + proxies
1919
## What the audit controls
2020

2121
- Protected attribute(s): Sex, Race, Origin, Age
22-
- Proxy feature(s) removed in `fair.py`: Relationship, Marital Status, Hours, Occupation
22+
- Proxy feature(s) removed in `fair.py`: Relationship, Marital Status, Hours, Occupation, fnlwgt (census sampling weight)
2323
- Fairness metric: Demographic Parity (difference in positive-prediction rate between groups)
2424

2525
## Expected result (published, paper-aligned)

COMPAS/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,6 @@ python3 "COMPAS/fair.py" # mitigated (protected attribute + proxies dropped)
2626

2727
| Group | Gap, biased (`unfair.py`) | Gap, mitigated (`fair.py`) | Reduction |
2828
|-------|--------------------------:|---------------------------:|----------:|
29-
| Race | 86.77% | 15.69% | 71% |
29+
| Race | 86.77% | 15.69% | 82% |
3030

3131
These match the "Results at a Glance" table in the [main README](../README.md#results-at-a-glance) and the frozen snapshot in `paper/results-frozen/`. The scripts are deterministic at `random_state=42`, so a correct local run reproduces them exactly. If your numbers differ, check the seed, the split, and your package versions before opening an issue - and never edit the frozen numbers to match a local run (see [CLAUDE.md](../CLAUDE.md)).

README.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -68,12 +68,12 @@ Each audit ships as both a pair of Python scripts (`unfair.py` / `fair.py`) for
6868

6969
| # | Domain | Protected Attribute | Proxies Removed | Gap Before → After | Reduction |
7070
|:-:|--------|--------------------|-----------------|--------------------|:---------:|
71-
| 01 | [Criminal Justice](#01--compas--criminal-justice-bias) | Race | Custody Status | 86.77% → 15.69% | **71%** |
71+
| 01 | [Criminal Justice](#01--compas--criminal-justice-bias) | Race | Custody Status | 86.77% → 15.69% | **82%** |
7272
| 02 | [Hiring](#02--ai-fair-recruitment--hiring-bias) | Gender | Age | 4.51% → 0.12% | **97.3%** |
7373
| 03 | [Lending](#03--german-credit-lending--lending-bias) | Age | Employment Tenure | 7.16% → 1.89% | **73.6%** |
7474
| 04 | [Healthcare](#04--insurance-denial--healthcare-bias) | Age, Gender | BMI, Smoker, Diabetic | Age: 7.93% → 3.18% | **60%** |
7575
|| | | | Gender: 5.44% → 1.54% | **72%** |
76-
| 05 | [Welfare](#05--benefits-denial--welfare-eligibility-bias) | Sex, Race, Origin, Age | Relationship, Marital Status, Hours, Occupation | Sex: 18.00% → 8.52% | **53%** |
76+
| 05 | [Welfare](#05--benefits-denial--welfare-eligibility-bias) | Sex, Race, Origin, Age | Relationship, Marital Status, Hours, Occupation, fnlwgt | Sex: 18.00% → 8.52% | **53%** |
7777
|| | | | Race: 12.75% → 6.90% | **46%** |
7878
|| | | | Origin: 4.40% → 0.52% | **88%** |
7979
| 06 | [Healthcare Readmission](#06--healthcare-readmission--clinical-bias) | Race, Gender, Age | Payer Code, Discharge Disposition, Medical Specialty, Prior Inpatient | Gender: 0.02% → 0.04% | **+100% ↑** |
@@ -371,7 +371,7 @@ X = pd.get_dummies(df[[
371371
| White Defendants | 69.02% |
372372
| **New Fairness Gap** | **15.69%** |
373373

374-
**Result: 71% reduction in the fairness gap.**
374+
**Result: 82% reduction in the fairness gap.**
375375

376376
> **Key insight:** Removing race alone isn't enough. Proxy variables like custody status carry the same racial signal because of historical over-policing of Black communities. Both the protected attribute *and* its proxies must be removed.
377377
@@ -546,7 +546,7 @@ Automated welfare and benefits systems use income-prediction models to screen ap
546546

547547
#### The Problem - `unfair.py`
548548

549-
Trained with sex, race, age, and national origin directly, plus four proxy variables that reconstruct those attributes even after the protected columns are removed.
549+
Trained with sex, race, age, and national origin directly, plus five proxy variables that reconstruct those attributes even after the protected columns are removed.
550550

551551
| Group | Ineligibility Flag Rate |
552552
|-------|:-----------------------:|
@@ -568,7 +568,7 @@ Trained with sex, race, age, and national origin directly, plus four proxy varia
568568

569569
#### The Fix - `fair.py`
570570

571-
Dropped all four protected attributes and all four proxy variables. Retained only the features a means-tested programme can legitimately consult under equality law.
571+
Dropped all four protected attributes and all five proxy variables. Retained only the features a means-tested programme can legitimately consult under equality law.
572572

573573
```python
574574
# THE FIX: Policy-defined economic signals only
@@ -586,6 +586,7 @@ features = [
586586
# marital.status removed ✓ (proxy: encodes sex via spousal status)
587587
# hours.per.week removed ✓ (proxy: encodes sex via caregiving gap)
588588
# occupation removed ✓ (proxy: encodes race via occupational segregation)
589+
# fnlwgt removed ✓ (proxy: census sampling weight, no causal link)
589590
]
590591
```
591592

0 commit comments

Comments
 (0)