You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Regenerate explainer pages after the simpsons-paradox/CDP/subgroup-fairness content fixes
make build-explainers after the 3 preceding content-fix commits. Also
picks up first-commit datePublished/dateModified for all 4 explainers
added in #541 (git-log-derived; couldn't resolve until those files had
a real commit history, the same one-time chicken-and-egg noted in
check_generated_files_current.py's own docstring).
@@ -204,17 +206,17 @@ <h3 id="how-it-works">How It Works</h3>
204
206
<p>The aggregate rate is a weighted average of within-stratum rates, weighted by how group <code>g</code> is distributed across the strata of <code>L</code>. Subtracting the two groups' aggregate rates gives a gap with two sources: different within-stratum rates (<code>P(Y_hat = 1 | G = g, L = l)</code> differs by <code>g</code>), and different stratum weights (<code>P(L = l | G = g)</code> differs by <code>g</code>).</p>
205
207
<p>Conditional demographic parity zeroes out the second source by construction: it only compares groups at the same value of <code>L</code>. What is left is the within-stratum gap. A sample-size-weighted average of the within-stratum gaps is the part of the aggregate gap that stratifying on <code>L</code> does <em>not</em> remove.</p>
206
208
<p>The catch is entirely in the choice of <code>L</code>. If <code>L</code> is a genuine exogenous factor, the within-stratum gap is the disparity net of a real confounder. If <code>L</code> is a <ahref="proxy-variables.html">proxy</a> for the protected attribute, conditioning on it absorbs the discrimination into the "legitimate" term and the within-stratum gap understates the real harm. CDP gives you a knob; it does not tell you where to set it.</p>
<p><code>Benefits Denial/</code> audits the Adult Census Income dataset (<code>adult.csv</code>, 32,561 rows). The target is <code>income == '>50K'</code> and <code>sex</code> is a declared protected attribute. Using the dataset's own outcome rates (the base rate a label-level demographic parity check compares):</p>
<p>Sample-weighted within-stratum gap: <strong>+18.4 pp</strong>. Conditioning on education removes almost none of the aggregate gap - within every education level men are markedly more likely to be high earners, and at the top three levels the gap is <em>larger</em> than the unconditional +19.6. CDP conditioned on education says: the disparity is not an artifact of women being less educated in this dataset.</p>
215
217
<h4id="l-marital-status-marital-status">L = marital status (<code>marital.status</code>)</h4>
<p>Sample-weighted within-stratum gap: roughly <strong>+3 pp</strong>, and inside the single largest stratum the sign reverses (women 0.9 points ahead). Conditioning on marital status makes most of the aggregate gap disappear.</p>
219
+
<p>Sample-weighted within-stratum gap: roughly <strong>+2 pp</strong>, and inside the single largest stratum the sign reverses (women 0.9 points ahead). Conditioning on marital status makes most of the aggregate gap disappear.</p>
218
220
<h4id="the-two-answers-disagree-and-that-is-the-point">The two answers disagree, and that is the point</h4>
219
221
<p>Same protected attribute, same dataset, same +19.6 pp aggregate. Condition on education and the gap stands. Condition on marital status and it nearly vanishes. CDP did not resolve the fairness question - it moved it to a new one: <em>is marital status a legitimate basis for a 20-point income difference between men and women, or is it a proxy for a sex-role division of unpaid labor that the outcome variable should not be conditioned on?</em> That is a normative and legal question, not a statistical one, and CDP's output is only as defensible as the answer to it. (This is the same aggregate-vs-stratified divergence covered from the other direction in <ahref="simpsons-paradox.html">Simpson's Paradox</a>.)</p>
220
222
<pre><codeclass="language-python">import pandas as pd
@@ -241,7 +243,7 @@ <h4 id="the-two-answers-disagree-and-that-is-the-point">The two answers disagree
241
243
_, w = within_stratum_gap(df, col)
242
244
print(f"condition on {col:16s}: weighted within-stratum gap = {w:+.4f}")
243
245
# condition on education.num : weighted within-stratum gap = +0.1844
244
-
# condition on marital.status : weighted within-stratum gap = +0.0305</code></pre>
246
+
# condition on marital.status : weighted within-stratum gap = +0.0243</code></pre>
245
247
<h3id="detection-code">Detection Code</h3>
246
248
<p>The following module computes the unconditional demographic parity gap, the conditional (within-stratum) gap for a chosen legitimate factor, and the fraction of the unconditional gap that conditioning explains away.</p>
247
249
<pre><codeclass="language-python">import numpy as np
<p>Against <code>Benefits Denial/adult.csv</code> this reports an <code>explained_fraction</code> near <code>0.06</code> for <code>education.num</code> (conditioning removes almost nothing) and near <code>0.84</code> for <code>marital.status</code> (conditioning removes most of the gap) - the same +19.6 pp aggregate, two incompatible readings.</p>
338
+
<p>Against <code>Benefits Denial/adult.csv</code> this reports an <code>explained_fraction</code> near <code>0.06</code> for <code>education.num</code> (conditioning removes almost nothing) and near <code>0.88</code> for <code>marital.status</code> (conditioning removes most of the gap) - the same +19.6 pp aggregate, two incompatible readings.</p>
337
339
<h3id="limitations-and-trade-offs">Limitations and Trade-offs</h3>
338
340
<h4id="1-the-conditioning-set-is-a-value-judgment-not-a-statistical-choice">1. The conditioning set is a value judgment, not a statistical choice</h4>
339
341
<p>Everything CDP reports is downstream of which factors you declared "legitimate". There is no test that tells you whether marital status, or occupation, or prior loan history, is a fair thing to condition on. Pick a factor that is genuinely a proxy for the protected attribute and CDP will confidently report that the disparity is "explained", which is exactly the <ahref="proxy-variables.html">proxy-variable</a> trap.</p>
0 commit comments