Skip to content

Commit da315ca

Browse files
Fix stale Output Interpretation numbers in underdiagnosis-bias.md (closes #523)
The 'Output Interpretation' prose cited figures from an earlier version of the detection-code block that were never regenerated after the code was edited. Running the file's own python block (seeded with np.random.seed(42), byte-identical across repeated runs) actually prints: - true_recall 0.625 (Group A) vs 0.292 (Group B) - a 33.3-point gap, not the claimed 80.5% / 41.2% / 39.3-point gap - obs_recall ~0.64 vs ~0.33, not the claimed ~0.85 / ~0.76 - Q4 biomarker-band recorded-diagnosis rate 0.785 / 0.497, not 0.887 / 0.536 Updated the two interpretation bullets to the real numbers and adjusted the framing (the observed-label recall already shows a gap here, so the 'looks fine on observed labels' line no longer fit). Regenerated underdiagnosis-bias.html, the package mirror copy, and the llms-full.txt section from the build script.
1 parent fb8a2c0 commit da315ca

4 files changed

Lines changed: 7 additions & 7 deletions

File tree

explainers/underdiagnosis-bias.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -353,7 +353,7 @@ <h3 id="detection-code">Detection Code</h3>
353353
print(&quot;Diagnostic label rate by biomarker band:&quot;)
354354
print(diagnosis_rates)</code></pre>
355355
<h4 id="output-interpretation">Output Interpretation</h4>
356-
<p>1. <strong>The Evaluation Trap</strong>: On observed EHR labels (<code>obs_recall</code>), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (<code>true_recall</code>), the model catches <strong>80.5%</strong> of sick Group A patients versus only <strong>41.2%</strong> of sick Group B patients - a <strong>39.3-point true false-negative gap</strong> hidden from standard evaluation. 2. <strong>The Biomarker Audit</strong>: In the highest biomarker band (Q4), Group A patients have a <strong>0.887</strong> diagnosis rate while Group B has a <strong>0.536</strong> rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.</p>
356+
<p>1. <strong>The Evaluation Trap</strong>: Against observed EHR labels (<code>obs_recall</code>), the model already recalls Group B worse than Group A (<strong>~0.64</strong> vs <strong>~0.33</strong>), but standard evaluation reads that as the model faithfully matching a lower &quot;disease rate&quot; in Group B. Against true disease status (<code>true_recall</code>), the real picture is worse: the model catches <strong>62.5%</strong> of sick Group A patients versus only <strong>29.2%</strong> of sick Group B patients - a <strong>33.3-point true false-negative gap</strong>, most of which standard evaluation scores as correct True Negatives. 2. <strong>The Biomarker Audit</strong>: In the highest biomarker band (Q4), Group A patients have a <strong>0.785</strong> recorded-diagnosis rate while Group B has a <strong>0.497</strong> rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.</p>
357357
<h3 id="limitations">Limitations</h3>
358358
<h4 id="1-unobserved-true-disease-state-y">1. Unobserved True Disease State (Y*)</h4>
359359
<p>In observational healthcare data, true disease status Y* is rarely recorded. Identifying underdiagnosis requires objective proxy biomarkers (e.g., lab results, physiological waveforms), prospective screening studies, or external clinical audit samples.</p>

explainers/underdiagnosis-bias.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -215,8 +215,8 @@ print(diagnosis_rates)
215215

216216
### Output Interpretation
217217

218-
1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation.
219-
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.
218+
1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives.
219+
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.
220220

221221
## Limitations
222222

faircode/_explainers/underdiagnosis-bias.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -215,8 +215,8 @@ print(diagnosis_rates)
215215

216216
### Output Interpretation
217217

218-
1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation.
219-
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.
218+
1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives.
219+
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.
220220

221221
## Limitations
222222

llms-full.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8877,8 +8877,8 @@ print(diagnosis_rates)
88778877

88788878
### Output Interpretation
88798879

8880-
1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation.
8881-
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.
8880+
1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives.
8881+
2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.
88828882

88838883
## Limitations
88848884

0 commit comments

Comments
 (0)