From da315ca46a68ec6be1ec85776c9733cef5451a7f Mon Sep 17 00:00:00 2001 From: propcgamer20-png Date: Thu, 10 Sep 2026 15:12:22 +0530 Subject: [PATCH] Fix stale Output Interpretation numbers in underdiagnosis-bias.md (closes #523) The 'Output Interpretation' prose cited figures from an earlier version of the detection-code block that were never regenerated after the code was edited. Running the file's own python block (seeded with np.random.seed(42), byte-identical across repeated runs) actually prints: - true_recall 0.625 (Group A) vs 0.292 (Group B) - a 33.3-point gap, not the claimed 80.5% / 41.2% / 39.3-point gap - obs_recall ~0.64 vs ~0.33, not the claimed ~0.85 / ~0.76 - Q4 biomarker-band recorded-diagnosis rate 0.785 / 0.497, not 0.887 / 0.536 Updated the two interpretation bullets to the real numbers and adjusted the framing (the observed-label recall already shows a gap here, so the 'looks fine on observed labels' line no longer fit). Regenerated underdiagnosis-bias.html, the package mirror copy, and the llms-full.txt section from the build script. --- explainers/underdiagnosis-bias.html | 2 +- explainers/underdiagnosis-bias.md | 4 ++-- faircode/_explainers/underdiagnosis-bias.md | 4 ++-- llms-full.txt | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/explainers/underdiagnosis-bias.html b/explainers/underdiagnosis-bias.html index 643e291..9cee562 100644 --- a/explainers/underdiagnosis-bias.html +++ b/explainers/underdiagnosis-bias.html @@ -353,7 +353,7 @@

Detection Code

print("Diagnostic label rate by biomarker band:") print(diagnosis_rates)

Output Interpretation

-

1. The Evaluation Trap: On observed EHR labels (obs_recall), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (true_recall), the model catches 80.5% of sick Group A patients versus only 41.2% of sick Group B patients - a 39.3-point true false-negative gap hidden from standard evaluation. 2. The Biomarker Audit: In the highest biomarker band (Q4), Group A patients have a 0.887 diagnosis rate while Group B has a 0.536 rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records.

+

1. The Evaluation Trap: Against observed EHR labels (obs_recall), the model already recalls Group B worse than Group A (~0.64 vs ~0.33), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (true_recall), the real picture is worse: the model catches 62.5% of sick Group A patients versus only 29.2% of sick Group B patients - a 33.3-point true false-negative gap, most of which standard evaluation scores as correct True Negatives. 2. The Biomarker Audit: In the highest biomarker band (Q4), Group A patients have a 0.785 recorded-diagnosis rate while Group B has a 0.497 rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label.

Limitations

1. Unobserved True Disease State (Y*)

In observational healthcare data, true disease status Y* is rarely recorded. Identifying underdiagnosis requires objective proxy biomarkers (e.g., lab results, physiological waveforms), prospective screening studies, or external clinical audit samples.

diff --git a/explainers/underdiagnosis-bias.md b/explainers/underdiagnosis-bias.md index c4d36c9..d9778f6 100644 --- a/explainers/underdiagnosis-bias.md +++ b/explainers/underdiagnosis-bias.md @@ -215,8 +215,8 @@ print(diagnosis_rates) ### Output Interpretation -1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation. -2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records. +1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives. +2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label. ## Limitations diff --git a/faircode/_explainers/underdiagnosis-bias.md b/faircode/_explainers/underdiagnosis-bias.md index c4d36c9..d9778f6 100644 --- a/faircode/_explainers/underdiagnosis-bias.md +++ b/faircode/_explainers/underdiagnosis-bias.md @@ -215,8 +215,8 @@ print(diagnosis_rates) ### Output Interpretation -1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation. -2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records. +1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives. +2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label. ## Limitations diff --git a/llms-full.txt b/llms-full.txt index dd42aae..54b1f74 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -8877,8 +8877,8 @@ print(diagnosis_rates) ### Output Interpretation -1. **The Evaluation Trap**: On observed EHR labels (`obs_recall`), the model appears to perform decently across both groups (~0.85 vs ~0.76). But on true disease status (`true_recall`), the model catches **80.5%** of sick Group A patients versus only **41.2%** of sick Group B patients - a **39.3-point true false-negative gap** hidden from standard evaluation. -2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.887** diagnosis rate while Group B has a **0.536** rate. Disparities in diagnostic coding among patients with matching objective clinical values flag underdiagnosis bias directly from EHR records. +1. **The Evaluation Trap**: Against observed EHR labels (`obs_recall`), the model already recalls Group B worse than Group A (**~0.64** vs **~0.33**), but standard evaluation reads that as the model faithfully matching a lower "disease rate" in Group B. Against true disease status (`true_recall`), the real picture is worse: the model catches **62.5%** of sick Group A patients versus only **29.2%** of sick Group B patients - a **33.3-point true false-negative gap**, most of which standard evaluation scores as correct True Negatives. +2. **The Biomarker Audit**: In the highest biomarker band (Q4), Group A patients have a **0.785** recorded-diagnosis rate while Group B has a **0.497** rate, despite sharing the same objective clinical values. Disparities in diagnostic coding within matched biomarker bands flag underdiagnosis bias directly from EHR records, without needing the unobserved true label. ## Limitations