fix: table parser honors escaped pipes; JS renderMarkdown matches heading offset

#552: build_explainers.py's split_row and explainers-ui.js's parseTable
both did a naive split("|") on every pipe in a table row, with no support
for GFM's "\|" escape. A literal pipe inside a cell (e.g. "P(S = 1 | X)")
started a spurious column, shifting every later column - already visibly
misrendering explainers/reject-inference.html (an IPW row with 6 <td> for
a 4-column table) and explainers/base-rate-fallacy.html (a "PPV
(P(Y = 1 | Ŷ = 1))" header split into two <th>). Both parsers now split
on unescaped "|" only and unescape "\|" -> "|" in each cell.
reject-inference.md's source pipes are now escaped as "\|"
(base-rate-fallacy.md already had them escaped); pages regenerated.

#553: a prior fix offset build_explainers.py's markdown headings by +1
(the page hero already renders a real <h1>), but explainers-ui.js's
renderMarkdown was never updated and still emitted raw levels. It now
applies the same Math.min(level + 1, 6). This path isn't reached in
production today, but "fixing" it later under the assumption it matched
the server renderer would have reintroduced the duplicate-<h1> bug.

Closes #552
Closes #553

Author: propcgamer20-png

assets/explainers-ui.js

@@ -109,8 +109,14 @@
       return null;
     }
 
-    const headers = rows[0].split('|').slice(1, -1).map(cell => cell.trim());
-    const bodyRows = rows.slice(2).map(row => row.split('|').slice(1, -1).map(cell => cell.trim()));
+    // Split on unescaped "|" only, then unescape "\|" -> "|" in each cell, so
+    // a literal pipe inside a cell (GFM's "\|") no longer starts a spurious
+    // column. Mirrors scripts/build_explainers.py's split_row().
+    const splitRow = row => row.trim().split(/(?<!\\)\|/).slice(1, -1)
+      .map(cell => cell.trim().replace(/\\\|/g, '|'));
+
+    const headers = splitRow(rows[0]);
+    const bodyRows = rows.slice(2).map(splitRow);
 
     const headerHtml = headers.map(cell => `<th>${inlineMarkdown(cell)}</th>`).join('');
     const bodyHtml = bodyRows.map(row => `<tr>${row.map(cell => `<td>${inlineMarkdown(cell)}</td>`).join('')}</tr>`).join('');
@@ -214,7 +220,9 @@
         flushParagraph();
         flushList();
         flushQuote();
-        const level = headingMatch[1].length;
+        // +1 offset: the explainer page's hero already renders a real <h1>.
+        // Mirrors scripts/build_explainers.py's render_markdown().
+        const level = Math.min(headingMatch[1].length + 1, 6);
         const headingText = headingMatch[2];
         const baseId = slugifyHeading(headingText);
         const nextCount = (headingCounts.get(baseId) || 0) + 1;

explainers/base-rate-fallacy.html

@@ -205,7 +205,7 @@ <h3 id="the-mathematics-of-the-base-rate-fallacy">The Mathematics of the Base Ra
 <pre><code>PPV = P(Y = 1 | Ŷ = 1) = (TPR * p) / (TPR * p + FPR * (1 - p))</code></pre>
 <h4 id="prevalence-impact-on-reliability">Prevalence Impact on Reliability</h4>
 <p>To see how background prevalence dictates prediction reliability, consider a screening model with fixed TPR = 0.90 and FPR = 0.10 evaluated across varying base rates (p):</p>
-<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Base Rate (p)</th><th>True Positives (TPR * p)</th><th>False Positives (FPR * (1 - p))</th><th>PPV (P(Y = 1 \</th><th>Ŷ = 1))</th><th>False Discovery Rate (1 - PPV)</th></tr></thead><tbody><tr><td><strong>1%</strong></td><td>0.0090</td><td>0.0990</td><td><strong>8.33%</strong></td><td><strong>91.67%</strong></td></tr><tr><td><strong>5%</strong></td><td>0.0450</td><td>0.0950</td><td><strong>32.14%</strong></td><td><strong>67.86%</strong></td></tr><tr><td><strong>10%</strong></td><td>0.0900</td><td>0.0900</td><td><strong>50.00%</strong></td><td><strong>50.00%</strong></td></tr><tr><td><strong>30%</strong></td><td>0.2700</td><td>0.0700</td><td><strong>79.41%</strong></td><td><strong>20.59%</strong></td></tr><tr><td><strong>50%</strong></td><td>0.4500</td><td>0.0500</td><td><strong>90.00%</strong></td><td><strong>10.00%</strong></td></tr></tbody></table></div>
+<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Base Rate (p)</th><th>True Positives (TPR * p)</th><th>False Positives (FPR * (1 - p))</th><th>PPV (P(Y = 1 | Ŷ = 1))</th><th>False Discovery Rate (1 - PPV)</th></tr></thead><tbody><tr><td><strong>1%</strong></td><td>0.0090</td><td>0.0990</td><td><strong>8.33%</strong></td><td><strong>91.67%</strong></td></tr><tr><td><strong>5%</strong></td><td>0.0450</td><td>0.0950</td><td><strong>32.14%</strong></td><td><strong>67.86%</strong></td></tr><tr><td><strong>10%</strong></td><td>0.0900</td><td>0.0900</td><td><strong>50.00%</strong></td><td><strong>50.00%</strong></td></tr><tr><td><strong>30%</strong></td><td>0.2700</td><td>0.0700</td><td><strong>79.41%</strong></td><td><strong>20.59%</strong></td></tr><tr><td><strong>50%</strong></td><td>0.4500</td><td>0.0500</td><td><strong>90.00%</strong></td><td><strong>10.00%</strong></td></tr></tbody></table></div>
 <p>At a 1% base rate, <strong>over 91% of flagged individuals are false alarms</strong>, despite the model having 90% sensitivity and 90% specificity.</p>
 <h4 id="the-chouldechova-impossibility-identity">The Chouldechova Impossibility Identity</h4>
 <p>When evaluating models across demographic groups A and B, Chouldechova (2017) demonstrated that the false positive rate (FPR), false negative rate (FNR), positive predictive value (PPV), and base rate (p) are linked by a strict identity:</p>

explainers/reject-inference.html

@@ -226,7 +226,7 @@ <h4 id="the-missingness-mechanism-missing-not-at-random-mnar">The Missingness Me
 <p>If a bank historically required younger applicants to meet a higher credit bar than older applicants, then the younger applicants present in the approved dataset (<code>S = 1</code>) represent an artificially selected, ultra-qualified subset of all young applicants. A model trained on this sample will overestimate the credit standards required for young borrowers to succeed.</p>
 <h4 id="core-reject-inference-techniques">Core Reject Inference Techniques</h4>
 <p>Practitioners use four main statistical approaches to correct for reject inference:</p>
-<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Method</th><th>Core Mechanism</th><th>Strengths</th><th>Key Vulnerability</th></tr></thead><tbody><tr><td><strong>Hard Parceling (Pseudo-Labeling)</strong></td><td>Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows.</td><td>Simple to implement in standard ML pipelines.</td><td>Propagates initial model errors and thresholding artifacts into retraining.</td></tr><tr><td><strong>Soft Parceling / Fuzzy Augmentation</strong></td><td>Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases.</td><td>Avoids hard threshold cutoffs; preserves prediction uncertainty.</td><td>Dilutes training signal if initial model probability estimates are miscalibrated.</td></tr><tr><td><strong>Inverse Probability Weighting (IPW)</strong></td><td>Estimate selection propensity w(X) = P(S = 1</td><td>X); weight approved cases by 1 / w(X) during training.</td><td>Theoretically unbiased under Missing At Random (MAR) assumptions.</td><td>Extreme weights when propensity P(S = 1</td><td>X) ≈ 0 create high estimator variance.</td></tr><tr><td><strong>Heckman Two-Stage Model</strong></td><td>Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε).</td><td>Explicitly models unobserved selection correlation ρ.</td><td>Relies heavily on bivariate normality and valid exclusion restrictions (Z).</td></tr></tbody></table></div>
+<div class="explainer-table-wrap"><table class="explainer-table"><thead><tr><th>Method</th><th>Core Mechanism</th><th>Strengths</th><th>Key Vulnerability</th></tr></thead><tbody><tr><td><strong>Hard Parceling (Pseudo-Labeling)</strong></td><td>Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows.</td><td>Simple to implement in standard ML pipelines.</td><td>Propagates initial model errors and thresholding artifacts into retraining.</td></tr><tr><td><strong>Soft Parceling / Fuzzy Augmentation</strong></td><td>Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases.</td><td>Avoids hard threshold cutoffs; preserves prediction uncertainty.</td><td>Dilutes training signal if initial model probability estimates are miscalibrated.</td></tr><tr><td><strong>Inverse Probability Weighting (IPW)</strong></td><td>Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training.</td><td>Theoretically unbiased under Missing At Random (MAR) assumptions.</td><td>Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance.</td></tr><tr><td><strong>Heckman Two-Stage Model</strong></td><td>Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε).</td><td>Explicitly models unobserved selection correlation ρ.</td><td>Relies heavily on bivariate normality and valid exclusion restrictions (Z).</td></tr></tbody></table></div>
 <hr>
 <h3 id="concrete-example-german-credit-lending-audit-03">Concrete Example: German Credit Lending - Audit 03</h3>
 <p><a href="../index.html#project-credit"><code>German Credit Lending/credit_customers.csv</code></a> is the dataset behind Audit 03 in this repository. Its <code>class</code> column contains exactly two values across all 1,000 rows: <code>good</code> (700 rows) and <code>bad</code> (300 rows).</p>

explainers/reject-inference.md

@@ -65,7 +65,7 @@ Practitioners use four main statistical approaches to correct for reject inferen
 |---|---|---|---|
 | **Hard Parceling (Pseudo-Labeling)** | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. |
 | **Soft Parceling / Fuzzy Augmentation** | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. |
-| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. |
+| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 \| X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 \| X) ≈ 0 create high estimator variance. |
 | **Heckman Two-Stage Model** | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). |
 
 ---

faircode/_explainers/reject-inference.md

@@ -65,7 +65,7 @@ Practitioners use four main statistical approaches to correct for reject inferen
 |---|---|---|---|
 | **Hard Parceling (Pseudo-Labeling)** | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. |
 | **Soft Parceling / Fuzzy Augmentation** | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. |
-| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. |
+| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 \| X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 \| X) ≈ 0 create high estimator variance. |
 | **Heckman Two-Stage Model** | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). |
 
 ---

llms-full.txt

@@ -9235,7 +9235,7 @@ Practitioners use four main statistical approaches to correct for reject inferen
 |---|---|---|---|
 | **Hard Parceling (Pseudo-Labeling)** | Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows. | Simple to implement in standard ML pipelines. | Propagates initial model errors and thresholding artifacts into retraining. |
 | **Soft Parceling / Fuzzy Augmentation** | Assign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases. | Avoids hard threshold cutoffs; preserves prediction uncertainty. | Dilutes training signal if initial model probability estimates are miscalibrated. |
-| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 | X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 | X) ≈ 0 create high estimator variance. |
+| **Inverse Probability Weighting (IPW)** | Estimate selection propensity w(X) = P(S = 1 \| X); weight approved cases by 1 / w(X) during training. | Theoretically unbiased under Missing At Random (MAR) assumptions. | Extreme weights when propensity P(S = 1 \| X) ≈ 0 create high estimator variance. |
 | **Heckman Two-Stage Model** | Stage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε). | Explicitly models unobserved selection correlation ρ. | Relies heavily on bivariate normality and valid exclusion restrictions (Z). |
 
 ---

scripts/build_explainers.py

@@ -142,7 +142,12 @@ def parse_table(lines, start_index):
         return None
 
     def split_row(row):
-        return [cell.strip() for cell in row.split("|")[1:-1]]
+        # Split on unescaped "|" only, then unescape "\|" -> "|" in each cell,
+        # so a literal pipe inside a cell (GFM's "\|") no longer starts a
+        # spurious column. Rows carry a leading and trailing "|", so the
+        # first and last split fragments are empty and dropped.
+        cells = re.split(r"(?<!\\)\|", row.strip())[1:-1]
+        return [cell.strip().replace("\\|", "|") for cell in cells]
 
     headers = split_row(rows[0])
     body_rows = [split_row(row) for row in rows[2:]]

tests/test_build_explainers.py

@@ -1,5 +1,24 @@
 import importlib
 import json
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+
+
+def test_explainers_ui_js_stays_in_render_parity_with_build_explainers():
+    # assets/explainers-ui.js's renderMarkdown/parseTable is a client-side
+    # port of scripts/build_explainers.py's render_markdown/split_row, but
+    # nothing exercises it on a real explainer page, so it has silently
+    # drifted before (#552, #553). Source-level guard (same idea as
+    # test_js_parity's DOM-renderer checks) that the two fixes are present in
+    # the JS too.
+    js = (REPO_ROOT / "assets" / "explainers-ui.js").read_text(encoding="utf-8")
+
+    # #553: +1 heading offset, capped at h6
+    assert "Math.min(headingMatch[1].length + 1, 6)" in js
+    # #552: split table rows on unescaped "|" and unescape "\|" -> "|"
+    assert r"/(?<!\\)\|/" in js
+    assert r"replace(/\\\|/g, '|')" in js
 
 
 def test_build_package_mirror_copies_data_and_markdown(tmp_path, monkeypatch):
@@ -84,3 +103,33 @@ def test_parse_table_still_accepts_three_dash_separator_row():
     assert result is not None
     headers, _body_rows, _next_index = result
     assert headers == ["A", "B"]
+
+
+def test_parse_table_honors_escaped_pipe_inside_a_cell():
+    # A literal pipe in a cell must be written "\|" (GFM) and must not start a
+    # new column; the "\" is stripped in the rendered cell. reject-inference.md
+    # and base-rate-fallacy.md both hit this (#552).
+    script = importlib.import_module("scripts.build_explainers")
+    lines = [
+        "| Method | Formula |",
+        "|---|---|",
+        r"| IPW | w(X) = P(S = 1 \| X) |",
+    ]
+
+    headers, body_rows, _ = script.parse_table(lines, 0)
+
+    assert headers == ["Method", "Formula"]
+    assert body_rows == [["IPW", "w(X) = P(S = 1 | X)"]]
+
+
+def test_render_markdown_offsets_heading_levels_by_one():
+    # The explainer page's hero already renders a real <h1>, so the markdown
+    # body's headings are shifted down one level (h1 -> h2, capped at h6).
+    # assets/explainers-ui.js's renderMarkdown must match this (#553).
+    script = importlib.import_module("scripts.build_explainers")
+
+    html = script.render_markdown("# Top\n\n## Sub\n\n###### Deep\n", set())
+
+    assert '<h2 id="top">Top</h2>' in html
+    assert '<h3 id="sub">Sub</h3>' in html
+    assert '<h6 id="deep">Deep</h6>' in html   # h6 + 1 stays h6, not h7