Skip to content

fix: non-numeric age sentinels get their own categorical group - #509

Merged
yakew7 merged 1 commit into
yakew7:mainfrom
propcgamer20-png:fix-age-categorical-sentinels
Sep 9, 2026
Merged

fix: non-numeric age sentinels get their own categorical group#509
yakew7 merged 1 commit into
yakew7:mainfrom
propcgamer20-png:fix-age-categorical-sentinels

Conversation

@propcgamer20-png

Copy link
Copy Markdown
Collaborator

_age_to_numeric() returned None for both a genuinely missing cell and free text with no embedded number at all (e.g. "unknown", "prefer not to say") - the age-banding loop then counted every None straight into null_count, indistinguishable from real missing data. Contradicted SPEC.md section 2's own documented "anything else: treat as categorical" rule for exactly this case.

$ python3 -c "
import pandas as pd
from faircode.profiler import profile
df = pd.DataFrame({'age': [25,30,45,'unknown','unknown','unknown','prefer not to say',22,33,41]})
d = profile(df)['dimensions'][0]
print(d['n_groups'], d['missing_pct'])
"
3 0.4 # (before) - 4 real responses vanish into missing_pct
5 0.0 # (after) - 'unknown' (3) and 'prefer not to say' (1) get their own groups

New _is_categorical_age_sentinel() (faircode/profiler.py) / isCategoricalAgeSentinel() (assets/profiler-engine.js) distinguish three cases per unparseable value: None/NaN (missing), a numeric value out of the valid age range like a -1/999 sentinel (also missing - matches this profiler's existing, tested behavior for range-invalid numeric sentinels), and free text with no embedded number at all (categorical). Only the third case gets its own group.

First attempt at this used pd.isna()/typeof-only-numeric as the distinguishing check and broke
test_negative_age_sentinels_are_missing_instead_of_an_elderly_group - a -1 sentinel isn't NaN, so it was wrongly routed to a new '-1' categorical group instead of staying missing. Caught by running the full profiler suite before committing; fixed by checking for an embedded number via the same regex _age_to_numeric()/ageToNumeric() already use, not just NaN-ness.

Verified Python/JS parity directly (not just via the test suite): built a CSV from the issue's exact repro and ran both
faircode.profiler.profile() and 'node scripts/engine-js.js profile' against it - identical n_groups, missing_pct, and group labels/counts on both engines.

Added two regression tests to tests/test_profiler.py: the issue's exact repro, and a mixed case (real None/NaN alongside a text sentinel) confirming missing_pct only counts the genuine blanks. Full suite: tests/test_profiler.py 38/38, tests/test_js_parity.py 21 passed + 1 skipped, full suite 401 passed/1 skipped/4 failed (the 4 are tests/test_xlsx_edge_cases.py, confirmed pre-existing and unrelated - SheetJS CDN blocked in this sandbox).

Known follow-up, out of scope here: _intersections()'s labelize() helper (used by compare(), not profile()) still routes these same sentinel values through ageBand()/None the old way for cross-tabulation purposes. The issue's own repro and acceptance criteria are profile()- only; left as-is rather than silently expanding scope.

Fixes #487

Summary

Type

  • Audit
  • Explainer
  • Bug fix
  • Other

Audit checklist

  • I opened or linked a corresponding issue first
  • The folder is named after the domain, not the dataset
  • unfair.py includes protected attributes and prints the required output format
  • fair.py removes protected attributes and identified proxy variables
  • Both scripts use random_state=42 and an 80/20 train/test split
  • Proxy variables were actually tested, not just guessed
  • unfair.png and fair.png are included as PNG screenshots
  • The dataset is public and accessible without login or payment
  • The dataset file is included, or DATA.md is included if the file is too large
  • README.md includes the new results row and audit section
  • A notebook was added if the audit benefits from one

Before fairness gap:

After fairness gap:

Reduction:

Protected attribute(s):

Proxy variables dropped:


Explainer checklist

  • The file is in explainers/ and uses lowercase hyphenated naming
  • It includes a plain-language definition
  • It uses a real example from this repo or a documented real-world case
  • It includes runnable Python detection or measurement code
  • It acknowledges limitations or trade-offs
  • It links to related explainers or repo projects
  • It includes 2-3 primary sources
  • The Explainers table in README.md was updated

Linked issue

Closes #

_age_to_numeric() returned None for both a genuinely missing cell and
free text with no embedded number at all (e.g. "unknown", "prefer not
to say") - the age-banding loop then counted every None straight into
null_count, indistinguishable from real missing data. Contradicted
SPEC.md section 2's own documented "anything else: treat as
categorical" rule for exactly this case.

  $ python3 -c "
  import pandas as pd
  from faircode.profiler import profile
  df = pd.DataFrame({'age': [25,30,45,'unknown','unknown','unknown','prefer not to say',22,33,41]})
  d = profile(df)['dimensions'][0]
  print(d['n_groups'], d['missing_pct'])
  "
  3 0.4   # (before) - 4 real responses vanish into missing_pct
  5 0.0   # (after)  - 'unknown' (3) and 'prefer not to say' (1) get their own groups

New _is_categorical_age_sentinel() (faircode/profiler.py) /
isCategoricalAgeSentinel() (assets/profiler-engine.js) distinguish three
cases per unparseable value: None/NaN (missing), a numeric value out of
the valid age range like a -1/999 sentinel (also missing - matches this
profiler's existing, tested behavior for range-invalid numeric
sentinels), and free text with no embedded number at all (categorical).
Only the third case gets its own group.

First attempt at this used pd.isna()/typeof-only-numeric as the
distinguishing check and broke
test_negative_age_sentinels_are_missing_instead_of_an_elderly_group - a
-1 sentinel isn't NaN, so it was wrongly routed to a new '-1' categorical
group instead of staying missing. Caught by running the full profiler
suite before committing; fixed by checking for an embedded number via
the same regex _age_to_numeric()/ageToNumeric() already use, not just
NaN-ness.

Verified Python/JS parity directly (not just via the test suite): built
a CSV from the issue's exact repro and ran both
faircode.profiler.profile() and 'node scripts/engine-js.js profile'
against it - identical n_groups, missing_pct, and group
labels/counts on both engines.

Added two regression tests to tests/test_profiler.py: the issue's exact
repro, and a mixed case (real None/NaN alongside a text sentinel)
confirming missing_pct only counts the genuine blanks. Full suite:
tests/test_profiler.py 38/38, tests/test_js_parity.py 21 passed + 1
skipped, full suite 401 passed/1 skipped/4 failed (the 4 are
tests/test_xlsx_edge_cases.py, confirmed pre-existing and unrelated -
SheetJS CDN blocked in this sandbox).

Known follow-up, out of scope here: _intersections()'s labelize()
helper (used by compare(), not profile()) still routes these same
sentinel values through ageBand()/None the old way for cross-tabulation
purposes. The issue's own repro and acceptance criteria are profile()-
only; left as-is rather than silently expanding scope.

Fixes yakew7#487
@vercel

vercel Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@propcgamer20-png is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

@yakew7 @ahmdkaml - new PR touching a path you own, please review.

@yakew7
yakew7 merged commit bbf493b into yakew7:main Sep 9, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Non-numeric age sentinels ("unknown", "prefer not to say") are silently folded into missing_pct instead of kept as a categorical group

2 participants