fix: non-numeric age sentinels get their own categorical group - #509
Merged
yakew7 merged 1 commit intoSep 9, 2026
Merged
Conversation
_age_to_numeric() returned None for both a genuinely missing cell and
free text with no embedded number at all (e.g. "unknown", "prefer not
to say") - the age-banding loop then counted every None straight into
null_count, indistinguishable from real missing data. Contradicted
SPEC.md section 2's own documented "anything else: treat as
categorical" rule for exactly this case.
$ python3 -c "
import pandas as pd
from faircode.profiler import profile
df = pd.DataFrame({'age': [25,30,45,'unknown','unknown','unknown','prefer not to say',22,33,41]})
d = profile(df)['dimensions'][0]
print(d['n_groups'], d['missing_pct'])
"
3 0.4 # (before) - 4 real responses vanish into missing_pct
5 0.0 # (after) - 'unknown' (3) and 'prefer not to say' (1) get their own groups
New _is_categorical_age_sentinel() (faircode/profiler.py) /
isCategoricalAgeSentinel() (assets/profiler-engine.js) distinguish three
cases per unparseable value: None/NaN (missing), a numeric value out of
the valid age range like a -1/999 sentinel (also missing - matches this
profiler's existing, tested behavior for range-invalid numeric
sentinels), and free text with no embedded number at all (categorical).
Only the third case gets its own group.
First attempt at this used pd.isna()/typeof-only-numeric as the
distinguishing check and broke
test_negative_age_sentinels_are_missing_instead_of_an_elderly_group - a
-1 sentinel isn't NaN, so it was wrongly routed to a new '-1' categorical
group instead of staying missing. Caught by running the full profiler
suite before committing; fixed by checking for an embedded number via
the same regex _age_to_numeric()/ageToNumeric() already use, not just
NaN-ness.
Verified Python/JS parity directly (not just via the test suite): built
a CSV from the issue's exact repro and ran both
faircode.profiler.profile() and 'node scripts/engine-js.js profile'
against it - identical n_groups, missing_pct, and group
labels/counts on both engines.
Added two regression tests to tests/test_profiler.py: the issue's exact
repro, and a mixed case (real None/NaN alongside a text sentinel)
confirming missing_pct only counts the genuine blanks. Full suite:
tests/test_profiler.py 38/38, tests/test_js_parity.py 21 passed + 1
skipped, full suite 401 passed/1 skipped/4 failed (the 4 are
tests/test_xlsx_edge_cases.py, confirmed pre-existing and unrelated -
SheetJS CDN blocked in this sandbox).
Known follow-up, out of scope here: _intersections()'s labelize()
helper (used by compare(), not profile()) still routes these same
sentinel values through ageBand()/None the old way for cross-tabulation
purposes. The issue's own repro and acceptance criteria are profile()-
only; left as-is rather than silently expanding scope.
Fixes yakew7#487
Contributor
|
@propcgamer20-png is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_age_to_numeric() returned None for both a genuinely missing cell and free text with no embedded number at all (e.g. "unknown", "prefer not to say") - the age-banding loop then counted every None straight into null_count, indistinguishable from real missing data. Contradicted SPEC.md section 2's own documented "anything else: treat as categorical" rule for exactly this case.
$ python3 -c "
import pandas as pd
from faircode.profiler import profile
df = pd.DataFrame({'age': [25,30,45,'unknown','unknown','unknown','prefer not to say',22,33,41]})
d = profile(df)['dimensions'][0]
print(d['n_groups'], d['missing_pct'])
"
3 0.4 # (before) - 4 real responses vanish into missing_pct
5 0.0 # (after) - 'unknown' (3) and 'prefer not to say' (1) get their own groups
New _is_categorical_age_sentinel() (faircode/profiler.py) / isCategoricalAgeSentinel() (assets/profiler-engine.js) distinguish three cases per unparseable value: None/NaN (missing), a numeric value out of the valid age range like a -1/999 sentinel (also missing - matches this profiler's existing, tested behavior for range-invalid numeric sentinels), and free text with no embedded number at all (categorical). Only the third case gets its own group.
First attempt at this used pd.isna()/typeof-only-numeric as the distinguishing check and broke
test_negative_age_sentinels_are_missing_instead_of_an_elderly_group - a -1 sentinel isn't NaN, so it was wrongly routed to a new '-1' categorical group instead of staying missing. Caught by running the full profiler suite before committing; fixed by checking for an embedded number via the same regex _age_to_numeric()/ageToNumeric() already use, not just NaN-ness.
Verified Python/JS parity directly (not just via the test suite): built a CSV from the issue's exact repro and ran both
faircode.profiler.profile() and 'node scripts/engine-js.js profile' against it - identical n_groups, missing_pct, and group labels/counts on both engines.
Added two regression tests to tests/test_profiler.py: the issue's exact repro, and a mixed case (real None/NaN alongside a text sentinel) confirming missing_pct only counts the genuine blanks. Full suite: tests/test_profiler.py 38/38, tests/test_js_parity.py 21 passed + 1 skipped, full suite 401 passed/1 skipped/4 failed (the 4 are tests/test_xlsx_edge_cases.py, confirmed pre-existing and unrelated - SheetJS CDN blocked in this sandbox).
Known follow-up, out of scope here: _intersections()'s labelize() helper (used by compare(), not profile()) still routes these same sentinel values through ageBand()/None the old way for cross-tabulation purposes. The issue's own repro and acceptance criteria are profile()- only; left as-is rather than silently expanding scope.
Fixes #487
Summary
Type
Audit checklist
unfair.pyincludes protected attributes and prints the required output formatfair.pyremoves protected attributes and identified proxy variablesrandom_state=42and an 80/20 train/test splitunfair.pngandfair.pngare included as PNG screenshotsDATA.mdis included if the file is too largeREADME.mdincludes the new results row and audit sectionBefore fairness gap:
After fairness gap:
Reduction:
Protected attribute(s):
Proxy variables dropped:
Explainer checklist
explainers/and uses lowercase hyphenated namingREADME.mdwas updatedLinked issue
Closes #