fix: match profiler-engine.js NA tokens to pandas' STR_NA_VALUES exactly - #534
Merged
yakew7 merged 1 commit intoSep 9, 2026
Merged
Conversation
Contributor
|
@propcgamer20-png is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel. A member of the Team first needs to authorize it. |
NA_TOKENS/isMissing() disagreed with pandas' real default na_values on two
counts, both backwards:
- it included bare lowercase "na", which pandas does NOT treat as missing
- it excluded "None", which pandas DOES treat as missing
and isMissing() lower-cased the cell before comparing, erasing pandas' own
case-sensitivity ("NA" is missing, "na" is not).
So a categorical column with literal "na"/"None" cells parsed differently
in the two engines, breaking the bit-for-bit parity SPEC.md promises.
NA_TOKENS now mirrors pandas.io.parsers.readers.STR_NA_VALUES verbatim and
the comparison is case-sensitive.
Closes yakew7#491
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
propcgamer20-png
force-pushed
the
fix/js-na-tokens-match-pandas
branch
from
September 9, 2026 15:28
8a2f3b5 to
0c1ec9f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
assets/profiler-engine.js'sNA_TOKENS/isMissing()disagreed with pandas' actual defaultna_values(pandas.io.parsers.readers.STR_NA_VALUES), whichfaircode/loaders.pygets by calling plainpd.read_csv(). Two things were backwards:NA_TOKENSincluded bare lowercase"na"- pandas does not treat that as missingNA_TOKENSexcluded"None"- pandas does treat that as missingand
isMissing()didString(v).trim().toLowerCase()before the lookup, erasing pandas' case-sensitivity ("NA"is missing,"na"is not).Repro (before)
A
statuscolumn with 2 literalnarows and 2 literalNonerows out of 10:Both report
0.2for opposite reasons - parity is broken for any dataset using literal"na"or"None"text.Fix
NA_TOKENSnow mirrorsSTR_NA_VALUESverbatim ('','#N/A','#N/A N/A','#NA','-1.#IND','-1.#QNAN','-NaN','-nan','1.#IND','1.#QNAN','<NA>','N/A','NA','NULL','NaN','None','n/a','nan','null') andisMissing()compares case-sensitively (drops.toLowerCase()).After
Test
test_python_js_na_token_parity_on_literal_na_and_none- asserts byte-identical Python/JS structured output for a column with literalna/Nonecells, plus that"na"survives as a group and"None"counts as missing.pytest tests/test_js_parity.py-> 22 passed, 1 failed. The failure (test_python_js_profiler_parity_sniffs_quoted_newlines) is pre-existing onmainon this platform and unrelated - the 5 real-audit-dataset parity cases all still pass.Closes #491
🤖 Generated with Claude Code