Skip to content

fix: reject out-of-range profiler tunables instead of self-contradicting output - #533

Merged
yakew7 merged 1 commit into
yakew7:mainfrom
propcgamer20-png:fix/validate-out-of-range-tunables
Sep 9, 2026
Merged

fix: reject out-of-range profiler tunables instead of self-contradicting output#533
yakew7 merged 1 commit into
yakew7:mainfrom
propcgamer20-png:fix/validate-out-of-range-tunables

Conversation

@propcgamer20-png

Copy link
Copy Markdown
Collaborator

Problem

min_share / intersection_floor / imbalance_flag / missing_flag are accepted as plain floats with no range validation anywhere in the call chain (cli.py argparse -> mcp_server._build_opts -> profiler._resolve_opts).

r = _profile_dataset_impl("mini.csv", min_share=1.5, include_provenance=False)
r["overall_score"], r["grade"]   # -> 100 A
r["flags"]  # -> ["sex: 'F' is under-represented (50.0%)", "sex: 'M' is under-represented (50.0%)"]

min_share=1.5 can't be a share, yet every group is flagged under-represented while the score stays a perfect "A" - the two halves of the same report directly contradict each other. A 15-instead-of-0.15 typo produces exactly this.

Fix

_resolve_opts (Python) and resolveOpts (JS, for parity) now validate on every profile() call:

tunable rule
min_share, intersection_floor, missing_flag, reference_flag in [0, 1]
imbalance_flag >= 1 (it's a max_share / min_share ratio)
min_group_size >= 1

Out-of-range values raise ValueError. The CLI already renders that as error: ... (exit 2); the MCP build_server() wrappers already translate it to ToolError. So profile_dataset, compare_datasets, faircode profile, and faircode compare are all covered through the one chokepoint. Defaults are unchanged and still valid.

$ faircode profile mini.csv --min-share 1.5
error: min_share must be between 0 and 1, got 1.5      # exit 2
$ faircode profile mini.csv --imbalance-flag 0.5
error: imbalance_flag must be >= 1, got 0.5

Tests

  • test_out_of_range_tunables_raise_instead_of_contradicting_themselves (parametrized, test_profiler.py)
  • test_profile_rejects_out_of_range_min_share (test_cli.py, argparse/exit-code path)
  • test_python_js_reject_out_of_range_min_share_parity (test_js_parity.py, both engines reject)

pytest on the touched suites -> 186 passed, 2 failed. Both failures (test_get_explainer_returns_content_and_metadata, test_python_js_profiler_parity_sniffs_quoted_newlines) are pre-existing on main on this platform and unrelated. ruff clean.

Closes #511

🤖 Generated with Claude Code

min_share / intersection_floor / missing_flag / reference_flag were
accepted as any float, and imbalance_flag / min_group_size as any number,
with no range check anywhere in the CLI -> _build_opts -> _resolve_opts
chain. min_share=1.5 (a percentage/fraction typo) was silently accepted:
every group got flagged "under-represented" in flags while
overall_score/grade stayed 100/"A" - the two halves of the same report
contradicting each other with no error.

_resolve_opts (and its JS mirror resolveOpts) now validate:
  - min_share, intersection_floor, missing_flag, reference_flag in [0, 1]
  - imbalance_flag >= 1
  - min_group_size >= 1
raising ValueError, which the CLI already renders as "error: ..." (exit 2)
and the MCP server wraps as a ToolError. Covers profile_dataset,
compare_datasets, `faircode profile`, and `faircode compare`.

Closes yakew7#511

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@propcgamer20-png is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

@yakew7 @ahmdkaml - new PR touching a path you own, please review.

@yakew7
yakew7 merged commit 4a42779 into yakew7:main Sep 9, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP profile_dataset/compare_datasets accept out-of-range min_share/threshold params, producing self-contradictory output

2 participants