[udsy19]

What

HistogramFilter.score_document counts in-histogram characters over text.strip() but divides by len(text) — the un-stripped length:

cnt = len([c for c in text.strip() if c in self._histogram])
return 1 if cnt / len(text) > self._threshold else 0

Numerator and denominator range over different character sets. Leading/trailing whitespace inflates the denominator, lowering the ratio and flipping an otherwise-kept document to rejected. Whitespace-only/empty input additionally raises ZeroDivisionError.

With histogram {a, b, c} and threshold 0.8, "abc" scores 1.0 (kept) while " abc " scores 0.43 (dropped) — identical content, opposite verdicts. Legitimate text is silently discarded from curated datasets.

Fix

Compute the ratio over the same stripped text used for the numerator, and return 0 for empty/whitespace-only input:

stripped = text.strip()
if not stripped:
    return 0
cnt = len([c for c in stripped if c in self._histogram])
return 1 if cnt / len(stripped) > self._threshold else 0

Tests

Adds tests/stages/text/filters/histogram/test_histogram_filter.py covering whitespace-invariance, the empty/whitespace-only guard, and a below-threshold rejection. Verified fail-on-revert: with the source reverted the whitespace document flips to rejected and empty input raises ZeroDivisionError; with the fix all pass.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

HistogramFilter.score_document now counts characters and computes their ratio over the same stripped text.

• Histogram filtering ignores whitespace around text content.

_{Reviews (2) · Last reviewed commit: "Merge branch 'main' into fix/histogram-f..." · Reviewed by Greptile}

[sarahyurick]

/ok to test a5df36a