[data] fix: Reject truncated prompt-completion rows without supervision

[yaoyu-33]

What changed

Prompt/completion preprocessing now rejects a non-empty completion when truncation and requested special tokens leave no supervised next-token target.

User impact

A supported completion-only configuration such as add_bos=True, add_sep=True, add_eos=False, and seq_length=2 reserves the entire sequence for BOS and SEP. The shared tokenizer previously accepted the ordinary non-empty row after truncating both text segments away. Direct-HF then emitted only ignored labels, while GPT SFT emitted an all-zero loss mask, so training silently counted the sample while receiving no language-model signal.

Root cause and fix

tokenize_prompt_completion_example() checked that the final sequence had at least two tokens, but did not check the post-shift supervision invariant. The fix adds a local guard after skipped-token masking: completion-only rows with a non-empty completion must retain at least one supervised token after the next-token shift. Full-loss behavior, intentionally empty completions, and sampler-generated padding rows are unchanged.

Validation

The unchanged regression test failed on origin/main at 4a3f2c2ed162cafd027e941712d2022d5309ddbe:

uv run python -m pytest \
  tests/unit_tests/data/test_sft_processing.py::test_prompt_completion_rejects_truncation_without_supervision \
  -q --tb=short

FAILED: DID NOT RAISE ValueError

After the fix:

uv run python -m pytest \
  tests/unit_tests/data/test_sft_processing.py \
  tests/unit_tests/data/collators/test_sft.py \
  -q --tb=short

35 passed

Additional checks:

git diff --check
uv run pre-commit run --all-files

Both passed.

Scope

This does not change public APIs, full-loss semantics, chat preprocessing, packing policy, or supported model/backend scope.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yaoyu-33]

/ok to test 9d8152f

[claude]

Light review - LGTM

The added guard in tokenize_prompt_completion_example is a correct fail-closed improvement: for loss_mode == "completion" with a non-empty completion, it now raises if no supervised token survives instead of silently producing a zero-loss training row. The loss_mask[1:] slice is safe because the earlier input_ids.numel() < 2 check guarantees at least two tokens, so the slice is never empty. Verified the new test's arithmetic reaches the raise (max_length=2, add_bos plus add_sep gives body_budget=0, completion truncated to empty).

Note (non-blocking): this is a behavior change. Dataset rows that previously produced a silent zero-supervision sample will now error during preprocessing. That aligns with the explicit-error-over-silent-fallback principle and looks intentional.

Minor test gap: the guard also fires when skipped_tokens masks out every supervised target token, but only the truncation path is covered.

Suggested test cases

• No perf tests impacted (only data/sft_processing.py and its unit test changed).
• Add unit case: completion whose tokens are all in skipped_tokens, expect the new ValueError.
• Add unit case: loss_mode=completion with an empty completion, guard is skipped, no ValueError raised.
• Add unit case: loss_mode=full with the same truncation setup, guard does not apply, no ValueError raised.