Skip to content

[None][fix] Isolate parallel unittest retry state - #21162

Draft
bvolpato wants to merge 2 commits into
NVIDIA:mainfrom
bvolpato:bvolpato/bound-unittest-retry
Draft

bvolpato wants to merge 2 commits into
NVIDIA:mainfrom
bvolpato:bvolpato/bound-unittest-retry

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 28, 2026 •

Copy link
Copy Markdown

Description

Motivation. A parallel unittest process can exit unsuccessfully after its tests pass. With an empty last-failed cache, the wrapper's --lf retry executes the entire passing workload again. A successful rerun can then hide the process failure. This wastes test execution and makes the CI result unreliable.

Change. Give each parallel attempt and its serial retry a private pytest cache, and set --lfnf=none. Pytest teardown removes the cache.

Verified before/after. The reproduction injects a nonzero controller exit after a passing run with real pytest-xdist workers:

Scenario Before After
Process fails without a recorded failed test Passing workload runs again; wrapper can return success Passing workload is not rerun; wrapper remains failed
A test fails once and passes on retry Retry recovers Targeted retry still recovers

The regressions demonstrate failure preservation and avoided re-execution. Saved CI minutes and changes in failure frequency have not been measured. The original process failure still requires diagnosis.

Test Coverage

  • Extended test_unittest_culprits.py with actual pytest-xdist worker processes and actual sequential last-failed selection. Empty and stale caches preserve process failures, while a test that fails once can pass its targeted retry.
  • The regressions fail against the original wrapper and pass with the change on Python 3.12 with pytest 8.4.2 and 9.0.3.
  • The helper module is marked cpu_only and is covered by the existing unittest/tools CPU CI entry.
  • Scoped pre-commit and git diff --check pass. Jenkins and GPU workloads were not run locally.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: bvolpato <brunocvcunha@gmail.com>
Signed-off-by: bvolpato <brunocvcunha@gmail.com>
@svc-trtllm-gh-bot svc-trtllm-gh-bot added the Community want to contribute PRs initiated from Community label Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Community want to contribute PRs initiated from Community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants