Skip to content

fix: Report each failed attempt once and tag the final attempt - #46

Merged
joshuarli merged 4 commits into
mainfrom
fix/always-report-reruns
Oct 20, 2026
Merged

joshuarli merged 4 commits into
mainfrom
fix/always-report-reruns

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

With PYTEST_SENTRY_ALWAYS_REPORT=1 and pytest-rerunfailures, pytest_runtest_makereport re-captures the whole accumulated exception chain on every attempt. A test that fails N times therefore sends 1 + 2 + … + N events, and a flaky test that passes on retry re-sends all of its earlier failures, tagged pytest.result:passed. Events also can't tell "failed, will be retried" apart from "failed every attempt", so alerts on pytest.result:failed fire on flaky retries even when CI is green.

This came up in getsentry/sentry's master test alerting. There, one hard-failing test produced ~20 events per CI run, and the alert went critical on most weekday afternoons while master was green.

Changes:

  • In always-report mode, each attempt captures only its own exception. Earlier attempts were already sent.
  • The default mode is unchanged: flaky tests are still reported once, when they pass after failing.
  • Call-phase events are tagged with pytest.attempt (rerunfailures' execution_count, 1-based) and pytest.final_attempt. pytest.final_attempt is true when rerunfailures won't retry the test again, or always when rerunfailures isn't installed or no reruns are configured. With these tags, pytest.result:failed pytest.final_attempt:true is a clean "this test actually failed" signal.

getsentry/sentry carries the same behavior as a local plugin for now, because it's still on pytest-sentry 0.3.0 with sentry-sdk 2.x (getsentry/sentry#128233).

Tooling port (second commit)

  • uv: pyproject.toml (uv_build backend) + committed uv.lock replace setup.py, setup.cfg, dev-requirements.txt, .flake8 and Makefile.
  • Python >=3.13 only (breaking); CI tests 3.13 and 3.14.
  • Latest deps: pytest 9.1.1, sentry-sdk 3.0.0a7, wrapt 2.5.1, pytest-rerunfailures 16.7, Ruff 0.17.1, ty 0.0.90.
  • Ruff select = ["ALL"] + preview (only formatter-conflicting / mutually exclusive rules ignored); ty with every rule as an error and error-on-warning. All findings fixed, no blanket suppressions.
  • CI: new lint job (ruff check, ruff format --check, ty check), uv-based test matrix, uv build for dist (artifact name unchanged for craft), actions pinned to SHAs. bump-version.sh now updates pyproject.toml and uv.lock.

Why every test job was failing

Not this change: sentry-sdk 3.0.0a7 crashes at import with OpenTelemetry >=1.44 (contextvars_context.py calls Context.pop(), and Context became immutable in 1.44). The lockfile constrains opentelemetry-api/sdk<1.44 via [tool.uv] constraint-dependencies (dev/CI only, not in package metadata). Downstream users on OTel >=1.44 still hit it until sentry-sdk is fixed.

via @joshuarli.

--

View Junior Session [Sentry]

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit bd5114d. Configure here.

Comment thread pytest_sentry/hooks.py Outdated
- Replace setup.py/setup.cfg/dev-requirements/flake8/Makefile with a
  uv-managed pyproject.toml (uv_build backend) and a committed uv.lock.
- Require Python >=3.13; CI tests 3.13 and 3.14.
- Upgrade dependencies to latest: pytest 9.1.1, sentry-sdk 3.0.0a7,
  wrapt 2.5.1, pytest-rerunfailures 16.7, ruff 0.17.1, ty 0.0.90.
- Pin OpenTelemetry <1.44 for dev/CI: sentry-sdk 3.0.0a7 crashes at import
  with newer OpenTelemetry (Context.pop on an immutable Context), which is
  what broke every test job on this PR.
- Ruff with select=ALL + preview, ty with all rules as errors; fix all
  findings (types, docstrings, imports) without blanket suppressions.
- CI: lint/format/type-check job, uv-based test matrix, uv build for dist
  (artifact name unchanged for craft); pin actions to SHAs.
- bump-version.sh updates pyproject.toml and uv.lock.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Comment thread scripts/bump-version.sh
…ision

Comparing execution_count to the rerun count missed attempts that
pytest-rerunfailures does not retry for other reasons (--only-rerun,
--rerun-except, a false condition, --max-suite-reruns), tagging hard
failures as final_attempt:false.

Collect the attempt's exceptions in pytest_runtest_makereport and capture
them in pytest_runtest_logfinish, after rerunfailures has marked a retried
attempt's reports as 'rerun'. An attempt with no 'rerun' report is final.
@joshuarli
joshuarli merged commit e2699d9 into main Oct 20, 2026
15 checks passed
@joshuarli
joshuarli deleted the fix/always-report-reruns branch October 20, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant