fix: Report each failed attempt once and tag the final attempt (#46)

* fix: Report each failed attempt once and tag the final attempt
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>

* build: Port to uv, Ruff and ty; require Python 3.13+

- Replace setup.py/setup.cfg/dev-requirements/flake8/Makefile with a
  uv-managed pyproject.toml (uv_build backend) and a committed uv.lock.
- Require Python >=3.13; CI tests 3.13 and 3.14.
- Upgrade dependencies to latest: pytest 9.1.1, sentry-sdk 3.0.0a7,
  wrapt 2.5.1, pytest-rerunfailures 16.7, ruff 0.17.1, ty 0.0.90.
- Pin OpenTelemetry <1.44 for dev/CI: sentry-sdk 3.0.0a7 crashes at import
  with newer OpenTelemetry (Context.pop on an immutable Context), which is
  what broke every test job on this PR.
- Ruff with select=ALL + preview, ty with all rules as errors; fix all
  findings (types, docstrings, imports) without blanket suppressions.
- CI: lint/format/type-check job, uv-based test matrix, uv build for dist
  (artifact name unchanged for craft); pin actions to SHAs.
- bump-version.sh updates pyproject.toml and uv.lock.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>

* fix: Decide pytest.final_attempt from rerunfailures' actual retry decision

Comparing execution_count to the rerun count missed attempts that
pytest-rerunfailures does not retry for other reasons (--only-rerun,
--rerun-except, a false condition, --max-suite-reruns), tagging hard
failures as final_attempt:false.

Collect the attempt's exceptions in pytest_runtest_makereport and capture
them in pytest_runtest_logfinish, after rerunfailures has marked a retried
attempt's reports as 'rerun'. An attempt with no 'rerun' report is final.

* fix(release): Reject versions that could inject sed commands in bump-version.sh

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>

Author: sentry-junior[bot]

.flake8

@@ -3,30 +3,47 @@ on:
   push:
     branches: [main, release/**, test-me-*]
 
+permissions:
+  contents: read
+
 jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      with:
+        persist-credentials: false
+    - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+    - run: uv sync --locked
+    - run: uv run --locked ruff check --output-format=github .
+    - run: uv run --locked ruff format --check .
+    - run: uv run --locked ty check --output-format=github
   test:
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        python: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
+        python: ['3.13', '3.14']
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-python@v4
+    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      with:
+        persist-credentials: false
+    - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
       with:
         python-version: ${{ matrix.python }}
-    - run: pip install -r dev-requirements.txt
-    - run: PYTEST_SENTRY_ALWAYS_REPORT=1 pytest tests
+    - run: uv sync --locked
+    - run: uv run --locked pytest tests
+      env:
+        PYTEST_SENTRY_ALWAYS_REPORT: '1'
   dist:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-python@v4
+    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
       with:
-        python-version: '3.x'
-    - run: |
-        pip install build
-        python -m build
-    - uses: actions/upload-artifact@v4
+        persist-credentials: false
+    - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+    - run: uv build
+    - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
       with:
         name: ${{ github.sha }}
         path: dist/*

.github/workflows/main.yml

@@ -21,7 +21,7 @@ jobs:
         with:
           app-id: ${{ vars.SENTRY_RELEASE_BOT_CLIENT_ID }}
           private-key: ${{ secrets.SENTRY_RELEASE_BOT_PRIVATE_KEY }}
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           token: ${{ steps.token.outputs.token }}
           fetch-depth: 0

.github/workflows/release.yml

@@ -20,3 +20,4 @@ venv
 .pytest_cache
 .hypothesis
 semaphore
+.ruff_cache

.gitignore

@@ -15,6 +15,7 @@ This is what a test run will look like in Sentry.
 ## Quickstart
 ### Prerequisites
 
+* Python 3.13 or newer.
 * You are using [pytest](https://pytest.org) to run your tests.
 * You are using [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) to rerun flaky tests.
 
@@ -70,6 +71,12 @@ Now every test failure that is "fixed" by retrying the test is reported to
 Sentry, but still does not break CI. Tests that consistently fail will not be
 reported.
 
+With `PYTEST_SENTRY_ALWAYS_REPORT=1`, every failed attempt is reported instead,
+each one exactly once. Error events are tagged with `pytest.attempt` (1-based)
+and `pytest.final_attempt` (`true` once `pytest-rerunfailures` will not retry
+the test again), so you can tell flaky retries apart from tests that failed
+every attempt, e.g. alert on `pytest.result:failed pytest.final_attempt:true`.
+
 # Tracking the performance of your testsuite
 
 By default `pytest-sentry` will send [Performance](https://sentry.io/for/performance/) data to Sentry:
@@ -157,9 +164,9 @@ the isolation `Scope` being used to send events to Sentry. Use `with use_isolati
 to temporarily switch context. You can use this to set custom tags like so::
 
 ```python
-  def test_foo(sentry_test_scope):
-      with use_isolation_scope(sentry_test_scope):
-          sentry_sdk.set_tag("pull_request", os.environ['EXAMPLE_CI_PULL_REQUEST'])
+def test_foo(sentry_test_scope):
+    with use_isolation_scope(sentry_test_scope):
+        sentry_sdk.set_tag("pull_request", os.environ["EXAMPLE_CI_PULL_REQUEST"])
 ```
 
 Why all the hassle with the context manager? Just imagine if your tested
@@ -175,6 +182,20 @@ This can be enabled for builds on the `main` or release branch, to catch
 certain kinds of tests that are flaky across builds, but consistently fail or
 pass within one testrun.
 
+# Development
+
+The project is managed with [uv](https://docs.astral.sh/uv/), linted and
+formatted with [Ruff](https://docs.astral.sh/ruff/) and type-checked with
+[ty](https://docs.astral.sh/ty/):
+
+```bash
+uv sync
+uv run ruff check .
+uv run ruff format --check .
+uv run ty check
+PYTEST_SENTRY_ALWAYS_REPORT=1 uv run pytest tests
+```
+
 # License
 
 Licensed under 2-clause BSD, see [LICENSE](LICENSE).

Makefile

@@ -0,0 +1,85 @@
+[project]
+name = "pytest-sentry"
+version = "0.5.1"
+description = "A pytest plugin to send testrun information to Sentry.io"
+readme = "README.md"
+license = "BSD-2-Clause"
+license-files = ["LICENSE"]
+authors = [{ name = "Markus Unterwaditzer", email = "markus@unterwaditzer.net" }]
+requires-python = ">=3.13"
+classifiers = [
+    "Framework :: Pytest",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+]
+dependencies = [
+    "pytest>=9.1.1",
+    "sentry-sdk>=3.0.0a7",
+    "wrapt>=2.5.1",
+]
+
+[project.urls]
+Repository = "https://github.com/getsentry/pytest-sentry"
+
+[project.entry-points.pytest11]
+sentry = "pytest_sentry"
+
+[dependency-groups]
+dev = [
+    "pytest-rerunfailures>=16.7",
+    "ruff>=0.17.1",
+    "ty>=0.0.90",
+]
+
+[build-system]
+requires = ["uv_build>=0.13.4,<0.14"]
+build-backend = "uv_build"
+
+[tool.uv]
+# sentry-sdk 3.0.0a7 crashes on import with opentelemetry >= 1.44, whose
+# Context became immutable (sentry_sdk/opentelemetry/contextvars_context.py
+# calls Context.pop). Dev/CI-only constraint; not part of the package metadata.
+constraint-dependencies = ["opentelemetry-api<1.44", "opentelemetry-sdk<1.44"]
+
+[tool.uv.build-backend]
+module-root = ""
+
+[tool.ruff]
+target-version = "py313"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["ALL"]
+preview = true
+explicit-preview-rules = false
+ignore = [
+    # Conflict with the formatter: https://docs.astral.sh/ruff/formatter/#conflicting-lint-rules
+    "missing-trailing-comma",
+    "single-line-implicit-string-concatenation",
+    # Mutually exclusive with D211 / D212, which are enforced instead.
+    "incorrect-blank-line-before-class",
+    "multi-line-summary-second-line",
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = [
+    "assert",    # pytest uses bare asserts
+    "magic-value-comparison", # magic values in assertions
+]
+
+[tool.ruff.format]
+preview = true
+
+[tool.ty.environment]
+python-version = "3.13"
+
+[tool.ty.rules]
+all = "error"
+
+[tool.ty.terminal]
+error-on-warning = true
+
+[tool.pytest.ini_options]
+addopts = ["-p", "no:cacheprovider", "-p", "pytester"]

README.md

@@ -1,9 +1,26 @@
-from .client import Client  # noqa: F401
-from .fixtures import sentry_test_scope  # noqa: F401
-from .hooks import (  # noqa: F401
+# Copyright (c) 2018 Markus Unterwaditzer and individual contributors.
+"""A pytest plugin that reports test runs to Sentry."""
+
+from pytest_sentry.client import Client
+from pytest_sentry.fixtures import sentry_test_scope
+from pytest_sentry.hooks import (
     pytest_fixture_setup,
     pytest_load_initial_conftests,
     pytest_runtest_call,
+    pytest_runtest_logfinish,
     pytest_runtest_makereport,
     pytest_runtest_protocol,
 )
+from pytest_sentry.integration import PytestIntegration
+
+__all__ = [
+    "Client",
+    "PytestIntegration",
+    "pytest_fixture_setup",
+    "pytest_load_initial_conftests",
+    "pytest_runtest_call",
+    "pytest_runtest_logfinish",
+    "pytest_runtest_makereport",
+    "pytest_runtest_protocol",
+    "sentry_test_scope",
+]

dev-requirements.txt

@@ -1,42 +1,38 @@
+# Copyright (c) 2018 Markus Unterwaditzer and individual contributors.
+"""The Sentry client used to report tests."""
+
 import os
+from typing import Any, cast
+
 import sentry_sdk
 
-from .integration import PytestIntegration
+from pytest_sentry.integration import PytestIntegration
+
+_TRUTHY = frozenset({"1", "true", "yes"})
 
 
 class Client(sentry_sdk.Client):
-    """
-    A client that is used to report errors from tests.
-    Configured via `PYTEST_SENTRY_*` environment variables.
+    """A client that reports errors from tests.
+
+    Configured via `PYTEST_SENTRY_*` environment variables. Explicitly passed
+    options take precedence over the environment.
     """
 
-    def __init__(self, *args, **kwargs):
-        kwargs.setdefault(
-            "dsn",
-            os.environ.get("PYTEST_SENTRY_DSN", None),
-        )
-        kwargs.setdefault(
-            "traces_sample_rate",
-            float(os.environ.get("PYTEST_SENTRY_TRACES_SAMPLE_RATE", 1.0)),
-        )
-        kwargs.setdefault(
-            "profiles_sample_rate",
-            float(os.environ.get("PYTEST_SENTRY_PROFILES_SAMPLE_RATE", 0.0)),
-        )
-        kwargs.setdefault(
-            "_experiments",
-            {},
-        ).setdefault(
-            "auto_enabling_integrations",
-            True,
-        )
-        kwargs.setdefault(
-            "environment",
-            os.environ.get("SENTRY_ENVIRONMENT", "test"),
-        )
-        kwargs.setdefault("integrations", []).append(PytestIntegration())
-
-        debug = os.environ.get("PYTEST_SENTRY_DEBUG", "").lower() in ("1", "true", "yes")
-        kwargs.setdefault("debug", debug)
-
-        sentry_sdk.Client.__init__(self, *args, **kwargs)
+    def __init__(self, dsn: str | None = None, **options: object) -> None:
+        """Create a client, filling in options from the environment."""
+        integrations = options.pop("integrations", None)
+        merged: dict[str, object] = {
+            "dsn": dsn if dsn is not None else os.environ.get("PYTEST_SENTRY_DSN"),
+            "traces_sample_rate": float(os.environ.get("PYTEST_SENTRY_TRACES_SAMPLE_RATE", "1.0")),
+            "profiles_sample_rate": float(
+                os.environ.get("PYTEST_SENTRY_PROFILES_SAMPLE_RATE", "0.0")
+            ),
+            "environment": os.environ.get("SENTRY_ENVIRONMENT", "test"),
+            "debug": os.environ.get("PYTEST_SENTRY_DEBUG", "").lower() in _TRUTHY,
+        } | options
+        merged["integrations"] = [
+            *(integrations if isinstance(integrations, list) else []),
+            PytestIntegration(),
+        ]
+        # Options come from users and are forwarded as-is; the SDK validates them.
+        sentry_sdk.Client.__init__(self, **cast("dict[str, Any]", merged))