Skip to content

build(deps): Bump pytest-xdist to 3.8.0, drop pytest-workaround-12888 - #127938

Merged
joshuarli merged 2 commits into
masterfrom
joshua/build/xdist-bump-remove-pytest-workaround
Oct 17, 2026
Merged

joshuarli merged 2 commits into
masterfrom
joshua/build/xdist-bump-remove-pytest-workaround

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Bumps pytest-xdist 3.0.2 → 3.8.0 (latest), and with it execnet 1.9.0 → 2.1.2. Also removes the pytest-workaround-12888 dev dependency and keeps xdist's --dist=loadfile ordering exactly as it was.

Bump is done. getsentry/pypi#2626 mirrored pytest-xdist==3.8.0 and execnet==2.1.2 to internal PyPI (pypi.devinfra.sentry.io). The bump comes from that index via uv@0.12.10 add --dev "pytest-xdist>=3.8.0", the same command tools/bump_version.py runs in the bump-version workflow. It's applied on this branch rather than through a separate workflow PR so it lands together with the ordering guard below. No public index or override was added, and the lock's wheel hashes match the upstream release artifacts.

Removing pytest-workaround-12888

The package was only a .pth file that ran import readline early, to work around pytest-dev/pytest#12888. pytest-dev/pytest#13176 fixed that upstream, and pytest 9.0.3 (our locked version) already calls _readline_workaround() from _pytest/capture.py in pytest_load_initial_conftests. Nothing else in the repo refers to it.

Keeping test ordering the same (--dist=loadfile)

Starting in xdist 3.5.0 (pytest-dev/pytest-xdist#632), LoadScopeScheduling sorts work units by test count, largest first, before handing them out. LoadFileScheduling subclasses it, so --dist=loadfile is affected too, and this is on by default. On its own, the bump would change the order files are handed to workers. That changes which files share a worker and the order they run in. It would also override the SENTRY_SHUFFLE_TESTS ordering.

xdist 3.8.0 (pytest-dev/pytest-xdist#1083) added --no-loadscope-reorder to turn this off. This PR sets the same thing (config.option.loadscopereorder = False) in sentry.testutils.pytest.sentry:pytest_configure, so the work queue keeps collection order exactly as it did on 3.0.2. getsentry loads the same plugin, so it's covered too. --dist=loadfile and how we pick worker counts are not changed.

I checked the rest of the scheduler diff between 3.0.2 and 3.8.0 (scheduler/loadscope.py, scheduler/loadfile.py). Apart from the reorder, the changes are typing and switching OrderedDict to plain dict. Insertion order is kept, and dropping extra nodes still pops the last node. File scoping in _split_scope is unchanged. testrunuid is still uuid4().hex per run, so the deterministic PYTEST_XDIST_TESTRUNUID-seeded cell name works the same.

xdist 3.0.2 → 3.8.0 changes that matter for our CI (-n 2/3 --dist=loadfile)

Things that stay the same:

  • The pytest_xdist_setupnodes hookspec.
  • PYTEST_XDIST_WORKER / PYTEST_XDIST_TESTRUNUID / PYTEST_XDIST_WORKER_COUNT, still set in xdist/remote.py.
  • The --looponfail/looponfailroots and --rsyncdir deprecation warnings: same text, so the existing filterwarnings entries still match.

Other changes in this range don't touch the code paths we use (--dist=loadfile, -n), and no scheduling options are adopted or proposed.

Verification (local, env synced from this branch's uv.lock via internal PyPI)

  • pytest -n 3 --dist=loadfile --reuse-db -q tests/sentry/api/bases/: 159 passed, with the workaround package uninstalled.
  • Ordering: with -n 1 --dist=loadfile on tests/sentry/api/bases/, files ran in collection order (test_organization 64 tests, test_organization_events 10, test_organizationmember 8, test_project 44, test_team 33), not sorted by size. A synthetic suite gave the same order on 3.0.2, on 3.8.0 + --no-loadscope-reorder, and on 3.8.0 + the plugin pin. Plain 3.8.0 reordered it.
  • getsentry tests/getsentry/test_features.py with -n 2 --dist=loadfile: 121 passed, with and without -W ignore::DeprecationWarning:xdist.plugin. Checked earlier against the same xdist/execnet versions.
  • prek run and mypy are clean.

via @joshuarli.

--

View Junior Session [Sentry]

pytest 9.0.3 already imports readline early in _pytest/capture.py
(_readline_workaround, pytest-dev/pytest#13176), so the .pth-based
workaround for pytest-dev/pytest#12888 is redundant.

pytest-xdist 3.5+ reorders --dist=loadfile work units by test count by
default. Opt out from the shared sentry pytest plugin so file order and
worker assignment stay identical after the bump (covers getsentry too).

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 17, 2026
Pulls in execnet 2.1.2 (xdist 3.6+ requires execnet>=2.1). Both are served by
the internal index after getsentry/pypi#2626.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
@sentry-junior sentry-junior Bot changed the title build(deps): Prep pytest-xdist 3.8.0 bump, drop pytest-workaround-12888 build(deps): Bump pytest-xdist to 3.8.0, drop pytest-workaround-12888 Oct 17, 2026
@joshuarli
joshuarli marked this pull request as ready for review October 17, 2026 16:37
@joshuarli
joshuarli requested review from a team as code owners October 17, 2026 16:37
@joshuarli
joshuarli merged commit ddbc338 into master Oct 17, 2026
91 checks passed
@joshuarli
joshuarli deleted the joshua/build/xdist-bump-remove-pytest-workaround branch October 17, 2026 16:38
joshuarli added a commit that referenced this pull request Oct 17, 2026
Follow-up to retro feedback on #127938, #127939, #127941, #127942 and
#127943. I checked each item against the code before changing anything.
Below, each one is marked done or declined. No test case or assertion
was dropped or weakened. Scheduling stays on `--dist=loadfile` with
deterministic order, and xdist 3.8.0 keeps `loadscopereorder = False`.

### Filtered sharded runs (first commit)

Since #127943, `pytest-selected.py` splits test files across shards
before pytest applies `-m`/`-k`. Snuba's `ci` job runs sentry master's
tests with `PYTEST_ADDOPTS="-k 'not __In' -m snuba_ci"` over 16 shards.
As a result, most of those shards collected 0 tests and exited with code
5, and fail-fast cancelled the rest. This breaks the sentry tests job on
every Snuba PR (e.g. getsentry/snuba#8556).

The first commit falls back to the pre-#127943 per-test hash sharding
whenever `-m` or `-k` appears in the args or in `PYTEST_ADDOPTS`. In
that case every shard collects everything, and the sentry plugin keeps
that shard's hash share after collection. Exit code 5 is still treated
as a failure. Sentry's own backend CI passes neither flag, so it keeps
per-file sharding.

I reproduced Snuba's collection locally against this checkout: same args
as `make test-python-ci`, the same `PYTEST_ADDOPTS` filter,
`--collect-only`, 16 shards. Unsharded, 78 tests are selected.

| | Shards with 0 tests (exit 5) | Tests per shard | Union of shards ==
unsharded (`Counter`) |
|---|---|---|---|
| #127943 | 8 of 16, including shards 0 and 12 | 0–23 | yes |
| this fix | 0 of 16 | 2–8 | yes |

New tests: `tests/sentry/testutils/pytest/test_sentry.py` runs
`pytest-selected.py` and the real plugins over a throwaway project with
`-m`, `-k`, and Snuba's `PYTEST_ADDOPTS` form. It checks that every
shard runs something and that the shards together run each selected test
exactly once. Without the fix these cases fail with exit code 5. There
are also unit tests for the flag detection. getsentry's CI doesn't pass
`-m`/`-k` to its `pytest-selected.py`.

### #127939: subtests in the JSON report (done)

The rerun adapter in `json_report_reruns.py` treated every repeated
`report.when` under a nodeid as a rerun. I reproduced this with the
locked stack (pytest 9, pytest-json-report 1.5.0, rerunfailures, xdist).
A test with 3 passing subtests that ran once got 3 `reruns` entries, and
its `setup` moved into one of them. When a subtest failed, the only
top-level traceback was "contains 1 failed subtest". pytest's exit code
was still correct.

- The adapter now detects `pytest.SubtestReport`. It keeps one parent
`setup`/`call`/`teardown` and stores subtests in a new `subtests` list,
each with its `context` (`msg`, repr'd `kwargs`), outcome, duration and
longrepr. rerunfailures holds back parent reports until an attempt ends,
so subtests are buffered and attached to the `call` they ran in. A
genuine rerun still goes to `reruns` and carries that attempt's subtests
along.
- Outcome: the parent stays `passed` when every subtest passes and
becomes `failed` when any subtest fails. I normalized away pytest's
"subtests passed" label because `report-backend-test-failures.js`
selects `outcome === 'failed'`.
- Failure comments now show each failed subtest's context and traceback,
up to 5 per test.
- `backend-test-durations.py extract` still sums only the parent phases.
The parent `call` already includes the subtest loop, so subtest
durations are not added again. Since setup no longer moves into
`reruns`, it is counted again.
- `tests/sentry/testutils/pytest/test_json_report_reruns.py` runs the
real plugin stack (json-report, this adapter, rerunfailures, xdist `-n
2`) in a subprocess. It checks that passing subtests make no reruns, a
failed subtest keeps its context and assertion traceback, parent setup
appears once, and a genuine rerun is recorded. Against the old adapter
it fails 6 of 8 cases.

### #127941: transactional flush race (done)

The race was real. The table inventory and the open-transaction check
were separate statements. A row committed into a table with no pages
between those two steps survived teardown.

- Each database is now cleaned in one transaction: `lock_timeout = 1s`,
then `LOCK TABLE <every managed table> IN EXCLUSIVE MODE` (this blocks
writes but allows reads), then inventory, fingerprint check and
`DELETE`. A lock timeout or deadlock falls back to Django's flush, which
waits for writers the way `TRUNCATE` always has. The open-transaction
check stays, both before the lock (so a known-open writer skips the 1s
wait) and under it. The lock blocks writers that have already written. A
transaction that has only read can still write after the lock is
released.
- Setting `SENTRY_LEGACY_TRANSACTIONAL_FLUSH=1` keeps Django's teardown.
`pytest_unconfigure` restores the original method and clears the
seeded-row cache. The cache is keyed by alias plus database `NAME`.
- Stable contenttype/permission IDs are unchanged (intentional).
- `tests/sentry/testutils/pytest/test_transactional_flush.py` drives
`_fixture_teardown` explicitly, so it doesn't depend on test order. It
covers:
- ORM and raw SQL rows across default/control/secondary, plus a large
TOAST value
- inserting, updating and deleting seeded contenttype/permission/site
rows, which forces the fallback and restores state
  - the `available_apps` and `serialized_rollback` fallbacks
- a second connection that tries to commit into an empty table during
cleanup: it blocks on the lock and no row survives
- a writer already open at teardown start, cleaned up through the
fallback
  - lock-timeout fallback, the escape hatch, and unconfigure

The race tests synchronize on `pg_blocking_pids` / `pg_stat_activity`,
not sleeps, and they fail with the lock disabled.
- Lock cost, measured locally against the real test schema (254 default,
70 control, 6 secondary tables). A warm teardown across all DBs went
from ~15–18 ms to ~23–27 ms. A cold connection went from ~85–92 ms to
~101–112 ms. That is roughly 10–20 ms per teardown against ~1.27 s for
Django's flush, so the lock keeps almost all of the saving. CI agrees
(see below).
- Audits:
- No non-internal `pg_trigger` rows exist in any test DB, so DELETE vs
TRUNCATE trigger behavior doesn't matter.
- The `post_migrate` receivers the skip bypasses are only Django's
`create_contenttypes`, `create_permissions` and `create_default_site`.
The fingerprint covers all of their rows.
- Dead tuples: across 117 transactional teardowns in one session, the
first-quarter median was 0.126 s and the last-quarter median 0.116 s.
Teardowns did not slow down over the session.
- Declined: making the plugin opt-in, separate toggles for DELETE vs the
`post_migrate` skip, and non-Postgres support. Sentry tests only run on
Postgres, and those backends already take Django's path.

### #127942: Relay readiness (done)

`wait_until_ready()` checks the time left before each probe. The request
timeout is `min(1s, remaining)`, sleeps are capped by the time left, and
no probe starts after the deadline. A 200 from a probe that started
before the deadline counts as ready even if it arrives late. It catches
`requests.RequestException` instead of `Exception`. The failure message
includes the last HTTP status or exception plus the container logs, and
the container is removed even if fetching the logs fails. Every test
still gets a fresh container. Unit tests with an injected clock, probe
and sleep cover: immediate ready, 503→200, connection error→success,
permanent 503, request timeout, near-deadline success, and bounded
sleep.

### #127939: subtest isolation (done / declined)

- `test_buffer.py`: each permutation now builds a fresh `SpansBuffer`,
which resets `any_shard_at_limit` and the cached properties. The options
and Redis setup stay shared. I re-checked with reverse permutation order
and an injected early failure: exactly the injected subtest failed and
the rest passed. The redis-cluster variants ran green in the CI run
below.
- `test_client_config.py`: declined. Each factory clears caches and
builds its own users and orgs, and the org/project lookups depend on the
request. I found no shared global query or cache, and reverse order
already passed all 758 subtests.
- Declined: documenting how to reproduce a single subtest. There's no
natural place for it, and subtests can't be selected by nodeid.

### #127943: sharding boundaries (done)

- Absolute roots, `..` roots, and `--rootdir` or `-c`/`--config-file`
(in the args or `PYTEST_ADDOPTS`) produce paths that don't match the
sentry plugin's rootdir-relative comparison. `pytest-selected.py` now
falls back to the legacy hash sharding for them. `pytest_shard_plugin`
raises a usage error if the rootdir isn't the working directory. CI
passes the relative root `tests` and keeps the fast path.
- Files outside `tests/` were not duplicated across shards.
`pytest_collection_modifyitems` already drops them by
`SELECTED_TESTS_FILE`; they are only imported. A new test covers this.
- `tests/sentry/testutils/pytest/test_sentry.py` runs the real
`pytest-selected.py`, sentry plugin and shard plugin over a throwaway
project with 3 shards. It asserts that the `Counter` of unsharded
nodeids equals the sum of the shard Counters. Cases: relative root, a
split file, parametrized and inherited tests, an ignored dir, a root
outside `tests/`, absolute root and `--rootdir` (fallback), and an
actual `-n 2` run.
- Duration files now record schema version, unit, commit, run id,
attempt, shard index and shard count. `merge` rejects mixed runs or
attempts, missing or duplicate shards, missing provenance, and negative
or non-finite values. It writes a `source` block into the merged file.
The checked-in weights were not regenerated.
- A subtest loop is one pytest item, so `file_weight`/parts can't split
it.

### CI timing

These numbers come from a full `workflow_dispatch` backend run on this
branch (22 shards, no selection, 22/22 green), taken before the
`-m`/`-k` fix was added. That fix doesn't touch sentry's own CI path.
They're compared with the master push run for a636572, the only
master run that includes all five PRs. Each value is the "Run backend
test" step: summed across shards, then the median and the slowest shard.

| Run | Sum | Median shard | Max shard |
|---|---|---|---|
| master a636572
([39051886634](https://github.com/getsentry/sentry/actions/runs/39051886634))
| 128.5 min | 365 s | 436 s |
| this branch
([39056799745](https://github.com/getsentry/sentry/actions/runs/39056799745))
| 135.8 min | 378 s | 429 s |

The slowest shard didn't change. The summed time is +7.3 min (+5.7%),
but that is runner noise, not the lock. Comparing each shard's per-file
durations, the 72 files with transactional tests moved +2.6% (+0.9 min),
and the 2,452 files that never reach the flush code moved +5.6% (+16.6
min). The four new test files add 46 s in total. At 10–20 ms each across
roughly 1,200 transactional tests, the lock costs well under a minute
over the whole run, a small fraction of #127941's saving. Master runs
before these PRs varied by ±10 min in summed time.

<!-- junior-request-attribution:start -->
via @joshuarli.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:agent%3Aaca1f40f54b2553bb3f2b5aa308eca88 -->

--

[View Junior
Session](https://junior-prod.sentry.dev/conversations/agent%3Aaca1f40f54b2553bb3f2b5aa308eca88)
[[Sentry]](https://sentry.sentry.io/explore/conversations/agent%3Aaca1f40f54b2553bb3f2b5aa308eca88/?project=4510944073809921)

<!-- junior-session-footer:end -->

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant