Repository navigation
build(deps): Bump pytest-xdist to 3.8.0, drop pytest-workaround-12888 - #127938
Merged
joshuarli merged 2 commits intoOct 17, 2026
Merged
Conversation
pytest 9.0.3 already imports readline early in _pytest/capture.py (_readline_workaround, pytest-dev/pytest#13176), so the .pth-based workaround for pytest-dev/pytest#12888 is redundant. pytest-xdist 3.5+ reorders --dist=loadfile work units by test count by default. Opt out from the shared sentry pytest plugin so file order and worker assignment stay identical after the bump (covers getsentry too). Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Pulls in execnet 2.1.2 (xdist 3.6+ requires execnet>=2.1). Both are served by the internal index after getsentry/pypi#2626. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
joshuarli
approved these changes
Oct 17, 2026
joshuarli
marked this pull request as ready for review
October 17, 2026 16:37
joshuarli
deleted the
joshua/build/xdist-bump-remove-pytest-workaround
branch
October 17, 2026 16:38
joshuarli
added a commit
that referenced
this pull request
Oct 17, 2026
Follow-up to retro feedback on #127938, #127939, #127941, #127942 and #127943. I checked each item against the code before changing anything. Below, each one is marked done or declined. No test case or assertion was dropped or weakened. Scheduling stays on `--dist=loadfile` with deterministic order, and xdist 3.8.0 keeps `loadscopereorder = False`. ### Filtered sharded runs (first commit) Since #127943, `pytest-selected.py` splits test files across shards before pytest applies `-m`/`-k`. Snuba's `ci` job runs sentry master's tests with `PYTEST_ADDOPTS="-k 'not __In' -m snuba_ci"` over 16 shards. As a result, most of those shards collected 0 tests and exited with code 5, and fail-fast cancelled the rest. This breaks the sentry tests job on every Snuba PR (e.g. getsentry/snuba#8556). The first commit falls back to the pre-#127943 per-test hash sharding whenever `-m` or `-k` appears in the args or in `PYTEST_ADDOPTS`. In that case every shard collects everything, and the sentry plugin keeps that shard's hash share after collection. Exit code 5 is still treated as a failure. Sentry's own backend CI passes neither flag, so it keeps per-file sharding. I reproduced Snuba's collection locally against this checkout: same args as `make test-python-ci`, the same `PYTEST_ADDOPTS` filter, `--collect-only`, 16 shards. Unsharded, 78 tests are selected. | | Shards with 0 tests (exit 5) | Tests per shard | Union of shards == unsharded (`Counter`) | |---|---|---|---| | #127943 | 8 of 16, including shards 0 and 12 | 0–23 | yes | | this fix | 0 of 16 | 2–8 | yes | New tests: `tests/sentry/testutils/pytest/test_sentry.py` runs `pytest-selected.py` and the real plugins over a throwaway project with `-m`, `-k`, and Snuba's `PYTEST_ADDOPTS` form. It checks that every shard runs something and that the shards together run each selected test exactly once. Without the fix these cases fail with exit code 5. There are also unit tests for the flag detection. getsentry's CI doesn't pass `-m`/`-k` to its `pytest-selected.py`. ### #127939: subtests in the JSON report (done) The rerun adapter in `json_report_reruns.py` treated every repeated `report.when` under a nodeid as a rerun. I reproduced this with the locked stack (pytest 9, pytest-json-report 1.5.0, rerunfailures, xdist). A test with 3 passing subtests that ran once got 3 `reruns` entries, and its `setup` moved into one of them. When a subtest failed, the only top-level traceback was "contains 1 failed subtest". pytest's exit code was still correct. - The adapter now detects `pytest.SubtestReport`. It keeps one parent `setup`/`call`/`teardown` and stores subtests in a new `subtests` list, each with its `context` (`msg`, repr'd `kwargs`), outcome, duration and longrepr. rerunfailures holds back parent reports until an attempt ends, so subtests are buffered and attached to the `call` they ran in. A genuine rerun still goes to `reruns` and carries that attempt's subtests along. - Outcome: the parent stays `passed` when every subtest passes and becomes `failed` when any subtest fails. I normalized away pytest's "subtests passed" label because `report-backend-test-failures.js` selects `outcome === 'failed'`. - Failure comments now show each failed subtest's context and traceback, up to 5 per test. - `backend-test-durations.py extract` still sums only the parent phases. The parent `call` already includes the subtest loop, so subtest durations are not added again. Since setup no longer moves into `reruns`, it is counted again. - `tests/sentry/testutils/pytest/test_json_report_reruns.py` runs the real plugin stack (json-report, this adapter, rerunfailures, xdist `-n 2`) in a subprocess. It checks that passing subtests make no reruns, a failed subtest keeps its context and assertion traceback, parent setup appears once, and a genuine rerun is recorded. Against the old adapter it fails 6 of 8 cases. ### #127941: transactional flush race (done) The race was real. The table inventory and the open-transaction check were separate statements. A row committed into a table with no pages between those two steps survived teardown. - Each database is now cleaned in one transaction: `lock_timeout = 1s`, then `LOCK TABLE <every managed table> IN EXCLUSIVE MODE` (this blocks writes but allows reads), then inventory, fingerprint check and `DELETE`. A lock timeout or deadlock falls back to Django's flush, which waits for writers the way `TRUNCATE` always has. The open-transaction check stays, both before the lock (so a known-open writer skips the 1s wait) and under it. The lock blocks writers that have already written. A transaction that has only read can still write after the lock is released. - Setting `SENTRY_LEGACY_TRANSACTIONAL_FLUSH=1` keeps Django's teardown. `pytest_unconfigure` restores the original method and clears the seeded-row cache. The cache is keyed by alias plus database `NAME`. - Stable contenttype/permission IDs are unchanged (intentional). - `tests/sentry/testutils/pytest/test_transactional_flush.py` drives `_fixture_teardown` explicitly, so it doesn't depend on test order. It covers: - ORM and raw SQL rows across default/control/secondary, plus a large TOAST value - inserting, updating and deleting seeded contenttype/permission/site rows, which forces the fallback and restores state - the `available_apps` and `serialized_rollback` fallbacks - a second connection that tries to commit into an empty table during cleanup: it blocks on the lock and no row survives - a writer already open at teardown start, cleaned up through the fallback - lock-timeout fallback, the escape hatch, and unconfigure The race tests synchronize on `pg_blocking_pids` / `pg_stat_activity`, not sleeps, and they fail with the lock disabled. - Lock cost, measured locally against the real test schema (254 default, 70 control, 6 secondary tables). A warm teardown across all DBs went from ~15–18 ms to ~23–27 ms. A cold connection went from ~85–92 ms to ~101–112 ms. That is roughly 10–20 ms per teardown against ~1.27 s for Django's flush, so the lock keeps almost all of the saving. CI agrees (see below). - Audits: - No non-internal `pg_trigger` rows exist in any test DB, so DELETE vs TRUNCATE trigger behavior doesn't matter. - The `post_migrate` receivers the skip bypasses are only Django's `create_contenttypes`, `create_permissions` and `create_default_site`. The fingerprint covers all of their rows. - Dead tuples: across 117 transactional teardowns in one session, the first-quarter median was 0.126 s and the last-quarter median 0.116 s. Teardowns did not slow down over the session. - Declined: making the plugin opt-in, separate toggles for DELETE vs the `post_migrate` skip, and non-Postgres support. Sentry tests only run on Postgres, and those backends already take Django's path. ### #127942: Relay readiness (done) `wait_until_ready()` checks the time left before each probe. The request timeout is `min(1s, remaining)`, sleeps are capped by the time left, and no probe starts after the deadline. A 200 from a probe that started before the deadline counts as ready even if it arrives late. It catches `requests.RequestException` instead of `Exception`. The failure message includes the last HTTP status or exception plus the container logs, and the container is removed even if fetching the logs fails. Every test still gets a fresh container. Unit tests with an injected clock, probe and sleep cover: immediate ready, 503→200, connection error→success, permanent 503, request timeout, near-deadline success, and bounded sleep. ### #127939: subtest isolation (done / declined) - `test_buffer.py`: each permutation now builds a fresh `SpansBuffer`, which resets `any_shard_at_limit` and the cached properties. The options and Redis setup stay shared. I re-checked with reverse permutation order and an injected early failure: exactly the injected subtest failed and the rest passed. The redis-cluster variants ran green in the CI run below. - `test_client_config.py`: declined. Each factory clears caches and builds its own users and orgs, and the org/project lookups depend on the request. I found no shared global query or cache, and reverse order already passed all 758 subtests. - Declined: documenting how to reproduce a single subtest. There's no natural place for it, and subtests can't be selected by nodeid. ### #127943: sharding boundaries (done) - Absolute roots, `..` roots, and `--rootdir` or `-c`/`--config-file` (in the args or `PYTEST_ADDOPTS`) produce paths that don't match the sentry plugin's rootdir-relative comparison. `pytest-selected.py` now falls back to the legacy hash sharding for them. `pytest_shard_plugin` raises a usage error if the rootdir isn't the working directory. CI passes the relative root `tests` and keeps the fast path. - Files outside `tests/` were not duplicated across shards. `pytest_collection_modifyitems` already drops them by `SELECTED_TESTS_FILE`; they are only imported. A new test covers this. - `tests/sentry/testutils/pytest/test_sentry.py` runs the real `pytest-selected.py`, sentry plugin and shard plugin over a throwaway project with 3 shards. It asserts that the `Counter` of unsharded nodeids equals the sum of the shard Counters. Cases: relative root, a split file, parametrized and inherited tests, an ignored dir, a root outside `tests/`, absolute root and `--rootdir` (fallback), and an actual `-n 2` run. - Duration files now record schema version, unit, commit, run id, attempt, shard index and shard count. `merge` rejects mixed runs or attempts, missing or duplicate shards, missing provenance, and negative or non-finite values. It writes a `source` block into the merged file. The checked-in weights were not regenerated. - A subtest loop is one pytest item, so `file_weight`/parts can't split it. ### CI timing These numbers come from a full `workflow_dispatch` backend run on this branch (22 shards, no selection, 22/22 green), taken before the `-m`/`-k` fix was added. That fix doesn't touch sentry's own CI path. They're compared with the master push run for a636572, the only master run that includes all five PRs. Each value is the "Run backend test" step: summed across shards, then the median and the slowest shard. | Run | Sum | Median shard | Max shard | |---|---|---|---| | master a636572 ([39051886634](https://github.com/getsentry/sentry/actions/runs/39051886634)) | 128.5 min | 365 s | 436 s | | this branch ([39056799745](https://github.com/getsentry/sentry/actions/runs/39056799745)) | 135.8 min | 378 s | 429 s | The slowest shard didn't change. The summed time is +7.3 min (+5.7%), but that is runner noise, not the lock. Comparing each shard's per-file durations, the 72 files with transactional tests moved +2.6% (+0.9 min), and the 2,452 files that never reach the flush code moved +5.6% (+16.6 min). The four new test files add 46 s in total. At 10–20 ms each across roughly 1,200 transactional tests, the lock costs well under a minute over the whole run, a small fraction of #127941's saving. Master runs before these PRs varied by ±10 min in summed time. <!-- junior-request-attribution:start --> via @joshuarli. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:agent%3Aaca1f40f54b2553bb3f2b5aa308eca88 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/agent%3Aaca1f40f54b2553bb3f2b5aa308eca88) [[Sentry]](https://sentry.sentry.io/explore/conversations/agent%3Aaca1f40f54b2553bb3f2b5aa308eca88/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps
pytest-xdist3.0.2 → 3.8.0 (latest), and with itexecnet1.9.0 → 2.1.2. Also removes thepytest-workaround-12888dev dependency and keeps xdist's--dist=loadfileordering exactly as it was.Bump is done. getsentry/pypi#2626 mirrored
pytest-xdist==3.8.0andexecnet==2.1.2to internal PyPI (pypi.devinfra.sentry.io). The bump comes from that index viauv@0.12.10 add --dev "pytest-xdist>=3.8.0", the same commandtools/bump_version.pyruns in thebump-versionworkflow. It's applied on this branch rather than through a separate workflow PR so it lands together with the ordering guard below. No public index or override was added, and the lock's wheel hashes match the upstream release artifacts.Removing
pytest-workaround-12888The package was only a
.pthfile that ranimport readlineearly, to work around pytest-dev/pytest#12888. pytest-dev/pytest#13176 fixed that upstream, and pytest 9.0.3 (our locked version) already calls_readline_workaround()from_pytest/capture.pyinpytest_load_initial_conftests. Nothing else in the repo refers to it.Keeping test ordering the same (
--dist=loadfile)Starting in xdist 3.5.0 (pytest-dev/pytest-xdist#632),
LoadScopeSchedulingsorts work units by test count, largest first, before handing them out.LoadFileSchedulingsubclasses it, so--dist=loadfileis affected too, and this is on by default. On its own, the bump would change the order files are handed to workers. That changes which files share a worker and the order they run in. It would also override theSENTRY_SHUFFLE_TESTSordering.xdist 3.8.0 (pytest-dev/pytest-xdist#1083) added
--no-loadscope-reorderto turn this off. This PR sets the same thing (config.option.loadscopereorder = False) insentry.testutils.pytest.sentry:pytest_configure, so the work queue keeps collection order exactly as it did on 3.0.2. getsentry loads the same plugin, so it's covered too.--dist=loadfileand how we pick worker counts are not changed.I checked the rest of the scheduler diff between 3.0.2 and 3.8.0 (
scheduler/loadscope.py,scheduler/loadfile.py). Apart from the reorder, the changes are typing and switchingOrderedDictto plaindict. Insertion order is kept, and dropping extra nodes still pops the last node. File scoping in_split_scopeis unchanged.testrunuidis stilluuid4().hexper run, so the deterministicPYTEST_XDIST_TESTRUNUID-seeded cell name works the same.xdist 3.0.2 → 3.8.0 changes that matter for our CI (
-n 2/3 --dist=loadfile)--maxfailoutput gets more useful.main_thread_onlyexecmodel). Before this, execnet could run tests in a non-main thread.shouldstop/shouldfail(for example--maxfail) instead of running more tests before the controller stops them.looponfailroots.main_thread_onlymodel and pytest-cov readingrinfo. Relevant to the coverage jobs (pytest-cov4.0.0).execnet>=2.1(1.9.0 → 2.1.2) andpytest>=7. We're on pytest 9.0.3 and Python 3.13. 3.7.0 adds official 3.13 support.Things that stay the same:
pytest_xdist_setupnodeshookspec.PYTEST_XDIST_WORKER/PYTEST_XDIST_TESTRUNUID/PYTEST_XDIST_WORKER_COUNT, still set inxdist/remote.py.--looponfail/looponfailrootsand--rsyncdirdeprecation warnings: same text, so the existingfilterwarningsentries still match.Other changes in this range don't touch the code paths we use (
--dist=loadfile,-n), and no scheduling options are adopted or proposed.Verification (local, env synced from this branch's
uv.lockvia internal PyPI)pytest -n 3 --dist=loadfile --reuse-db -q tests/sentry/api/bases/: 159 passed, with the workaround package uninstalled.-n 1 --dist=loadfileontests/sentry/api/bases/, files ran in collection order (test_organization64 tests,test_organization_events10,test_organizationmember8,test_project44,test_team33), not sorted by size. A synthetic suite gave the same order on 3.0.2, on 3.8.0 +--no-loadscope-reorder, and on 3.8.0 + the plugin pin. Plain 3.8.0 reordered it.tests/getsentry/test_features.pywith-n 2 --dist=loadfile: 121 passed, with and without-W ignore::DeprecationWarning:xdist.plugin. Checked earlier against the same xdist/execnet versions.prek runand mypy are clean.via @joshuarli.
--
View Junior Session [Sentry]