Run CodSpeed benchmarks in parallel shards

[patrick91]

Summary

• split the CodSpeed benchmark job across four parallel GitHub Actions runners
• distribute all benchmark cases with pytest-test-groups
• add the sharding plugin to the locked development and test dependencies
• require uv sync --locked before the OIDC-enabled CodSpeed step
• keep the existing CodSpeed simulation mode and benchmark coverage unchanged

Motivation

The original benchmark job took 24 minutes, with 22 minutes 41 seconds spent running the simulated benchmarks. The stadium benchmark accounted for nearly half of that time.

The four shards place the expensive stadium cases on separate runners and also distribute the large-query and subscription cases. In the first complete CI run, the slowest shard finished in 9 minutes 34 seconds.

Locking pytest-test-groups ensures its distribution is verified against the SHA-256 hashes in uv.lock instead of being resolved dynamically inside a job with id-token: write.

Verification

• installed pytest-test-groups 1.2.1 with uv sync --locked
• confirmed all 35 pytest benchmark cases are selected exactly once across shard sizes 9, 9, 9, and 8
• ran all four shard selections locally; all benchmark tests passed
• confirmed all four GitHub Actions shard jobs passed and uploaded their results
• confirmed CodSpeed combined the uploads into the same 34-benchmark report produced by the original unsharded run
• parsed and checked the updated workflow configuration
• ran the applicable pre-commit hooks

Summary by Sourcery

Run the CodSpeed benchmark suite in four parallel shards to reduce CI execution time.

Enhancements:

• Parallelize CodSpeed benchmark execution across four GitHub Actions shards while preserving the existing benchmark suite and simulation mode.

Build:

• Add pytest-test-groups to the development dependencies and lockfile, and require locked dependency synchronization for benchmark jobs.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Needs a human reviewer. If the shard selection or added pytest-test-groups dependency is wrong, benchmark runs could omit or duplicate tests and publish incomplete or misleading CodSpeed results. Reverting restores the previous benchmark job, and affected measurements can be corrected by rerunning the benchmarks, but already-published results may outlive the revert.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[greptile-apps]

Greptile Summary

The PR splits the CodSpeed benchmark workflow across four parallel GitHub Actions jobs while preserving simulation mode.

• Adds a four-shard benchmark matrix with failures reported independently.
• Uses pytest-test-groups to distribute the benchmark collection across the matrix.
• Introduces a runtime-resolved CI plugin in the OIDC-enabled benchmark job.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking supply-chain hardening opportunity for the runtime-resolved sharding plugin.

The sharding configuration partitions the current benchmark collection completely, but pytest-test-groups is fetched outside the lockfile and executes in a job capable of requesting an OIDC token.

Files Needing Attention: .github/workflows/test.yml

Security Review

The runtime-only sharding plugin is not covered by the repository lockfile, so a compromised package distribution would execute within an OIDC-enabled benchmark job.

Important Files Changed

Filename	Overview
.github/workflows/test.yml	Adds deterministic four-way benchmark sharding; the remaining non-blocking concern is integrity protection for the dynamically resolved plugin.

_{Reviews (1): Last reviewed commit: "Shard CodSpeed benchmarks across runners" | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 34 untouched benchmarks

---

_{Comparing ci/shard-codspeed-benchmarks (98defa8) with main (e104af8)¹}

Footnotes

1. No successful run was found on main (ff33238) during the generation of this report, so e104af8 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[github-actions]

Hi, thanks for contributing to this project!

We noticed that this PR is missing a RELEASE.md file. We use that to automatically do releases here on GitHub and, most importantly, to PyPI!

So as soon as this PR is merged, a release will be made 🚀.

Here's an example of RELEASE.md:

---
release type: patch
---

Description of the changes, ideally with some examples, if adding a new feature.

Release type can be one of patch, minor or major. We use [semver](https://semver.org/), so make sure to pick the appropriate type. If in doubt feel free to ask :)

[sourcery-ai]

Sourcery assessment

Approved.

Sourcery withdrew this approval because the latest commits introduced blocking findings.

[botberry]

Apollo Federation Subgraph Compatibility Results

Federation 1 Support	Federation 2 Support
_service 🟢 @key (single) 🟢 @key (multi) 🟢 @key (composite) 🟢 repeatable @key 🟢 @requires 🟢 @provides 🟢 federated tracing 🟢	@link 🟢 @shareable 🟢 @tag 🟢 @override 🟢 @inaccessible 🟢 @composeDirective 🟢 @interfaceObject 🟢

Learn more:

• Apollo Federation Subgraph Specification
• Compatibility Tests

[sourcery-ai]

Sourcery assessment

Approved.