[chrisjsewell]

Draft: the publish path can only be fully exercised by a tag push.

What and why

The release build is broken for both distributions, so the next tag push cannot publish anything.

Both publish jobs in tests.yml run pip install flit~=3.4 and then flit build.
That specifier now resolves flit 3.12.0, which requires flit_core>=3.12 with no cap, so it pulls in flit_core 4.2.0, and flit build crashes:

TypeError: SdistBuilder.build() got an unexpected keyword argument 'gen_setup_py'

This PR changes .github/workflows/tests.yml (the file name is unchanged, because both PyPI trusted publishers name it):

• A new build job runs on every push and pull request.
  It builds myst-parser with python -m build --outdir dist-parser.
  It then rewrites the project with the existing .github/workflows/docutils_setup.py, unchanged, and builds myst-docutils with python -m build --outdir dist-docutils.
  • python -m build uses the [build-system] pin in pyproject.toml (flit_core >=3.4,<4), so the release build cannot drift away from it again.
  • It runs twine check --strict on all four files.
  • It checks that each directory holds exactly one wheel and one sdist, and that they belong to the expected project. This catches a rewrite that silently did nothing.
  • On a tag, it checks that the built version equals the tag.
  • It uploads each directory as its own artifact, dist-parser and dist-docutils.
    They are kept for the repository's default retention period, not a short one,
    so a publish job that runs late (after a slow environment approval, or as a re-run) still finds them.
  • It checks out with persist-credentials: false.
• build is added to the check aggregator, so a broken release build fails a PR instead of a tag push.
• publish and publish-docutils no longer check out or build anything.
  Each one downloads its own artifact and uploads it with pypa/gh-action-pypi-publish from its own packages-dir, using trusted publishing (environment pypi, id-token: write, both unchanged).
  So what is published is exactly what CI built and checked.
  publish-docutils still runs only after publish succeeds.

How it was tested

The red run reproduces the bug on master (723cffc):

$ uv venv -p 3.13 venv-flit && uv pip install --python venv-flit/bin/python "flit~=3.4" tomlkit
  # resolves flit 3.12.0 and flit-core 4.2.0
$ flit build
TypeError: SdistBuilder.build() got an unexpected keyword argument 'gen_setup_py'   # rc=1

For the green runs, the new job's steps were run locally with Python 3.13, build 1.6.1, twine 7.0.0 and tomlkit 0.15.2, at both v5.1.0 and this branch:

• python -m build --outdir dist-parser, then docutils_setup.py, then python -m build --outdir dist-docutils: rc=0 at both refs.
• twine check --strict on all four files: PASSED.
• The name and version check script, extracted from the workflow file and run directly:
  • no tag: rc=0;
  • TAG=v5.1.0: rc=0;
  • TAG=v5.2.0: rc=1, with tag v5.2.0 != built version(s) ['5.1.0'] for each directory;
  • with the docutils_setup.py step skipped: rc=1, with dist-docutils: expected myst_docutils, got ['myst_parser'].

These builds were compared with the files on PyPI for 5.1.0, built from the v5.1.0 tag with the new job:

	myst-parser	myst-docutils
wheel file list	identical (35 entries)	identical (35 entries)
wheel METADATA	byte-identical	byte-identical
sdist PKG-INFO	byte-identical	byte-identical
sdist file list	34 entries vs 51 (see below)	34 entries vs 51 (see below)

On this branch the only file-list change from that comparison is the new myst_parser/slugs.py module, in both wheels and both sdists.
The contents of other modules differ too, because master has moved on since 5.1.0.

Linting:

• actionlint .github/workflows/tests.yml reports no new findings. The one finding left is a SC2102 info at line 93, which is already on master, in the unchanged check-myst-docutils install step.
• pre-commit run --all-files: all hooks pass.

Notes for the reviewer

• The sdists get smaller. The flit CLI built sdists in VCS mode, so they included every git-tracked file not excluded. python -m build goes through the flit_core backend, which ships only the package plus pyproject.toml, README.md, LICENSE and PKG-INFO.
  These files drop out of both sdists:

  • .github/
  • .gitignore
  • .pre-commit-config.yaml
  • .readthedocs.yml
  • AGENTS.md
  • CHANGELOG.md
  • codecov.yml
  • example-include.md
  • tox.ini

  tests/ and docs/ were already excluded, by [tool.flit.sdist] exclude in pyproject.toml. None of these files is needed to build or install, and the wheels are unchanged.
  If CHANGELOG.md or tox.ini should stay in the sdist, a [tool.flit.sdist] include list in pyproject.toml adds them back. I left that out because it is a packaging-policy choice.
  The same goes for shipping tests/ in the sdist, which would also mean dropping it from the existing exclude list.

• What myst-docutils declares is unchanged. docutils_setup.py is untouched, so myst-docutils still declares no runtime docutils requirement: docutils and sphinx stay in its [sphinx] extra, as in 5.1.0. Whether to change that is a separate decision.

• The tag path cannot be fully tested before a tag. The download and upload steps follow the PyPA trusted-publishing guide's pattern. Trusted publishers are registered for both projects, against workflow tests.yml and environment pypi.

• Kept out of scope:

  • the test-matrix work;
  • flit_core <5 (⬆️ Update flit-core requirement from <4,>=3.4 to >=3.4,<5 #1185);
  • a top-level permissions: contents: read;
  • creating GitHub releases automatically.