Skip to content

Commit bc609c8

Browse files
dannycjoneskevinjqliuclaude
authored
ci: check new crates are reserved on crates.io before merge (#3185) (#3218)
* ci: check new crates are reserved on crates.io before merge Trusted publishing can't create a crate, and the release workflow publishes the whole workspace in one `cargo publish --workspace` run, which cargo aborts if any crate already exists at the release version. So the existing "publish the new crate by hand after the workflow fails" instructions no longer work. Add a lint step that fails if a publishable crate is missing from crates.io, and rewrite the release guide appendix around reserving the crate with a placeholder version when it is added. * docs: reserve the crate name with a standalone placeholder crate Overriding the version inside the workspace fails because iceberg depends on the new crate at the workspace version. Verified with cargo publish --dry-run. * ci: match the existing publishable-crate filter; document partial publish recovery * docs: clarify that reserving a crate needs a token with the publish-new scope Owning the existing crates does not imply the token in ~/.cargo/credentials.toml has publish-new, and crates.io reports a missing scope as a generic authentication failure. Say so, and point at where to create the token. * docs: fix stale first-publish note, empty placeholder, and owner invitations - The release steps still said the publish workflow is expected to fail on new crates and linked to a heading that no longer exists. - `cargo new --lib` generates a non-empty lib.rs; use mkdir and touch. - `cargo owner --add <user>` sends an invitation that must be accepted. * ci: fail the crates.io check if the crate list is empty A failed `cargo metadata` inside `for pkg in $(...)` never trips `bash -e`, so the loop ran zero times and the step passed without checking anything. Capture the list first and fail if it is empty. --------- (cherry picked from commit cb912c9) Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent d126ce6 commit bc609c8

2 files changed

Lines changed: 92 additions & 25 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,22 @@ jobs:
7878
- name: Check crates package LICENSE and NOTICE
7979
run: make check-license-notice
8080

81+
# Trusted publishing cannot create a crate (https://crates.io/docs/trusted-publishing),
82+
# so a new publishable crate must be reserved on crates.io before it is merged.
83+
# See "Adding a new crate" in website/src/release.md.
84+
- name: Check publishable crates exist on crates.io
85+
run: |
86+
pkgs=$(cargo metadata --no-deps --format-version=1 | jq -r '.packages[] | select(.publish != []) | .name')
87+
# A failed `cargo metadata` would otherwise leave the loop empty and the step green.
88+
[ -n "$pkgs" ] || { echo "::error::No publishable crates found in the workspace, expected at least one"; exit 1; }
89+
for pkg in $pkgs; do
90+
# --registry is required, otherwise cargo resolves the local workspace member.
91+
if ! cargo info --registry crates-io "$pkg" >/dev/null; then
92+
echo "::error::$pkg is not on crates.io. Reserve it before merging; see 'Adding a new crate' in website/src/release.md"
93+
exit 1
94+
fi
95+
done
96+
8197
- name: Cargo Machete
8298
run: cargo machete
8399

‎website/src/release.md‎

Lines changed: 76 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -531,9 +531,8 @@ The creation of the final release tag triggers the publish workflow for crates.
531531
Python packages are manually triggered later.
532532
Please verify that the triggered workflows for the crates succeeded.
533533

534-
Note, this workflow for crates is expected to fail if new crates are being published for the first time.
535-
In this instance, a committer must manually publish the crate in order to continue.
536-
See [publishing a crate for the first time](#publishing-a-crate-for-the-first-time) for what steps to take.
534+
New crates are reserved on crates.io when they are added, so the workflow should not fail on a new crate.
535+
If it does, see [if the Publish workflow fails on a missing crate](#if-the-publish-workflow-fails-on-a-missing-crate).
537536
Once all the crates are published, Python publishing should start.
538537

539538
Python publishing is performed by a GitHub workflow, however the trigger is manual.
@@ -614,45 +613,97 @@ however it is in the community's best interest to publish it soon after.
614613

615614
## Appendix
616615

617-
### Publishing a crate for the first time
616+
### Adding a new crate
618617

619-
Publishing the Iceberg crates is automated using GitHub Actions,
620-
which authenticates with crates.io using trusted publishing.
618+
Every publishable crate must exist on crates.io before the pull request that adds it is merged.
619+
CI enforces this with a check that fails for any publishable crate missing from crates.io.
621620

622-
Trusted publishing must first be configured for each crate.
623-
If the crate has never been published, crates.io rejects attempts to publish.
621+
The release workflow publishes crates with [trusted publishing](https://crates.io/docs/trusted-publishing), which cannot create a crate.
622+
crates.io requires the first version of a crate to be published manually with an API token ([announcement](https://blog.rust-lang.org/2025/07/11/crates-io-development-update-2025-07/)).
623+
Publishing the crate by hand after the release workflow fails does not work either: the workflow runs `cargo publish --workspace`, which refuses to run if any crate already exists at the release version.
624624

625-
When a release workflow fails due to the presence of a new crate, a committer must perform steps enumerated below.
626-
Once complete, future versions can be published automatically using GitHub Actions.
625+
A committer therefore reserves the crate when it is added, in three steps: publish a placeholder version, configure owners, and configure trusted publishing.
627626

628-
#### Initial crate publish
627+
#### Prerequisites
629628

630-
Manually publish the crate using the following command.
631-
You **must** have the source code checked out matching the pushed Git tag for the release.
629+
Any crates.io account can publish an unclaimed name, and the first publisher becomes the crate's only owner.
630+
Reserve the crate promptly once the pull request is open, and have a committer do it, since the reserver must hand ownership to the project.
631+
632+
The committer needs:
633+
634+
- A crates.io account linked to their GitHub account.
635+
- Membership in the `apache/iceberg-private` GitHub team, with the `read:org` permission granted to crates.io, in order to add that team as an owner.
636+
- A crates.io API token with the `publish-new` and `change-owners` scopes, used with `cargo login`.
637+
Existing tokens usually lack `publish-new`; create one at [crates.io/settings/tokens](https://crates.io/settings/tokens).
638+
639+
Committers who already own the existing Iceberg crates meet the first two.
640+
List them with `cargo owner --list iceberg`.
641+
642+
#### Step 1: Reserve the crate name
643+
644+
Publish an empty placeholder crate as version `0.0.0`, so it never collides with a real release.
645+
The placeholder is created outside the repository; overriding the version inside the workspace does not work because other workspace crates depend on the new crate at the workspace version.
632646

633647
```shell
634-
cargo publish --package <package-name>
648+
mkdir -p /tmp/<package-name>/src
649+
cd /tmp/<package-name>
650+
touch src/lib.rs
651+
cat > Cargo.toml <<'EOF'
652+
[package]
653+
name = "<package-name>"
654+
version = "0.0.0"
655+
edition = "2024"
656+
license = "Apache-2.0"
657+
description = "Placeholder that reserves the crate name for Apache Iceberg Rust. See the repository for the real crate."
658+
repository = "https://github.com/apache/iceberg-rust"
659+
EOF
660+
cargo publish --dry-run
661+
cargo publish
635662
```
636663

637-
#### Configure crate permissions
664+
A `403 Forbidden: authentication failed` error means the token lacks the `publish-new` scope.
638665

639-
After publishing succeeds, the crate must be configured to allow other committers to publish.
666+
#### Step 2: Configure crate owners
640667

641-
Add the GitHub team that Apache Iceberg committers are a member of.
668+
Add the GitHub team that owns the existing Iceberg crates.
642669

643670
```shell
644-
cargo owner --add github:<github-team-org>:<github-team-name>
671+
cargo owner --add github:apache:iceberg-private <package-name>
645672
```
646673

647-
Additionally, add two PMC members (excluding yourself) as owners of the crate.
648-
A GitHub team cannot manage permissions for the crate, so it is important that individuals have ownership to continue being able to manage the crate should a PMC member become inactive.
649-
See the [Cargo documentation for `cargo owner`](https://doc.rust-lang.org/cargo/reference/publishing.html#cargo-owner) for reference.
674+
Then add two PMC members other than yourself as individual owners.
650675

651676
```shell
652-
cargo owner --add <github-handle>
677+
cargo owner --add <github-handle> <package-name>
653678
```
654679

655-
#### Configure trusted publishing
680+
This sends an invitation; ask each invitee to accept it on crates.io.
681+
682+
Individual owners are required because a team owner can publish and yank versions but cannot add or remove owners.
683+
Without them, the crate could become unmanageable if a PMC member becomes inactive.
684+
See the [Cargo documentation for `cargo owner`](https://doc.rust-lang.org/cargo/reference/publishing.html#cargo-owner).
685+
686+
#### Step 3: Configure trusted publishing
687+
688+
As a crate owner, open the crate's settings on the crates.io website and add a trusted publisher, following the [crates.io trusted publishing documentation](https://crates.io/docs/trusted-publishing).
689+
Use an existing Iceberg crate as the reference:
656690

657-
Review the [crates.io trusted publishing documentation](https://crates.io/docs/trusted-publishing) for the latest instructions on how to configure it.
658-
You should also review another already-existing crate for reference.
691+
- Repository: `apache/iceberg-rust`
692+
- Workflow: `publish.yml`
693+
- Environment: `publish`
694+
695+
Then re-run CI on the pull request.
696+
The next release publishes the crate's first real version along with the rest of the workspace.
697+
698+
#### If the Publish workflow fails on a missing crate
699+
700+
Reserve the crate as above, using the `0.0.0` placeholder and never the release version.
701+
702+
Then check the failed run's log for `Uploaded` lines to see which crates were already published, since cargo uploads crates one at a time in dependency order.
703+
704+
- If nothing was uploaded, re-run the failed workflow run from the GitHub Actions UI. Re-running keeps the original tag as the workflow ref.
705+
- If some crates were uploaded, publish the rest from a clean checkout of the release tag, excluding each crate that was already published:
706+
707+
```shell
708+
cargo publish --workspace --all-features --exclude <published-crate> --exclude <published-crate>
709+
```

0 commit comments

Comments
 (0)