Skip to content

Release scripts check the tree for cleanliness once, then build from it for minutes #163

Description

@kiki830621

Problem

The release scripts check for a clean working tree once, at step 0, then spend several minutes building from that same tree. Anything that edits a source file during the build lands in the signed, notarized artifact — and no gate downstream can see it.

# step 0
[[ -z "$(git status --porcelain)" ]] \
    || { echo "error: working tree not clean …"; exit 3; }

# step 1 — minutes later, reads the tree again
swift build -c release

This is the same TOCTOU shape the scripts already defend against for the upload artifact: step 6 re-verifies the signature and sha256 of the exact bytes about to be uploaded, with the comment FINAL GATE — re-verify the exact upload artifact (TOCTOU guard). The source tree gets no equivalent, and it has the wider window: the universal build is the longest step in the pipeline.

Near-miss (this is not hypothetical)

Today, during a scripts/release.sh 4.0.4 run on che-word-mcp, I edited Sources/CheWordMCP/Server.swift while the build was in progress — working on an unrelated issue and treating the release as a background job. Caught it by checking the build log:

93%: Compile CheWordMCP (x86_64)
93%: Compiling AuthorAliasMap.swift, …, Server.swift, SessionState.swift

The compiler was reading that file at that moment. The run was killed before it reached notarization; no tag and no release were created, and the release was redone from a clean tree afterwards.

Had it completed, the output would have been a Developer ID signed, Apple notarized binary containing an uncommitted, untested edit — with every downstream gate passing, because they all verify the artifact's provenance, never its correspondence to a commit. The wrapper's sha256 and signature checks would have been satisfied. git tag v4.0.4 would have pointed at a commit whose source is not what shipped.

Why the existing gates cannot catch it

gate what it proves catches this?
step 0 clean-tree the tree was clean then no — point-in-time
step 3 signature gate Developer ID, correct team no — a tampered build is still correctly signed
step 6 final gate uploaded bytes == verified bytes no — both are the wrong bytes
wrapper install verify sha256 + signature match the release no — matches the wrong binary faithfully

Every check is about the artifact. None ties the artifact to a revision.

Suggested fix

Re-check the tree immediately after the build, before signing, and fail if it moved:

# step 0
TREE_AT_START=$(git rev-parse HEAD)
DIRT_AT_START=$(git status --porcelain)

# after step 1, before codesign
[[ "$(git rev-parse HEAD)" == "$TREE_AT_START" && "$(git status --porcelain)" == "$DIRT_AT_START" ]] \
    || { echo "error: working tree changed during the build — the binary may contain code that is in no commit; refusing to sign" >&2; exit 3; }

Cheap, and it closes the window where it matters: before the signature is applied, so nothing carrying an unknown edit ever gets a Developer ID.

Worth considering alongside it: stamp the built commit into the release notes (--target already uses git rev-parse HEAD), so an artifact can be traced to a revision after the fact rather than only forward from one.

Scope

Same code in at least:

  • PsychQuant/macdocscripts/release-cli.sh
  • PsychQuant/che-word-mcpscripts/release.sh

and likely the other che-*-mcp release scripts, which share this pipeline's shape. Filed here because both gate blocks cite PsychQuant/macdoc#119 as the origin of the requirement.

Honest note on severity

The window only opens if someone edits during a release, which is unusual for a human running one release at a time. It is much less unusual for an agent treating the release as a background task and continuing to work — which is exactly how it happened. That makes this cheaper to fix than to remember.

Current Status

Phase: verified
Last updated: 2026-08-24 by idd-verify

Key Decisions

  • Build every release from a fresh detached worktree at captured SOURCE_HEAD with a fresh .build.
  • Concurrent primary-tree edits are isolated; isolated-tree drift fails before codesign.
  • Pin release target and notes to the captured commit.

Scope Changes

  • Replaced the initial post-build primary-tree snapshot with source isolation after adversarial review.
  • Explicitly trust compiler/build plugins; reproducible-toolchain attestation remains outside this issue.

Blocking

  • (none — four verify-gated PRs await human review/merge)

Commits / PRs

Cross-repo implementation checklist

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions