Skip to content

Commit 614bf9a

Browse files
authored
Fix macOS notarization by signing the Dart Sass snapshot (#14718)
Dart Sass 1.101.0 changed its macOS AOT snapshot (dart-sass/src/sass.snapshot) from an ELF blob to a native Mach-O. Apple's notary ignores non-native files but requires every Mach-O in the payload to be signed, so the never-signed snapshot flipped notarization to Invalid and no ticket was issued -- the "CloudKit Record not found" seen at the stapling step was a symptom of the missing ticket, not a propagation race. The previous exit-code-only check masked the rejection entirely. - Sign dart-sass/src/sass.snapshot (no entitlements; it is loaded by the already-signed dart VM, so it needs only a valid signature and secure timestamp) - Verify notarytool returns status: Accepted before stapling, dumping the developer log on rejection, so a rejected submission fails the build instead of proceeding silently - Echo notarytool output and log command stdout on failure, so the notarization verdict is visible in CI - Keep a short, fail-loud retry around stapling as narrow insurance against a genuine, rare CloudKit ticket-propagation gap - Document the incident and add a pre-merge step to the dependency upgrade checklist: bundled-binary bumps must be validated by dispatching create-release.yml, since normal CI never signs or notarizes the installers - Add a repo rule for single current-year copyright headers on new files
1 parent 663c449 commit 614bf9a

7 files changed

Lines changed: 236 additions & 18 deletions

File tree

‎.claude/rules/copyright.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
paths:
3+
- "**/*.ts"
4+
- "**/*.js"
5+
- "**/*.lua"
6+
---
7+
8+
# Copyright headers
9+
10+
New source files get a single current-year header — `Copyright (C) <year> Posit Software, PBC` (e.g. `2026`), not a back-dated `2020-<year>` range. Leave existing files' headers alone unless you're already substantially editing them.
11+
12+
Don't guess the year — read it from the system (`date +%Y`, or the current date stated at session start). Copying a header from an existing file carries a stale year.

‎dev-docs/upgrade-dependencies.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,17 @@ Change version numbers in `./configuration` to correspond to new versions.
22

33
Update hardcoded version strings in `src/command/check/check.ts` (`versionConstraints` array, ~line 249) so that they match the new versions in `configuration`. The `configuration` file warns about this in a comment.
44

5+
## Verify installer signing & notarization before merging (bundled binaries)
6+
7+
Regular CI does **not** build, sign, or notarize the installers — that only happens in `create-release.yml`. A bundled binary bump (Deno, Pandoc, Dart Sass, Typst, esbuild, …) can change what the platform code-signing/notarization steps must cover, so a bump that passes normal CI can still break the macOS/Windows release build. **Before merging any PR that bumps a bundled binary**, dispatch the release workflow on the PR branch without publishing and confirm the installer jobs pass:
8+
9+
```bash
10+
gh workflow run create-release.yml --repo quarto-dev/quarto-cli --ref <branch> -f publish-release=false
11+
# then watch make-installer-mac and make-installer-win in the resulting run
12+
```
13+
14+
Why this bites (real incident, #14664): Dart Sass 1.101.0 changed its macOS AOT snapshot (`dart-sass/src/sass.snapshot`) container from ELF to Mach-O. Apple's notary ignores non-native files but requires every Mach-O signed, so the never-signed snapshot flipped notarization from Accepted to Invalid — invisible to normal CI, only caught at release time. When a bump does require a new signing entry, add it in `package/src/macos/installer.ts` / the Windows `sign-files` paths; see `llm-docs/code-signing-installers.md`.
15+
516
## Upgrade deno
617

718
### Upgrade standard library

‎llm-docs/code-signing-installers.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ Sequence:
8282
1. **Decide whether to sign** — reads `QUARTO_APPLE_APP_DEV_ID`. Empty → skip everything below with a warning.
8383
2. **Sign individual binaries** with `codesign` (via `signCode()` at `package/src/macos/installer.ts:293`). Two lists:
8484
- **With entitlements** (`entitlements.plist`): per-arch `deno`, `dart-sass/src/dart`, `esbuild`, `pandoc`, `typst`, optional `typst-gather`, optional `deno_dom/libplugin.dylib`. Entitlements grant JIT (`com.apple.security.cs.allow-jit`) and unsigned-executable-memory (`com.apple.security.cs.allow-unsigned-executable-memory`) — required by Deno's V8 and by the other JIT-using tools.
85-
- **Without entitlements**: shell wrappers and JS bundle — `quarto`, `quarto.js`, `dart-sass/sass`.
85+
- **Without entitlements**: shell wrappers, JS bundle, and the Dart Sass AOT snapshot — `quarto`, `quarto.js`, `dart-sass/sass`, `dart-sass/src/sass.snapshot`. The snapshot is a Mach-O loaded by the (entitled) `dart` VM, not its own process, so it needs a valid signature + secure timestamp but no entitlements. See the notarization-Invalid troubleshooting below for why it must be signed.
8686
- `codesign` invocation: `-s <id> --timestamp --options=runtime --force --deep --verbose=4` (`+ --entitlements <plist>` for the first list). `--options=runtime` opts into Hardened Runtime, a notarization prerequisite.
8787
3. **Build the tarball** (`quarto-<v>-macos.tar.gz`) from the signed `pkg-working` tree. The tarball ships unwrapped binaries with the signatures embedded — no `.pkg`, no notarization, but each Mach-O is signed individually.
8888
4. **Build the core `.pkg`** with `pkgbuild` (identifier `org.rstudio.quarto`), then sign it via `productsign --sign <installer-id> --timestamp` (function `signPackage()` at `package/src/macos/installer.ts:282`). Renames signed output back over the original. Note: the `pkgbuild` invocation passes `--install-location` twice (`/Library/Quarto` then `/Applications/quarto`); CLI argv semantics mean the later flag wins, so the effective install location is `/Applications/quarto`. The earlier `/Library/Quarto` value is dead and should be cleaned up next time the file is touched.
@@ -121,6 +121,21 @@ If the Account Holder is unreachable (departed staff, etc.):
121121

122122
Determining which type the current signing team uses: inspect any past released `.pkg` from a non-CI environment (CI logs mask the Team ID portion of cert Common Names). On a Mac, `pkgutil --check-signature quarto-X.Y.Z-macos.pkg` shows the full chain. On Linux/WSL, the `.pkg` is a xar archive — its TOC contains base64-encoded X.509 certs that can be extracted with Python's stdlib and parsed with `openssl x509`. The cert subject's `O = ...` field is a company name for org enrollment, a person name for individual enrollment.
123123

124+
### Troubleshooting: notarization Invalid "Archive contains critical validation errors"
125+
126+
Symptom: `notarizeAndWait()` throws `Notarization was not accepted (status: Invalid)`, and the dumped `xcrun notarytool log <id>` lists per-file `issues` such as:
127+
128+
```text
129+
"The binary is not signed with a valid Developer ID certificate."
130+
"The signature does not include a secure timestamp."
131+
```
132+
133+
Root cause: the named file is a Mach-O binary inside the bundle that no `codesign` call covered. Every native Mach-O in the payload must be individually signed (with `--timestamp`); Apple's notary rejects the whole submission if any is unsigned. Fix: add the file to `signWithEntitlements` (if it JITs / is a directly-executed hardened-runtime process) or `signWithoutEntitlements` (data-like code loaded by another signed process) in `package/src/macos/installer.ts`, then re-run the release build.
134+
135+
This is easy to miss because a bundled binary can *change format across a version bump*. Concrete case (#14664): Dart Sass `src/sass.snapshot` shipped as an **ELF** blob through 1.87.0 — Apple's notary ignored it (not native code), so it was never signed and notarization passed for years. Dart Sass 1.101.0 switched the macOS AOT snapshot to a native **Mach-O** (`cf fa ed fe`), which the notary then required signed; the never-signed snapshot flipped the result to Invalid. Diagnosing this depended on the terminal-status check in `notarizeAndWait()` — `notarytool submit --wait` had exited 0, so an exit-code-only check would (and previously did) mask the rejection and proceed to `stapler staple`, whose downstream "CloudKit … Record not found" error is a misleading symptom of the missing ticket, not a propagation race.
136+
137+
Prevention: dispatch `create-release.yml` (publish-release=false) on the branch before merging any bundled-binary bump — see `dev-docs/upgrade-dependencies.md`.
138+
124139
### Historical: `waitForNotaryStatus` / `altool` (candidate for removal)
125140

126141
`waitForNotaryStatus()` at `package/src/macos/installer.ts:360` is **dead code** — defined and never called. Kept here so we have enough context to decide whether to delete it next time the file is touched.

‎package/src/macos/installer.ts‎

Lines changed: 95 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ import { Configuration } from "../common/config.ts";
1818
import { runCmd } from "../util/cmd.ts";
1919
import { getEnv } from "../util/utils.ts";
2020
import { makeTarball } from "../util/tar.ts";
21+
import { withRetry } from "../../../src/core/retry.ts";
22+
import { parseNotarizationResult } from "./notarize-status.ts";
2123

2224
// Packaging specific configuration
2325
// (Some things are global others may be platform specific)
@@ -102,6 +104,11 @@ export async function makeInstallerMac(config: Configuration) {
102104
"dart",
103105
));
104106
signWithoutEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "dart-sass", "sass"));
107+
// Dart Sass 1.101.0 ships src/sass.snapshot as a Mach-O binary that
108+
// Apple's notary service requires signed; the dart VM that loads it
109+
// carries the runtime entitlements, so the snapshot itself needs only a
110+
// valid Developer ID signature + secure timestamp.
111+
signWithoutEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "dart-sass", "src", "sass.snapshot"));
105112

106113
signWithEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "esbuild"));
107114
signWithEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "pandoc"));
@@ -253,14 +260,13 @@ export async function makeInstallerMac(config: Configuration) {
253260
const password = getEnv("QUARTO_APPLE_CONNECT_PW", "");
254261
const teamId = getEnv("QUARTO_APPLE_CONNECT_TEAMID", "");
255262
if (username.length > 0 && password.length > 0) {
256-
const requestId = await notarizeAndWait(
263+
await notarizeAndWait(
257264
packagePath,
258265
username,
259266
password,
260-
teamId
267+
teamId,
261268
);
262269

263-
264270
// Staple the notary to the package
265271
await stapleNotary(packagePath);
266272

@@ -344,16 +350,66 @@ async function notarizeAndWait(
344350
],
345351
);
346352

347-
if (result.status.success) {
348-
const match = result.stdout.match(/id: (.*)/);
349-
if (match) {
350-
const id = match[1];
351-
return id;
352-
} else {
353-
throw new Error("Notarization Failed to return an Id:\n" + result.stdout);
353+
// Always surface notarytool's own output (submission id + terminal status).
354+
// runCmd only logs stdout at debug level, so without this the Accepted /
355+
// Invalid verdict never appears in CI logs.
356+
info(result.stdout);
357+
358+
if (!result.status.success) {
359+
throw new Error("Notarization Failed\n" + result.stdout + result.stderr);
360+
}
361+
362+
const { id, status } = parseNotarizationResult(result.stdout);
363+
if (!id) {
364+
throw new Error("Notarization Failed to return an Id:\n" + result.stdout);
365+
}
366+
367+
// `notarytool submit --wait` can exit 0 even when the service rejects the
368+
// submission; the trustworthy signal is the terminal status. Only a genuinely
369+
// notarized package should reach the (offline-only) stapling step below.
370+
if (status !== "Accepted") {
371+
await dumpNotarizationLog(id, username, password, teamId);
372+
throw new Error(
373+
`Notarization was not accepted (status: ${status ?? "unknown"}) for ${input}`,
374+
);
375+
}
376+
377+
return id;
378+
}
379+
380+
// Best-effort dump of the full developer log for a submission, so a rejected
381+
// notarization shows its reasons in CI. Never masks the caller's own error.
382+
async function dumpNotarizationLog(
383+
id: string,
384+
username: string,
385+
password: string,
386+
teamId: string,
387+
) {
388+
try {
389+
const log = await runCmd(
390+
"xcrun",
391+
[
392+
"notarytool",
393+
"log",
394+
id,
395+
"--apple-id",
396+
username,
397+
"--password",
398+
password,
399+
"--team-id",
400+
teamId,
401+
],
402+
);
403+
info(log.stdout);
404+
if (log.stderr) {
405+
error(log.stderr);
354406
}
355-
} else {
356-
throw new Error("Notarization Failed\n" + result.stderr);
407+
} catch (err) {
408+
warning(
409+
`Could not fetch notarization log for ${id}: ${
410+
err instanceof Error ? err.message : String(err)
411+
}`,
412+
);
357413
}
358414
}
359415

@@ -419,9 +475,32 @@ async function waitForNotaryStatus(
419475
return notaryResult;
420476
}
421477

478+
// Once notarizeAndWait confirms the submission is Accepted, the ticket exists
479+
// and stapling normally succeeds immediately. Rarely, the ticket can take a
480+
// moment to become queryable in Apple's CloudKit-backed notary database, so an
481+
// immediate staple fails with:
482+
// CloudKit query for ... failed due to "Record not found".
483+
// Could not find base64 encoded ticket in response for ...
484+
// A short bounded retry absorbs that rare timing gap. Anything else - or a
485+
// persistent "Record not found" that outlasts the retry - is a real problem and
486+
// must fail the build (a rejected submission never reaches here; its status is
487+
// verified in notarizeAndWait).
488+
const isCloudKitPropagation = (err: Error) =>
489+
err.message.includes("CloudKit") &&
490+
err.message.includes("Record not found");
491+
422492
async function stapleNotary(input: string) {
423-
await runCmd(
424-
"xcrun",
425-
["stapler", "staple", input],
426-
);
493+
await withRetry(async () => {
494+
await runCmd(
495+
"xcrun",
496+
["stapler", "staple", input],
497+
);
498+
}, {
499+
// withRetry's `attempts` counts retries after the first call (see
500+
// src/core/retry.ts), so 3 here yields 4 total staple invocations.
501+
attempts: 3,
502+
minWait: 20000,
503+
maxWait: 30000,
504+
retry: isCloudKitPropagation,
505+
});
427506
}
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
/*
2+
* notarize-status.ts
3+
*
4+
* Copyright (C) 2026 Posit Software, PBC
5+
*
6+
*/
7+
8+
export interface NotarizationResult {
9+
id?: string;
10+
status?: string;
11+
}
12+
13+
// Parses the stdout of `xcrun notarytool submit --wait`.
14+
//
15+
// The output carries several `id:` lines (Submission ID, uploaded-file id,
16+
// processing-complete id) and a terminal `status:` line inside the final
17+
// "Processing complete" block. We take the first `id:` (the submission id,
18+
// which is what `xcrun notarytool log <id>` expects) and the terminal
19+
// `status:` (Accepted / Invalid / Rejected).
20+
export function parseNotarizationResult(stdout: string): NotarizationResult {
21+
// First `id:` line is the submission id (also what `notarytool log` wants).
22+
const idMatch = stdout.match(/^\s*id:\s*(\S+)/m);
23+
// Terminal `status:` line lives in the final "Processing complete" block.
24+
// (`Current status:` lines are streamed progress, not the verdict, and don't
25+
// match this anchored `status:` prefix.) Take the last one to be safe.
26+
const statusMatches = [...stdout.matchAll(/^\s*status:\s*(\S+)/gm)];
27+
const lastStatus = statusMatches.at(-1);
28+
return {
29+
id: idMatch?.[1],
30+
status: lastStatus?.[1],
31+
};
32+
}

‎package/src/util/cmd.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,14 @@ export async function runCmd(
3737
const code = output.code;
3838
info(`Status ${code}`);
3939
if (code !== 0) {
40+
// Some tools (e.g. xcrun stapler) write their actual failure reason to
41+
// stdout rather than stderr; without this, that reason is only visible
42+
// at debug level and failures show an empty error message.
43+
error(stdout);
4044
error(stderr);
41-
throw Error(`Command ${[runCmd, ...args]} failed.`);
45+
throw Error(
46+
`Command ${[runCmd, ...args]} failed.\n${stdout}\n${stderr}`,
47+
);
4248
}
4349

4450
return {
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
/*
2+
* macos-notarize-status.test.ts
3+
*
4+
* Tests the parser that extracts the submission id and terminal status from
5+
* `xcrun notarytool submit --wait` output. macOS release builds gate stapling
6+
* (and, since PR #14718, whether the release proceeds at all) on the notary
7+
* service returning `status: Accepted` -- a check the build previously skipped,
8+
* trusting only the process exit code.
9+
*
10+
* Inputs are hand-written samples of notarytool's own output (the contract
11+
* Apple hands us), so a change to Quarto's own build logic is what these guard,
12+
* not Apple's service.
13+
*
14+
* Copyright (C) 2026 Posit Software, PBC
15+
*/
16+
17+
import { unitTest } from "../test.ts";
18+
import { assertEquals } from "testing/asserts";
19+
import { parseNotarizationResult } from "../../package/src/macos/notarize-status.ts";
20+
21+
const acceptedOutput = `Conducting pre-submission checks for quarto-1.10.17-macos.pkg and initiating connection to the Apple notary service...
22+
Submission ID received
23+
id: 2efe2717-52ef-43a5-96dc-0797e4ca1041
24+
Successfully uploaded file
25+
id: 2efe2717-52ef-43a5-96dc-0797e4ca1041
26+
path: /Users/runner/work/quarto-cli/quarto-1.10.17-macos.pkg
27+
Waiting for processing to complete.
28+
Current status: Accepted..........
29+
Processing complete
30+
id: 2efe2717-52ef-43a5-96dc-0797e4ca1041
31+
status: Accepted
32+
`;
33+
34+
const invalidOutput = `Conducting pre-submission checks for quarto-1.10.17-macos.pkg and initiating connection to the Apple notary service...
35+
Submission ID received
36+
id: 9a9b9c9d-1111-2222-3333-444455556666
37+
Successfully uploaded file
38+
id: 9a9b9c9d-1111-2222-3333-444455556666
39+
path: /Users/runner/work/quarto-cli/quarto-1.10.17-macos.pkg
40+
Waiting for processing to complete.
41+
Current status: Invalid..........
42+
Processing complete
43+
id: 9a9b9c9d-1111-2222-3333-444455556666
44+
status: Invalid
45+
`;
46+
47+
unitTest("parseNotarizationResult - accepted submission", async () => {
48+
const result = parseNotarizationResult(acceptedOutput);
49+
assertEquals(result.id, "2efe2717-52ef-43a5-96dc-0797e4ca1041");
50+
assertEquals(result.status, "Accepted");
51+
});
52+
53+
unitTest("parseNotarizationResult - invalid submission", async () => {
54+
const result = parseNotarizationResult(invalidOutput);
55+
assertEquals(result.id, "9a9b9c9d-1111-2222-3333-444455556666");
56+
assertEquals(result.status, "Invalid");
57+
});
58+
59+
unitTest("parseNotarizationResult - no id or status when absent", async () => {
60+
const result = parseNotarizationResult("some unrelated output\n");
61+
assertEquals(result.id, undefined);
62+
assertEquals(result.status, undefined);
63+
});

0 commit comments

Comments
 (0)