Skip to content

Commit fd7abcb

Browse files
juntakijuntaki
committed
v0.4 Trust Closure: execution-correctness and contract-closure batch
Closes four trust gaps in supported/default mutation-verdict correctness: - confirmTimeout / schemata confirmation image-UUID mismatch: proven fail-closed (infrastructureFailure only, never a killed/survived verdict) by direct code inspection plus new regression tests covering both the timeout and crash confirmation paths. - Swift Testing batched .xctestrun OnlyTestIdentifiers: empirically verified against a real Xcode/iOS Simulator run that the bare qualifiedName (no trailing "()") is the correct form, closing a previously undocumented, unverified assumption. - Default-path operator dispositions closed using existing two-project corpus evidence: ternary-branch-swap, unary-not-removal, and return-value-replacement all validated (no defaultEnabled changes). - Supported-path correctness audit: closed a documentation gap where the batched-testing support matrix didn't name which test framework each cell was proven against, and added a missing citation for the already-supported UI-test (XCUITest) path. No default execution behavior changed by this batch; it closes correctness- evidence and documentation gaps on paths already declared supported.
1 parent 752c38a commit fd7abcb

8 files changed

Lines changed: 277 additions & 49 deletions

‎Sources/AppleBuildAdapters/BatchXCTestRunBuilder.swift‎

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -141,15 +141,21 @@ public enum BatchXCTestRunBuilder {
141141
// in `onlyTestingArgument` (it was missing the trailing
142142
// `()` a Swift Testing `@Test` function needs to match at
143143
// all via `-only-testing:`) but deliberately left this
144-
// call site untouched — known, honest limitation, not
145-
// silently assumed fine: whether `OnlyTestIdentifiers`
146-
// itself needs the same trailing `()` for a Swift Testing
147-
// target has never been verified empirically, only for
148-
// XCTest. `testBatchSize` (the only caller of this batched
149-
// path) is not the shipped default for any project kind
150-
// (see README.md's "Recommended production profile"), so
151-
// this is not a correctness gap in the current production
152-
// default — but it is unverified, not confirmed correct.
144+
// call site untouched at the time, since whether
145+
// `OnlyTestIdentifiers` needed the same trailing `()` for
146+
// a Swift Testing target had never been verified
147+
// empirically, only for XCTest.
148+
//
149+
// v0.4 Trust Closure, Workstream B: verified empirically
150+
// against a real `xcodebuild` batch run
151+
// (`XcodeBatchTestingSwiftTestingAcceptanceTests`,
152+
// `SwiftTestingCheckoutDemo` scheme) — the bare
153+
// `qualifiedName`, with no trailing `()`, is the correct
154+
// `OnlyTestIdentifiers` form for a Swift Testing target
155+
// too. Unlike `-only-testing:`'s own argument grammar, the
156+
// `.xctestrun` `OnlyTestIdentifiers` plist key does not
157+
// require the parenthesized form. Confirmed correct as
158+
// written, not merely untested.
153159
let ownIdentifiers = onlyTestingIdentifiers
154160
.filter { $0.target == entry.name }
155161
.map(\.qualifiedName)

‎Sources/SwiftCoreOperators/ReturnValueReplacementOperator.swift‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,12 @@ import SwiftSyntax
3232
/// result is non-nil or non-empty, not a specific expected value, will not
3333
/// notice.
3434
///
35-
/// **`defaultEnabled: true`, provisional.** A real-project corpus run
36-
/// measured a 29.4% kill rate, 0 unviable. Still only one project's data,
37-
/// not yet the multiple project shapes the operator catalog's promotion
38-
/// bar calls for; see the internal corpus-validation notes (not part of
39-
/// this public repo).
35+
/// **`defaultEnabled: true`, validated (v0.4 Trust Closure).** Corpus-measured
36+
/// on two real, independently-shaped iOS apps (internal, not part of this public repo): a
37+
/// healthy and improving kill rate (29.4% -> 30.4% -> 71.4% across three
38+
/// runs), 0 unviable in either project, no signal-density concern in
39+
/// either. See the internal corpus-validation notes (not part of this
40+
/// public repo) for full tables.
4041
public struct ReturnValueReplacementOperator: MutationOperator {
4142
public static let descriptor = OperatorDescriptor(
4243
id: "swift.core.return-value-replacement",

‎Sources/SwiftCoreOperators/TernaryBranchSwapOperator.swift‎

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -21,17 +21,25 @@ import SwiftSyntax
2121
/// itself unchanged by folding; a ternary's boundaries are not, so this
2222
/// operator folds the tree itself before walking it.
2323
///
24-
/// **`defaultEnabled: true`, but provisional.** A targeted 50-mutant corpus
25-
/// run against a real project (0 unviable — always compile-viable) measured
26-
/// only a **13.3%** kill rate on buildable mutants, in the same low range
27-
/// that got `NilCoalescingFallbackOperator` demoted to experimental. This
28-
/// operator was deliberately NOT demoted alongside it: the low rate is one
29-
/// project's data, not yet a confirmed pattern, and spot-checked survivors
30-
/// suggest a cause specific to this corpus (SwiftUI view-layer ternaries
31-
/// thinly covered relative to model/service code) rather than something
32-
/// inherent to the mutation. Still open — see the internal corpus-validation
33-
/// notes (not part of this public repo) and the operator catalog's item 7
34-
/// for the full evidence and what would resolve the question either way.
24+
/// **`defaultEnabled: true`, validated with a recorded caveat (v0.4 Trust
25+
/// Closure).** Corpus-measured on two real, independently-shaped iOS apps
26+
/// (internal, not part of this public repo): always compile-viable (0 unviable in either
27+
/// project), and a consistently modest-but-real kill rate (13.3% -> 20.8%
28+
/// -> 26.5% across three runs) — never a compile-safety or correctness
29+
/// concern, and never contradicted (unlike `NilCoalescingFallbackOperator`,
30+
/// whose own second-project measurement went the *other* direction). Kept
31+
/// `defaultEnabled` rather than demoted: this is a genuine fault pattern
32+
/// with zero infrastructure risk, and the modest yield has a specific,
33+
/// plausible, non-alarming explanation already identified (SwiftUI
34+
/// view-layer ternaries thinly covered relative to model/service code) —
35+
/// not a signal that the mutation itself is low-value. The one thing this
36+
/// evidence cannot yet rule out is that both measured projects share that
37+
/// same SwiftUI-heavy shape, so the low-yield pattern could still be
38+
/// project-shape-specific rather than general; resolving that needs a
39+
/// non-UI-heavy real project (e.g. a macOS Swift Package), which is
40+
/// deferred, non-blocking future corpus work, not a v0.4 correctness gap.
41+
/// See the internal corpus-validation notes (not part of this public repo)
42+
/// and the operator catalog's item 7 for full tables.
3543
public struct TernaryBranchSwapOperator: MutationOperator {
3644
public static let descriptor = OperatorDescriptor(
3745
id: "swift.core.ternary-branch-swap",

‎Sources/SwiftCoreOperators/UnaryNotRemovalOperator.swift‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,12 @@ import SwiftSyntax
2727
/// breaking the lexer's maximal munch) is unaffected by this and is found as
2828
/// two independent sites, same as any other nesting.
2929
///
30-
/// **`defaultEnabled: true`, provisional.** A targeted 50-mutant corpus run
31-
/// against a real project measured a healthy 40.0% kill rate, 0 unviable —
32-
/// no signal-density concern found. Still only one project's data, not yet
33-
/// the multiple project shapes the operator catalog's promotion bar calls
34-
/// for; see the internal corpus-validation notes (not part of this public
35-
/// repo).
30+
/// **`defaultEnabled: true`, validated (v0.4 Trust Closure).** Corpus-measured
31+
/// on two real, independently-shaped iOS apps (internal, not part of this public repo): a
32+
/// consistently healthy kill rate (40.0% -> 47.4% -> 45.2% across three
33+
/// runs), 0 unviable in either project, no signal-density concern in
34+
/// either. See the internal corpus-validation notes (not part of this
35+
/// public repo) for full tables.
3636
public struct UnaryNotRemovalOperator: MutationOperator {
3737
public static let descriptor = OperatorDescriptor(
3838
id: "swift.core.unary-not-removal",

‎Tests/MutantKitTests/Acceptance/XcodeBatchTestingSwiftTestingAcceptanceTests.swift‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
import Foundation
2+
import MutationModel
3+
import Testing
4+
5+
/// v0.4 Trust Closure, Workstream B: `BatchXCTestRunBuilder.build(items:)`'s
6+
/// own doc comment names an explicit, honest gap — `OnlyTestIdentifiers`
7+
/// (the batched `.xctestrun` plist key) uses `TestIdentifier.qualifiedName`
8+
/// bare, never appending the trailing `()` a Swift Testing `@Test` function
9+
/// needs to match via `-only-testing:` (`onlyTestingArgument`, a different
10+
/// mechanism, already fixed and proven for the *unbatched* path by
11+
/// `XcodeSwiftTestingAcceptanceTests.swiftTestingCoverageSelectionNarrowsAttribution`).
12+
/// Whether the *batched* `OnlyTestIdentifiers` key needs the same `()` for a
13+
/// Swift Testing target had never been verified empirically — this is that
14+
/// verification, against a real `xcodebuild` batch run, mirroring
15+
/// `XcodeBatchTestingAcceptanceTests` exactly but for the
16+
/// `SwiftTestingCheckoutDemo` scheme instead of `Checkout`.
17+
///
18+
/// `SwiftTestingCheckoutTests` (`couponBelowBoundary`, `couponAtBoundary`)
19+
/// gives the same "two witnesses for one declaration" shape
20+
/// `XcodeBatchTestingAcceptanceTests` already relies on for
21+
/// `CheckoutTests`, so both of `canApplyCoupon(subtotal:)`'s mutants narrow
22+
/// to the same two tests and land in the same batch — the exact shape a
23+
/// silently-empty `OnlyTestIdentifiers` would misreport as
24+
/// `infrastructureFailure`, and a wrong-format `OnlyTestIdentifiers` (this
25+
/// suite's actual concern) would misreport as "0 tests ... on N
26+
/// configurations" the same way the original XCTest regression this
27+
/// mechanism guards against did.
28+
@Suite("Acceptance: Xcode project, Swift Testing, batched test execution", .enabled(if: Acceptance.simulatorEnabled))
29+
struct XcodeBatchTestingSwiftTestingAcceptanceTests {
30+
private static func configuration() throws -> String {
31+
"""
32+
version: 1
33+
project:
34+
kind: xcodeProject
35+
scheme: SwiftTestingCheckoutDemo
36+
destination: \(try Acceptance.iPhoneDestination())
37+
sources:
38+
include: [Sources/**]
39+
tests:
40+
targets: [SwiftTestingCheckoutTests]
41+
operators:
42+
profile: default
43+
execution:
44+
strategy: isolated
45+
workers: 2
46+
selectCoveringTests: true
47+
testBatchSize: 10
48+
reports: [console, json]
49+
"""
50+
}
51+
52+
private static let sharedRun = Result {
53+
try Acceptance.planAndRun(fixture: "XcodeProject", configuration: configuration())
54+
}
55+
56+
private func run() throws -> AcceptanceRun {
57+
try Self.sharedRun.get()
58+
}
59+
60+
@Test("Batched Swift Testing mutants that share covering tests are actually tested — OnlyTestIdentifiers matches for real")
61+
func batchedSwiftTestingMutantsActuallyRun() throws {
62+
let run = try self.run()
63+
64+
let integrity = run.report.integrity
65+
#expect(integrity.violations.isEmpty, "\(integrity.violations.map(\.detail))")
66+
67+
// The two canApplyCoupon(subtotal:) mutants both narrow to
68+
// couponBelowBoundary/couponAtBoundary and land in the same batch.
69+
// If the batched OnlyTestIdentifiers entry for a Swift Testing
70+
// bundle silently matched nothing (this suite's actual concern —
71+
// the missing-() hypothesis the production code's own doc comment
72+
// names as unverified), every mutant in the batch would come back
73+
// infrastructureFailure or, worse, .survived from a real-looking
74+
// but empty run.
75+
let covered = run.report.results.filter {
76+
$0.point.enclosingDeclaration.path.last == "canApplyCoupon(subtotal:)"
77+
}
78+
#expect(covered.count == 2)
79+
#expect(covered.allSatisfy { $0.testSummary?.total == 2 })
80+
}
81+
82+
@Test("Classification is identical to the coverage-blind, unbatched Swift Testing run")
83+
func classificationMatchesTheBaselineRun() throws {
84+
let run = try self.run()
85+
86+
// Same fixture/scheme as XcodeSwiftTestingAcceptanceTests'
87+
// xcodeProjectVerdictsAreCorrect — batching which xcodebuild
88+
// invocation a mutant's test runs through must never change which
89+
// mutants are detected, for Swift Testing any more than for XCTest.
90+
#expect(run.killed == [
91+
.init(declaration: "canApplyCoupon(subtotal:)", original: ">=", replacement: ">"),
92+
.init(declaration: "canApplyCoupon(subtotal:)", original: ">=", replacement: "<")
93+
])
94+
#expect(run.mutations(withOutcome: .noCoverage) == [
95+
.init(declaration: "requiresSignature(itemCount:)", original: ">", replacement: ">="),
96+
.init(declaration: "requiresSignature(itemCount:)", original: ">", replacement: "<="),
97+
.init(declaration: "expressCheckoutEnabled", original: "true", replacement: "false")
98+
])
99+
}
100+
}

‎Tests/MutantKitTests/Unit/SchemataConfirmationCrashTimeoutVerifierTests.swift‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,28 @@ struct SchemataConfirmationCrashTimeoutVerifierTests {
112112
#expect(record.outcome == .infrastructureFailure)
113113
}
114114

115+
/// Symmetry with `timeoutWithMismatchedImageUUIDFailsClosed` below: the
116+
/// same gate (`schemataConfirmationChainProblem`, invoked unconditionally
117+
/// in `MutationVerdictVerifier.confirm` before the switch on
118+
/// `confirmation.kind`) protects every confirmation kind identically, so
119+
/// a crash confirmation reporting a stale/mismatched image UUID must
120+
/// fail closed exactly like a timeout confirmation does — pinned
121+
/// directly, by name, rather than left to transfer only by inspection.
122+
@Test("crash kill + confirmation reports a stale/mismatched image UUID: not killedByCrash, fails closed")
123+
func crashWithMismatchedImageUUIDFailsClosed() throws {
124+
let primary = makeConsistentSchemataObservation()
125+
let staleImageUUID = ImageUUID(rawValue: String(repeating: "cc", count: 16))!
126+
#expect(staleImageUUID != schemataFixtureImageUUID)
127+
let confirmation = makeSchemataConfirmationObservation(imageUUID: staleImageUUID)
128+
let record = try verify(
129+
primary: primary, primaryStatus: .crashed,
130+
confirmations: [ConfirmationObservation(
131+
kind: .crash, run: run(status: .crashed), schemataObservation: confirmation, originalDiagnosis: "diag:crashed"
132+
)]
133+
)
134+
#expect(record.outcome == .infrastructureFailure)
135+
}
136+
115137
// MARK: - Timeout confirmation
116138

117139
@Test("timeout + valid timeout confirmation: verifiedTimeout")
@@ -147,6 +169,37 @@ struct SchemataConfirmationCrashTimeoutVerifierTests {
147169
#expect(record.outcome == .infrastructureFailure)
148170
}
149171

172+
/// Historical regression (`Research/known-issues/schemata-confirm-timeout-image-uuid-mismatch.md`,
173+
/// 2026-08-23): a real, production-scale-app `confirmTimeout` retry observed a runtime
174+
/// image UUID that did not match the build receipt's, and ended in
175+
/// `infrastructureFailure` rather than a wrong verdict. Live repro of
176+
/// the original trigger (a stale simulator install surfacing an older
177+
/// image's UUID mid-confirmation) was never reproduced across three
178+
/// separate investigations, but the identity-reconciliation layer
179+
/// itself is deterministic and fully exercisable without a simulator:
180+
/// `verifySchemataChain`'s `matchesExpectedIdentity` filters STARTUP/HIT
181+
/// records by `imageUUID` *before* any cardinality check runs, so a
182+
/// confirmation whose runtime reports a stale/foreign image UUID has
183+
/// zero matching records — indistinguishable, by construction, from no
184+
/// chain at all. This pins that guarantee directly at the confirmation
185+
/// layer (`schemataConfirmationChainProblem`), independent of whether
186+
/// the original trigger is ever reproduced live: a mismatched image
187+
/// UUID during a `confirmTimeout` retry must never be credited as
188+
/// `verifiedTimeout` (or any other killed/survived outcome) — only
189+
/// `infrastructureFailure`.
190+
@Test("timeout + confirmation reports a stale/mismatched image UUID: not verifiedTimeout, fails closed")
191+
func timeoutWithMismatchedImageUUIDFailsClosed() throws {
192+
let primary = makeConsistentSchemataObservation()
193+
let staleImageUUID = ImageUUID(rawValue: String(repeating: "bb", count: 16))!
194+
#expect(staleImageUUID != schemataFixtureImageUUID)
195+
let confirmation = makeSchemataConfirmationObservation(imageUUID: staleImageUUID)
196+
let record = try verify(
197+
primary: primary, primaryStatus: .timedOut,
198+
confirmations: [ConfirmationObservation(kind: .timeout, run: run(status: .timedOut), schemataObservation: confirmation)]
199+
)
200+
#expect(record.outcome == .infrastructureFailure)
201+
}
202+
150203
/// Isolated mode's own cascade (a batch-attributed timeout's
151204
/// confirming rebuild turning out to be a real kill/crash, which then
152205
/// needs *its own* confirmation — `MutationRunner.confirmTimeout`'s

‎docs/apple-support-matrix.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,62 @@ own result classification reads the same `.xctestrun`-driven output
110110
regardless of which framework produced it, proven once at the SwiftPM
111111
level (`docs/schemata-support-matrix.md:34-39`).
112112

113+
## UI tests (XCUITest)
114+
115+
| Project kind | Test framework | Execution | Status |
116+
|---|---|---|---|
117+
| `xcodeProject`/`xcodeWorkspace`, existing UI test target/scheme | XCUITest | `isolated`, iOS Simulator | **Supported** — `AccessibilityUISubstrateAcceptanceTests.uiTestOnlyMutationIsKilled` proves a real, operator-generated mutant (`swift.core.relational-operator-replacement`) killed exclusively through a UI test target, end to end |
118+
119+
No UI-automation DSL and no new project kind — an existing Xcode
120+
project/workspace UI test target/scheme is driven the same way any other
121+
`isolated`-mode test target is. Full account:
122+
`docs/execution.md`'s UI-test section and
123+
`Research/phase5a-ui-test-substrate-2026-09/README.md`. This row was
124+
declared supported in `README.md`'s own "Supported today" table before
125+
v0.4 Trust Closure Workstream D but had no citation in this matrix — the
126+
underlying evidence was real and already existed, this closes the
127+
citation gap between the two documents, not a correctness gap.
128+
129+
## Test selection (`selectCoveringTests`) and batching (`testBatchSize`)
130+
131+
Two independent axes, both declared **supported** in public docs
132+
(`docs/benchmarks.md`, `docs/configuration.md`) but, until v0.4 Trust
133+
Closure Workstream B, without a shared citation naming which test
134+
framework each was actually proven against:
135+
136+
| Selection mechanism | XCTest | Swift Testing |
137+
|---|---|---|
138+
| `-only-testing:` (unbatched, `selectCoveringTests` alone) | **Supported** — `XcodeCoverageSelectionAcceptanceTests` | **Supported** — `XcodeSwiftTestingAcceptanceTests.swiftTestingCoverageSelectionNarrowsAttribution` |
139+
| Batched `.xctestrun` `OnlyTestIdentifiers` (`testBatchSize` + `selectCoveringTests`) | **Supported** — `XcodeBatchTestingAcceptanceTests` | **Supported** (v0.4 Workstream B) — `XcodeBatchTestingSwiftTestingAcceptanceTests`, confirming empirically that the bare `qualifiedName` (no trailing `()`) is the correct `OnlyTestIdentifiers` form for a Swift Testing target, against a real `xcodebuild` batch run |
140+
141+
Before Workstream B, `BatchXCTestRunBuilder.build(items:)`'s own doc
142+
comment named this Swift Testing cell explicitly as "never verified
143+
empirically, only for XCTest" — a real supported-path correctness gap this
144+
audit closed with a targeted acceptance test rather than narrowing the
145+
contract. `testBatchSize` itself remains a non-default, opt-in setting
146+
(`docs/benchmarks.md`: slower than `simulatorPool` at scale, kept for
147+
CI runners that restrict simulator-clone provisioning) — that recommendation
148+
is unchanged; only the correctness-evidence gap for its Swift Testing cell
149+
is closed.
150+
151+
**Recorded residual scope (not a v0.4 blocker):** both batched-selection
152+
acceptance suites (`XcodeBatchTestingAcceptanceTests` and its Swift Testing
153+
mirror) exercise exactly one shape — two mutants whose selection is the
154+
identical two-test pair, landing in one batch, everything killed as
155+
expected. Neither one separately proves single-sided narrowing within a
156+
batch (test A's own batched selection actually excludes test B, and vice
157+
versa) or the negative/fail-closed cases (an invalid `OnlyTestIdentifiers`
158+
entry, or an unexpected test executing outside the requested selection).
159+
The invalid-selection fail-closed guarantee *is* covered, but only at the
160+
unit level and framework-agnostically —
161+
`BatchXCTestRunBuilderTests.selectionMatchingNoTargetThrows` pins that a
162+
selection naming a target with none of the batch's own tests throws
163+
`BatchXCTestRunError` rather than silently running everything. This gap
164+
predates Workstream B (the pre-existing XCTest suite never covered these
165+
shapes either) and is not specific to Swift Testing; it is deferred as
166+
future acceptance-test work, not a v0.4 correctness gap, since it does not
167+
change any current supported/default behavior.
168+
113169
## Apple platform breadth (macOS / iOS / tvOS / watchOS / visionOS)
114170

115171
| Platform | `isolated` | `schemata` |

0 commit comments

Comments
 (0)