Skip to content

Commit f6a26b9

Browse files
JasonVMoCopilot
andauthored
Desktop Driver: list published accessibility actions (macOS AX Select/Toggle) (#4426)
* desktop-driver: list published accessibility actions Add an accessibilityActions element property that returns the names of the actions an element publishes in the native accessibility tree, gated by the accessibilityActions feature. The macOS helper reads AXUIElementCopyActionNames and reduces AppKit's custom-action strings (Name, Target and Selector fields) to the authored name so no process-local handle crosses the boundary. React Native macOS Fabric lists Toggle and Select by name; Paper lists only actions that declare a label, by that label. Hosts without the feature reject with unsupported operation. Listing only. Invoking custom actions remains out of scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * desktop-driver: number the accessibility action evidence C-A11Y-007 C-A11Y-006 is already taken by the react-native-macos state projection claim on user/jasonvmo/validate-sriramas-tests, which this branch does not contain. Using the next free ID avoids two different claims sharing one ID when the branches merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent dc47e15 commit f6a26b9

17 files changed

Lines changed: 185 additions & 3 deletions

File tree

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@fluentui-react-native/desktop-driver': patch
3+
---
4+
5+
List the accessibility actions an element publishes through a new `accessibilityActions` element property, gated by the `accessibilityActions` entry in `furn:features`. The macOS provider reports authored custom actions such as `Toggle` and `Select` by name without exposing provider handles, while Windows and Win32 keep returning `unsupported operation`. The property only lists actions; it does not invoke them.

‎packages/agentic/components/references/remaining-work.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,9 @@ is documented in the owning reference, component contract, or the repository foc
2525
- **Open gates:**
2626
- rerun the current `desktop-focus` cases on Windows Fabric and Win32 with real input and zero skips;
2727
- invoke UIA `TogglePattern.Toggle` and `SelectionItemPattern.Select`, and macOS AX custom actions, natively rather
28-
than inferring them from clicks or state reads;
28+
than inferring them from clicks or state reads (the Desktop Driver now lists macOS AX custom actions through
29+
`getProperty('accessibilityActions')`: Fabric publishes `Toggle` and `Select`, while Paper publishes nothing until
30+
the declaration carries a `label`, C-A11Y-007; invocation is still open);
2931
- qualify Narrator on both Windows endpoints and VoiceOver on macOS;
3032
- resolve or explicitly accept the native state announcement gaps: Win32 editors report UIA `IsEnabled` when
3133
noneditable (C-FOCUS-015), and RNmacOS Fabric does not project disabled or selected state without the

‎packages/agentic/desktop-driver/native/PROTOCOL.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,17 @@ destroyed. Helpers without the feature reject the command as
181181
`unsupported-operation`; Node never requests the log unless the feature is
182182
negotiated.
183183

184+
Helpers that advertise the `accessibilityActions` feature also accept an
185+
`accessibilityActions` command with the helper-private `elementId` and return
186+
`{ "actions": ["<name>", ...] }`: the names of the actions the platform
187+
accessibility tree publishes for the element, in provider order. Names are
188+
authored text only; provider handles, targets, and selectors never cross FDR1.
189+
The command is read-only and follows the liveness rules of other element
190+
queries. An element that publishes no actions returns an empty list. Helpers
191+
without the feature reject the command as `unsupported-operation`, which Node
192+
reports as `unsupported operation` rather than an empty list. The feature is
193+
additive and does not change the wire version.
194+
184195
### `cancel` and `cancelled`
185196

186197
When a command's `AbortSignal` fires, Node sends:

‎packages/agentic/desktop-driver/native/macos/README.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,7 @@ The AX provider supports:
227227
- bounded tree/source generation;
228228
- snapshots with explicit supported/unsupported state;
229229
- active element, focus, hit testing, clear, and accessibility activation;
230+
- the action names an element publishes (`accessibilityActions`), including authored custom actions;
230231
- primary and secondary application windows.
231232

232233
AppKit omits `AXEnabled` from elements that have no disabled state, and React
@@ -236,6 +237,17 @@ list does not contain `AXEnabled`, matching how assistive technologies treat
236237
it. An element that lists `AXEnabled` without a readable value still reports
237238
an unsupported enabled state. Fabric views always publish the attribute.
238239

240+
The `accessibilityActions` command reads `AXUIElementCopyActionNames` for a live
241+
element; an element whose actions are unsupported yields an empty list. AppKit
242+
lists an `NSAccessibilityCustomAction` as
243+
`Name:<name>\nTarget:<address>\nSelector:<selector>`, so the provider returns
244+
only `<name>` and no process address or selector leaves the helper. Standard
245+
actions pass through unchanged. React Native macOS Fabric publishes each declared
246+
action by `name` (`Toggle`, `Select`), whereas Paper publishes only actions that
247+
declare a `label` and names them by it (`C-A11Y-007` in
248+
`accessibility.action-transport-evidence`). The command lists actions only and
249+
never performs one.
250+
239251
Quartz Event Services supplies physical pointer, keyboard, text, and wheel
240252
input. Physical input requires Accessibility/PostEvent authority for the
241253
responsible process and an unlocked interactive Aqua session. Missing authority
@@ -385,7 +397,7 @@ FURN_DESKTOP_DRIVER_MACOS_SIGNING_IDENTITY="<identity>" \
385397

386398
The Swift self-test covers framing, JSON, input ledger and lock recovery,
387399
capture scaling, snapshot normalization, AX diagnostics, CoreGraphics fallback
388-
classification, permission and login-session diagnostics, and hello identity. The Node native
400+
classification, custom AX action name parsing, permission and login-session diagnostics, and hello identity. The Node native
389401
contract additionally builds, verifies, reuses, and directly resolves the app;
390402
starts the actual long-lived process; and exercises malformed-frame recovery.
391403

‎packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Accessibility.swift‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -332,6 +332,14 @@ final class AccessibilityEngine {
332332
}
333333
}
334334

335+
func accessibilityActions(_ elementID: String, token: CancellationToken) throws -> [String] {
336+
try requireAccessibility("Accessibility action inspection")
337+
try token.throwIfCancelled()
338+
let record = try requireRecord(elementID)
339+
try validateLive(record)
340+
return try copyAXActionNames(record.element).map(accessibilityActionName)
341+
}
342+
335343
func tryClearValue(_ elementID: String) throws -> Bool {
336344
let record = try requireRecord(elementID)
337345
try validateLive(record)

‎packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Driver.swift‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,9 @@ final class Driver {
122122
let elementID = try requireString(params, "elementId")
123123
let window = try windowForElement(elementID)
124124
return CommandResult(result: try accessibility.snapshot(elementID: elementID, window: window, token: token).json())
125+
case "accessibilityActions":
126+
let actions = try accessibility.accessibilityActions(try requireString(params, "elementId"), token: token)
127+
return CommandResult(result: ["actions": actions])
125128
case "activeElement":
126129
let window = try applications.requireWindow(try requireString(params, "windowId"))
127130
let snapshot = try accessibility.activeElement(window: window, token: token)
@@ -231,6 +234,7 @@ final class Driver {
231234
return [
232235
"endpoint": optionalString(params, "endpoint") ?? "macos",
233236
"features": [
237+
"accessibilityActions": trusted,
234238
"accessibilityClick": trusted,
235239
"elementScreenshot": screenshot,
236240
"focus": trusted,

‎packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Host.swift‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ func createHello() -> JSONObject {
1414
"minor": 0,
1515
],
1616
"features": [
17+
"accessibilityActions",
1718
"accessibilityClick",
1819
"activeElement",
1920
"attach",

‎packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/SelfTest.swift‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ private final class SelfTestRunner {
2727
testAXWindowDiagnostics()
2828
testCoreGraphicsWindowSelection()
2929
testReactNativeAccessibilityFallbacks()
30+
testAccessibilityActionNames()
3031
testPermissionDiagnostic()
3132
testLoginSessionState()
3233
testWindowEventMonitor()
@@ -661,6 +662,29 @@ private final class SelfTestRunner {
661662
"hello carries the generated build identity"
662663
)
663664
check((hello["protocol"] as? JSONObject)?["major"] as? Int == 1, "hello reports the wire protocol")
665+
check(
666+
(hello["features"] as? [String])?.contains("accessibilityActions") == true,
667+
"hello advertises accessibility action inspection"
668+
)
669+
}
670+
671+
private func testAccessibilityActionNames() {
672+
check(
673+
accessibilityActionName("Name:Toggle\nTarget:0x6190039a0880\nSelector:didActivateAccessibilityCustomAction:") == "Toggle"
674+
&& accessibilityActionName("Name:Select\nTarget:0x61900399fe80\nSelector:didActivateAccessibilityCustomAction:") == "Select",
675+
"a custom action is listed by its authored name without the provider target or selector"
676+
)
677+
check(
678+
accessibilityActionName("Name:Open in tab\nTarget:0x1\nSelector:didActivateAccessibilityCustomAction:") == "Open in tab",
679+
"a custom action name keeps its spaces"
680+
)
681+
check(
682+
accessibilityActionName("Name:a\nTarget:b\nTarget:0x1\nSelector:s:") == "a\nTarget:b",
683+
"an authored name containing the target marker is not truncated"
684+
)
685+
check(accessibilityActionName("Name:Toggle") == "Toggle", "a custom action without provider fields keeps its name")
686+
check(accessibilityActionName("AXPress") == "AXPress", "a standard accessibility action is listed unchanged")
687+
check(accessibilityActionName("") == "", "an empty action name stays empty")
664688
}
665689
}
666690

‎packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Support.swift‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -435,6 +435,20 @@ func copyAXActionNames(_ element: AXUIElement) throws -> [String] {
435435
return names as? [String] ?? []
436436
}
437437

438+
// AppKit lists an NSAccessibilityCustomAction as "Name:<name>\nTarget:<address>\nSelector:<selector>". The target
439+
// and selector are process-local details, so only the authored name is exposed. Standard AX actions are unchanged.
440+
func accessibilityActionName(_ raw: String) -> String {
441+
let prefix = "Name:"
442+
guard raw.hasPrefix(prefix) else {
443+
return raw
444+
}
445+
let remainder = raw.dropFirst(prefix.count)
446+
guard let target = remainder.range(of: "\nTarget:", options: .backwards) else {
447+
return String(remainder)
448+
}
449+
return String(remainder[..<target.lowerBound])
450+
}
451+
438452
func axPoint(_ value: CFTypeRef?) -> CGPoint? {
439453
guard let value, CFGetTypeID(value) == AXValueGetTypeID() else {
440454
return nil

‎packages/agentic/desktop-driver/references/protocol.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,19 @@ and Win32 read UI Automation's `HelpText` without provider defaults, preserving
134134
an explicitly empty string. Missing or unsupported help text, including on the
135135
current macOS provider, returns `unsupported operation` instead of an empty value.
136136

137+
The `accessibilityActions` property (also readable as an attribute) lists the
138+
action names the native accessibility tree publishes for an element, in provider
139+
order, when `furn:features` advertises `accessibilityActions`. The macOS provider
140+
reads `AXUIElementCopyActionNames` and lists an authored custom action, such as
141+
`Toggle` or `Select`, by name without any provider handle, alongside any standard
142+
actions the element publishes. An element that publishes none returns `[]`; hosts
143+
without the feature, including Windows and Win32, return `unsupported operation`
144+
rather than `[]`. The property only lists actions; it does not invoke them.
145+
What an element publishes depends on the renderer: React Native macOS Fabric
146+
publishes every declared `accessibilityActions` entry by `name`, while Paper
147+
publishes only entries that declare a `label` and lists them by that label
148+
(`C-A11Y-007` in `accessibility.action-transport-evidence`).
149+
137150
## Errors
138151

139152
| Condition | WebDriver error |

0 commit comments

Comments
 (0)