Expose passive input and capture capabilities across automation surfaces

Author: JE-Chen

CHANGELOG.md

@@ -13,6 +13,13 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's
 
 ## Unreleased
 
+- Add passive input/capture capability snapshots to both SDK namespaces,
+  AC_probe_capabilities, read-only MCP, Builder and Diagnostics Actions.
+  Prerequisites remain unverified; local native state is not refreshed or
+  treated as a permission grant. XWayland scope is explicit, provider
+  refusals cannot imply CLI fallback, and default results omit environment
+  values, executable paths, command arguments and native error strings.
+
 - Diagnostics opens in a localized not-run state. Only its explicit Actions
   command starts the existing active checks; opening or retranslating the tab
   cannot implicitly capture the screen or request portal consent. Existing

Progress.md

@@ -13,8 +13,9 @@
 `TODO` — 真機與付費 API 驗收暫緩，待明確恢復後執行；相關項目保留待驗證。
 
 `WIP` — 改版仍缺以下內容；原生平台與正式環境驗收缺口另列於後。
-計畫 D 仍缺 `wrapper/capabilities.py` 的無副作用公開探測、獨立輸入／擷取診斷與
-SDK／AC／MCP／Diagnostics GUI 介面，以及 portal 關閉／恢復與合成器重啟驗收。
+計畫 D1 仍缺 portal session 關閉／恢復、compositor 重啟的狀態與原生驗收、
+擷取結果與擁有者 context 的接合，以及主動診斷與原生 session 的重試／取消／停止控制。
+原生指標超出 EIS 區域的拒絕仍被誤分類為選用依賴缺失，須停止隱含的 CLI 備援。
 D2 的原生 EI 釋放驗證與 D3 的錄製／全域停止／Windows 原生崩潰證據仍缺，詳見計畫與下方生命週期待辦。
 計畫 E（Android／iOS）、G（MCP 逐步揭露）、H（型別深化與總驗收）尚未開始。F 還缺：
 

README.md

@@ -18,7 +18,7 @@ from JSON files / CLI / servers, and a **GUI tab**. Nothing is GUI-only.
 
 - **One API, seven platforms.** `wrapper/platform_wrapper.py` picks the backend at import
   time; your script does not change between Windows, macOS, X11, and Wayland.
-- **Scriptable without Python.** 809 `AC_*` commands cover the whole feature set, so a
+- **Scriptable without Python.** 810 `AC_*` commands cover the whole feature set, so a
   JSON file can do anything the library can — including loops, branches, try/catch,
   macros, and variables.
 - **Headless by default.** `import je_auto_control` never loads Qt. The GUI is an
@@ -233,7 +233,7 @@ desktop app; tab commands live in the window's **Actions** menu.
 | Natural-language planner | `plan_actions`, `run_from_description` | `AC_llm_plan` | LLM Planner |
 | Computer-use agent | `AgentLoop`, `run_agent` | `AC_run_agent` | Computer Use |
 | Record & replay | `record`, `stop_record` | `AC_record`, `AC_stop_record` | Record |
-| JSON scripting | `execute_action`, `execute_files` | all 809 commands | Script, Script Builder |
+| JSON scripting | `execute_action`, `execute_files` | all 810 commands | Script, Script Builder |
 | Variables & flow control | `execute_action_with_vars` | `AC_set_var`, `AC_loop`, `AC_for_each`, `AC_try`, `AC_retry` | Variables |
 | Data-driven runs | — | `AC_for_each_row` (CSV / JSON / SQLite / Excel) | Data Sources |
 | Assertions | `assert_text`, `assert_image` | `AC_assert_text` + 20 more | Assertions |
@@ -338,7 +338,7 @@ under a relative name are written below that directory.
 
 | Surface | Start it with | Notes |
 |---|---|---|
-| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | 709 tools for Claude Desktop / Claude Code / custom tool loops: the full `ac_*` surface plus short model-friendly aliases for common GUI actions. Speaks the stateless MCP 2026-07-28 beside the `initialize`-based revisions. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. |
+| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | 710 tools for Claude Desktop / Claude Code / custom tool loops: the full `ac_*` surface plus short model-friendly aliases for common GUI actions. Speaks the stateless MCP 2026-07-28 beside the `initialize`-based revisions. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. |
 | **REST API** | `je_auto_control start-rest` | Bearer token, per-IP rate limit + lockout, SQLite audit hook, `/metrics`, `/openapi.json`, `/docs` Swagger UI, `/dashboard`. |
 | **TCP socket server** | `je_auto_control start-server` | Newline-framed JSON action lists. Binds `127.0.0.1` by default. |
 | **pytest plugin** | installed automatically | Fixtures plus a Gherkin step library for pytest-bdd / behave. |
@@ -473,12 +473,45 @@ returning the blank XWayland root. The `screen_capture` check in
 `je_auto_control.api.run_diagnostics()` (and the GUI Diagnostics tab) reports
 which tier is in use.
 
+Passive **input/capture capabilities** use `BackendContext`, `CapabilityStatus`,
+`CapabilitySnapshot` and `probe_capabilities` from `je_auto_control.api` (also
+exported at the package root). Call
+`probe_capabilities(BackendContext.current()).to_dict()`, use no-argument
+`AC_probe_capabilities` / read-only `ac_probe_capabilities`, or choose
+**Diagnostics → Actions → Inspect input and capture capabilities**. Builder
+provides the same no-argument command. `python -m examples.capabilities` prints
+live prerequisites and a deterministic XWayland scope example without desktop operations.
+
+Input describes key/pointer events; the existing wtype text path is separate.
+The probe reads environment/PATH and nonblocking cached native state; it never
+loads optional native libraries, connects, polls events, captures or requests
+permission. Installed tools or a bus address mean `unverified`, not readiness.
+Cached libei readiness requires a completed handshake and active keyboard,
+absolute pointer, button and scroll families. A partial grant remains
+unverified; paused families and removed required devices report paused/revoked.
+The eight states are `available`, `unverified`, `needs_permission`,
+`needs_dependency`, `unsupported`, `paused`, `revoked` and `failed`.
+`available` is only the last local observation and may be stale; every real
+operation rechecks authority. XWayland always has `desktop_wide=False` and cannot
+cover native Wayland windows. Other scope flags remain subject to granted
+regions/permissions and do not verify every monitor.
+
+Contexts detach environment/executable facts and may carry an owner's inert
+input/capture observations. They grant no authority. Default live results omit
+environment values, command arguments, executable paths and native error text;
+AC/MCP accept no supplied observations. Explicit CLI configuration remains
+distinct from automatic fallback after observed optional native absence;
+permission/pause/revocation/failure metadata cannot imply CLI fallback.
+Capture helper and portal selection are diagnosed independently. Rendering and
+language changes reuse the snapshot without probing. Native portal restore,
+compositor restart, capture-result/context integration and cancelable active
+diagnostics remain in `Progress.md`.
+
 Opening Diagnostics displays a localized not-run state and performs no checks.
 Choose **Actions → Run diagnostics** to start the existing active checks, which
 can capture the screen or require portal consent. Changing the display language
 does not start checks. The explicit `run_diagnostics`, `AC_diagnose` and MCP
-diagnostic calls retain their active behavior; passive capability probing is
-still pending in `Progress.md`.
+diagnostic calls retain their active behavior. Passive capability inspection is a separate action below.
 
 Abandoned screenshot portal requests attempt `Request.Close` on the originating
 bus before disconnect, using a separate 0.25-second D-Bus I/O budget. A returned
@@ -523,7 +556,7 @@ ignore synthetic input, and fall back silently when the driver is absent.
 
 | Resource | What's in it |
 |---|---|
-| [`examples/`](examples/) | 41 self-contained scripts: screenshot + click, OCR, scheduler, remote desktop, agent loop, observability, recording, variables, hotkeys, triggers, reports, MCP, REST, secrets, plugins, computer use, Wayland, cross-host DAGs, chat-ops, pytest/BDD, anchor locators, action journals. |
+| [`examples/`](examples/) | 42 self-contained scripts: screenshot + click, OCR, scheduler, remote desktop, agent loop, observability, recording, variables, hotkeys, triggers, reports, MCP, REST, secrets, plugins, computer use, Wayland, cross-host DAGs, chat-ops, pytest/BDD, anchor locators, action journals. |
 | [Read the Docs](https://autocontrol.readthedocs.io/en/latest/) | Full API reference, English and 中文. |
 | [architecture_explore.md](architecture_explore.md) | Every module's responsibility, layer by layer. |
 | [docs/CAPABILITY_MATRIX.md](docs/CAPABILITY_MATRIX.md) | Capability × platform matrix. |

README/README_zh-CN.md

@@ -19,7 +19,7 @@
 
 - **一套 API，七个平台。** `wrapper/platform_wrapper.py` 在导入时挑选后端；同一份脚本在
   Windows、macOS、X11 与 Wayland 上都不需要改写。
-- **不写 Python 也能脚本化。** 809 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库
+- **不写 Python 也能脚本化。** 810 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库
   能做的任何事——包含循环、分支、try/catch、宏与变量。
 - **默认无头运行。** `import je_auto_control` 绝不会加载 Qt。GUI 是可选包，包在同一个无头内核之外。
 - **四种定位方式。** 模板匹配、OCR、无障碍树、视觉语言模型——可通过锚点定位器与自愈回退串接组合。
@@ -221,7 +221,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()"
 | 自然语言规划 | `plan_actions`、`run_from_description` | `AC_llm_plan` | LLM Planner |
 | Computer-use agent | `AgentLoop`、`run_agent` | `AC_run_agent` | Computer Use |
 | 录制与回放 | `record`、`stop_record` | `AC_record`、`AC_stop_record` | Record |
-| JSON 脚本 | `execute_action`、`execute_files` | 全部 809 个命令 | Script、Script Builder |
+| JSON 脚本 | `execute_action`、`execute_files` | 全部 810 个命令 | Script、Script Builder |
 | 变量与流程控制 | `execute_action_with_vars` | `AC_set_var`、`AC_loop`、`AC_for_each`、`AC_try`、`AC_retry` | Variables |
 | 数据驱动执行 | — | `AC_for_each_row`（CSV／JSON／SQLite／Excel） | Data Sources |
 | 断言 | `assert_text`、`assert_image` | `AC_assert_text` 等 21 个 | Assertions |
@@ -319,7 +319,7 @@ je_auto_control version
 
 | 接口 | 启动方式 | 说明 |
 |---|---|---|
-| **MCP 服务器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 709 个工具，供 Claude Desktop／Claude Code／自定义 tool loop 使用：完整的 `ac_*` 工具接口，以及常用 GUI 操作的短名称 alias。除了以 `initialize` 握手的各版协议，也支持无状态的 MCP 2026-07-28。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 |
+| **MCP 服务器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 710 个工具，供 Claude Desktop／Claude Code／自定义 tool loop 使用：完整的 `ac_*` 工具接口，以及常用 GUI 操作的短名称 alias。除了以 `initialize` 握手的各版协议，也支持无状态的 MCP 2026-07-28。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 |
 | **REST API** | `je_auto_control start-rest` | Bearer token、按 IP 限流与锁定、SQLite 审计 hook、`/metrics`、`/openapi.json`、`/docs` Swagger UI、`/dashboard`。 |
 | **TCP socket 服务器** | `je_auto_control start-server` | 以换行分隔的 JSON 动作列表。默认绑定 `127.0.0.1`。 |
 | **pytest 插件** | 安装后自动生效 | 提供 fixture 与供 pytest-bdd／behave 使用的 Gherkin step library。 |
@@ -438,10 +438,31 @@ Wayland 的屏幕截取需要合成器对应的工具，因为没有单一工具
 `je_auto_control.api.run_diagnostics()`（以及 GUI 的 Diagnostics 标签页）的 `screen_capture`
 检查会报告当前使用的是哪一层。
 
+**输入／截取能力**提供 `je_auto_control.api`（也由包根目录导出）的
+`BackendContext`、`CapabilityStatus`、`CapabilitySnapshot` 与 `probe_capabilities`。
+调用 `probe_capabilities(BackendContext.current()).to_dict()`、无参数的
+`AC_probe_capabilities`／只读 `ac_probe_capabilities`，或选择
+**Diagnostics → Actions → 查看输入与截取能力**；Builder 提供相同无参数命令。
+`python -m examples.capabilities` 显示本机前置条件与确定性的 XWayland 范围示例，不操作桌面。
+
+输入项描述按键／指针事件；现有 wtype 文字路径独立。探测只读环境、PATH 与不等待锁的现有原生状态，
+不加载可选原生库、不连接、不读取事件、不截取或请求授权。找到工具或 bus 地址只代表
+`unverified`，不等于可用。缓存 libei 须完成握手，且键盘、绝对指针、按钮与滚轮事件种类都在运行才报告可用；
+部分授权保持未验证，事件种类暂停或必要设备移除则报告暂停／撤销。八种状态为 `available`、`unverified`、`needs_permission`、
+`needs_dependency`、`unsupported`、`paused`、`revoked`、`failed`。
+`available` 仅是最近的本机观察，可能过时；实际操作仍重新验证权限。XWayland 一律
+`desktop_wide=False`，无法覆盖原生 Wayland 窗口；其他范围标记仍受授权区域限制，不代表已验证每个屏幕。
+
+Context 会复制环境／工具数据，可由拥有者提供无原生对象的输入／截取观察，但不授予权限。
+默认本机结果不含环境值、命令参数、可执行文件路径或原生错误文字；AC／MCP 不接受外部提供观察。
+显式选择 CLI 与观察到可选原生依赖缺失后的自动备用分开；拒绝、暂停、撤销与失败不表示改走 CLI。
+截取工具／portal 选择独立诊断，重绘与切换语言只使用现有快照。原生 portal 恢复、compositor 重启、
+截取结果与 context 的接合，以及可取消的主动诊断仍列于 `Progress.md`。
+
 打开 Diagnostics 只显示已翻译的「尚未运行」状态，不执行检查。
 选择 **Actions → 运行诊断** 才会启动现有主动检查，可能截取屏幕或需要 portal 同意；
 切换语言不启动检查。显式的 `run_diagnostics`、`AC_diagnose` 与 MCP 诊断调用保持主动行为；
-无副作用的能力探测仍列于 `Progress.md`。
+无副作用的能力查看使用下方独立操作。
 
 未完成的 screenshot portal 请求会在断开原始 bus 前尝试 `Request.Close`，
 使用独立的 0.25 秒 D-Bus I/O 预算。服务器返回替代 handle 时会跟随该 handle；
@@ -478,7 +499,7 @@ Windows、macOS（pyobjc）与 X11（含 XWayland）；纯 Wayland 会话的协
 
 | 资源 | 内容 |
 |---|---|
-| [`examples/`](../examples/) | 41 个自包含脚本：截图点击、OCR、调度器、远程桌面、agent loop、可观测性、录制、变量、热键、触发器、报表、MCP、REST、密钥、插件、computer use、Wayland、跨主机 DAG、chat-ops、pytest/BDD、锚点定位、动作日志。 |
+| [`examples/`](../examples/) | 42 个自包含脚本：截图点击、OCR、调度器、远程桌面、agent loop、可观测性、录制、变量、热键、触发器、报表、MCP、REST、密钥、插件、computer use、Wayland、跨主机 DAG、chat-ops、pytest/BDD、锚点定位、动作日志。 |
 | [Read the Docs](https://autocontrol.readthedocs.io/en/latest/) | 完整 API 参考，含英文与中文。 |
 | [architecture_explore.md](../architecture_explore.md) | 逐层记录每个模块的职责。 |
 | [docs/CAPABILITY_MATRIX.md](../docs/CAPABILITY_MATRIX.md) | 能力 × 平台对照矩阵。 |