Awesome GUI Agent Paper List

A curated list of 766 research papers on GUI agents — models, frameworks, benchmarks, datasets, and more — spanning topics like GUI grounding, planning, memory, safety, and reinforcement learning.

🌐 Read this list on the web

The website is the recommended way to read this list:

→ osu-nlp-group.github.io/GUI-Agents-Paper-List

What it adds over the raw markdown:

• Full-text search across titles, authors, institutions, TLDRs, and keywords
• Multi-axis filtering — environment, keyword (AND/OR), author, institution, year, venue — with shareable URLs
• Per-paper detail pages with the full TLDR, all keywords, and related papers
• One-click BibTeX copy — LaTeX-paste-ready (@misc for arXiv-only, @inproceedings / @article for venue papers). Auto-generated, so please verify before citing.
• Interactive stats — quarterly publication trend by environment, top keywords / institutions / authors / venues
• Warm-paper light theme + dark theme, keyboard shortcuts (/, j/k, Esc, ?), no tracking

The structured store papers.yaml (and adjacent.yaml) is the source of truth — everything on the website and the README is generated from it.

Browse by Environment

🌐 Web (313) · 🖥️ Desktop (198) · 📱 Mobile (236) · 🖼️ General GUI (154)

Browse by Keyword

benchmark (233) · dataset (109) · framework (73) · reinforcement learning (72) · GUI grounding (66)
model (64) · safety (45) · security (44) · OSWorld (37) · WebArena (26)
long-horizon tasks (24) · world model (23) · training-free (21) · memory (20) · reward model (19)
prompt injection (17) · AndroidWorld (17) · GRPO (15) · Mind2Web (14) · survey (11)

Browse by Author

Wei Liu (23) · Jian Luan (23) · Pengzhi Gao (17) · Yu Su (15) · Huan Sun (15)
Zhuosheng Zhang (14) · Graham Neubig (14) · Zhengxi Lu (13) · Shuyan Zhou (13) · Mike Zheng Shou (12)
Qiushi Sun (11) · Fei Tang (11) · Tao Yu (11) · Boyuan Zheng (11) · Zichen Ding (10)
Yongliang Shen (10) · Jie Tang (10) · Tianbao Xie (10) · Yuanchun Li (10) · Kevin Qinghong Lin (10)

Contributing

We welcome contributions from the community!

• Missing a paper? Open an issue with the paper title, link, and any relevant details — we'll add it.
• Want to add papers yourself? Edit papers.yaml, run bash scripts/update_repo.sh, then submit the regenerated diff. See CLAUDE.md for the YAML schema and local update workflow.
• Spotted an error? Feel free to open an issue or PR to correct any paper metadata (authors, dates, institutions, etc.).

Recent Papers (from most recent to oldest)

This README shows the 500 most recent papers. See papers.yaml for the full structured source — including BibTeX, OpenReview / publisher / homepage / code / dataset links, and the bibtex_confirmed flag. For non-canonical adjacent papers see adjacent.yaml.

• Adaptive-Group Process Rewards for GUI Agents

  • Zirui Shang, Yang Liu, Xin Shu, Zhi Gao, Xinxiao Wu, Lifeng Fan
  • 🏛️ Institutions: BIT, BIGAI, Wuhan University, Shenzhen MSU-BIT University
  • 📅 Date: October 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [process rewards], [reinforcement learning], [credit assignment], [GUI agent], [PRM]
  • 📖 TLDR: Introduces Adaptive-Group Process Rewards to address sparse outcome rewards in GUI agent reinforcement learning. By grouping correlated interaction steps and computing adaptive process rewards on intermediate sub-goals, the method improves credit assignment and sample efficiency across desktop and web navigation benchmarks.

• Interaction-as-a-Judge

  • Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica, Anastasios Nikolas Angelopoulos, Ameet Talwalkar, Chris Donahue, Cho-Jui Hsieh
  • 🏛️ Institutions: CMU, UC Berkeley, UCLA
  • 📅 Date: October 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [Interaction-as-a-Judge], [computer-use agents], [evaluation], [interactive applications], [LMSYS]
  • 📖 TLDR: Proposes Interaction-as-a-Judge, an evaluation paradigm where computer-use agents actively interact with web applications, desktop software, and games to evaluate interactive functionality. Releases 6,858 application-trace pairs and 222 human-verified requirement benchmarks, demonstrating improved alignment with human quality assessments over static code-based evaluation.

• Learning How to Remember: Memory Modality Selection for GUI Agents

  • Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang, Jaewoo Kang
  • 🏛️ Institutions: Korea University, Myongji University, AIGEN Sciences
  • 📅 Date: October 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [Mobile], [Desktop]
  • 🔑 Key: [memory modality], [SwitchMem], [GUI agent], [multimodal memory], [OSWorld], [Mind2Web]
  • 📖 TLDR: Investigates memory representations in GUI agents and discovers that optimal memory modality (textual summary, action sequence, or screenshot embeddings) varies dynamically across task stages. Introduces SwitchMem, a policy that dynamically selects the memory modality for retrieved experience, boosting task success on OSWorld and Mind2Web.

• MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search

  • Minghua Chen, Xingming Liao, Weiqiang Zhong, Zhuowei Wang
  • 🏛️ Institutions: Guangdong University of Technology
  • 📅 Date: October 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [MHVF], [GUI grounding], [multi-hypothesis search], [verification], [training-free], [web agent]
  • 📖 TLDR: Proposes Multi-Hypothesis Verification and Filtering (MHVF), a training-free inference framework that generates diverse candidate click points and filters them through bidirectional verification between instruction intent and visual context. MHVF achieves substantial grounding accuracy gains on WebVoyager and ScreenSpot without additional model fine-tuning.

• Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agents

  • Yuchen Lu, Julian McAuley, Lei Zhang
  • 🏛️ Institutions: UC San Diego
  • 📅 Date: October 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [CarryOn], [context management], [object-oriented context], [long-horizon], [OSWorld], [BrowseComp-Plus]
  • 📖 TLDR: Proposes CarryOn, an object-oriented context runtime that organizes agent continuation around reusable task objects rather than raw interaction history. On OSWorld 2.0 and BrowseComp-Plus, CarryOn improves success rates by 5.8% and 7.0% while slashing token usage by 51.9% and 22.9%.

• Towards Safe Autonomy for Computer Use

  • Dominik Seip, Bartol Markovinović, Matthias Hein
  • 🏛️ Institutions: University of Tübingen, Tübingen AI Center
  • 📅 Date: October 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [safe autonomy], [computer use], [indirect prompt injection], [information flow], [security]
  • 📖 TLDR: Investigates safety architectures for computer-use agents subjected to indirect prompt injection. Analyzes the failure modes of safety classifiers and proposes capability-bounded execution policies with information-flow tracking, ensuring that untrusted web and application inputs cannot trigger unauthorized system actions.

• BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser Views

  • Yuetian Chen, Kaiyuan Zhang, Yuntao Du, Thomas Li, Mark Tenenholtz, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li
  • 🏛️ Institutions: Perplexity, Purdue University, Rutgers University
  • 📅 Date: October 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [BrowseSafe++], [benchmark], [prompt injection], [browser agent], [security], [detection]
  • 📖 TLDR: Introduces BrowseSafe++, an expanded benchmark of 38,675 webpage pairs evaluating prompt injection detection across diverse web contexts, multilingual content, and rendered browser views. Assesses commercial and open detector models, highlighting critical detection gaps when injections exploit visual DOM structures.

• HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks

  • Zijian Cai, Haowei Liu, Kailai Chen, Lin Peng, Cong Wan, Ziqi Zhang, Chunfeng Yuan, Bing Li, Weiming Hu, Haiyang Xu
  • 🏛️ Institutions: CASIA, UCAS, Alibaba Group, Northwestern University, Xi’an Jiaotong University, PeopleAI Inc., ShanghaiTech University
  • 📅 Date: October 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [HyWe], [hybrid web tasks], [web navigation], [web search], [unified agent], [Mind2Web]
  • 📖 TLDR: Proposes HyWe, a unified framework integrating broad information retrieval and deep interactive page navigation within a single agent trajectory. Equipped with mode-switching policies and specialized tool harnesses, HyWe substantially improves success on complex hybrid web tasks compared to isolated search or navigation agents.

• Learn from Move: the Next Step for GUI Agents

  • Jiahao Lu, Yize Shen, Chuanyuan Tan, Mianqiu Huang, Shicheng Fang, Xinchi Chen, Yining Zheng, Yuxin Wang, Xipeng Qiu
  • 🏛️ Institutions: Fudan, Shanghai Innovation Institute, Soochow University
  • 📅 Date: October 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [Learn from Move], [exploration], [GUI agents], [OpenMOSS], [interactive discovery]
  • 📖 TLDR: Focuses on enabling GUI agents to autonomously explore and learn unfamiliar software interfaces through interactive trial and feedback. Introduces an exploration-driven learning protocol and benchmark where agents discover hidden menu actions and state semantics, improving generalization to unseen desktop applications.

• PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI Agents

  • Zian Yuan, Wei Li
  • 🏛️ Institutions: University of Sydney
  • 📅 Date: October 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [Desktop], [Mobile]
  • 🔑 Key: [PrivacyGuard], [privacy], [PII sanitization], [cloud GUI agent], [security]
  • 📖 TLDR: Presents PrivacyGuard, a client-side privacy architecture for cloud-backed GUI agents that replaces personally identifiable information (PII) in instructions and screenshots with source-linked synthetic tokens. Using locally resolvable references, the agent completes tasks on remote cloud planners without leaking sensitive personal data.

• Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

  • Jingchao Wang, Dingjiang Huang
  • 🏛️ Institutions: East China Normal University
  • 📅 Date: October 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [Mobile]
  • 🔑 Key: [Seg2Click], [GUI grounding], [action masks], [segmentation], [visual grounding]
  • 📖 TLDR: Reformulates GUI visual grounding as executable-region prediction, training models with bounding-box supervision to output pixel-level action masks rather than single-point coordinates. Seg2Click avoids visual-token grid misalignment and achieves higher localization accuracy on fine-grained and dense interface targets.

• The Agent Clicks Approve: Injected Page Text Makes Browser Agents Authorize Access the Task Never Named

  • Md Imran Hossen
  • 🏛️ Institutions: University of Tennessee at Chattanooga
  • 📅 Date: October 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [indirect prompt injection], [browser agent], [confused deputy], [security], [authorization]
  • 📖 TLDR: Demonstrates a confused-deputy attack against browser agents acting in authenticated sessions, where adversarial text on a visited page steers the agent into approving third-party OAuth access or registering malicious webhook endpoints. Testing across commercial and open-source browser agents reveals high vulnerability rates to unprompted privilege escalation.

• A Baseline Assessment for the Agentic Web Technology Stack

  • Vagner Figueredo de Santana, Diogo Cortiz
  • 🏛️ Institutions: NIC.br, PUC-SP
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [MASTIF], [benchmark], [agentic web], [Mind2Web], [evaluation], [protocol comparison]
  • 📖 TLDR: Introduces MASTIF, an evaluation framework assessing combinations of 3 LLMs, 3 communication protocols, and 4 agent implementation frameworks across 300 Mind2Web benchmark tasks. Provides systematic baseline metrics on task completion, execution time, and communication verbosity across alternative agentic web stacks.

• CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving

  • Shangquan Sun, Ruoyu Chen, Zhirui Liu, Xiang Liu, Chenghao Qian, Wenqi Ren, Xiaochun Cao
  • 🏛️ Institutions: NTU, UCAS, Sun Yat-sen University, Beihang University, University of Leeds
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [CaptchaAgent], [CAPTCHA], [visual reasoning], [browser interaction], [grounding]
  • 📖 TLDR: Presents CaptchaAgent, an agent framework for reasoning-intensive CAPTCHAs combining count-committed visual perception with verified browser interaction. Evaluated across diverse modern CAPTCHA suites, CaptchaAgent improves Pass@1 by 21.3-25.8 percentage points over standard VLM baselines.

• ConClick: Element-Wise Contrastive Learning Improves GUI Grounding

  • Ruotian Zhang, Yuhui Zhang, Junzhe Huang, Yu Xin, Hehe Fan, Xiaoxiao Sun
  • 🏛️ Institutions: Independent Researcher
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Web]
  • 🔑 Key: [ConClick], [ReClick], [GUI grounding], [contrastive learning], [ScreenSpot-Pro], [OSWorld-G]
  • 📖 TLDR: Analyzes GUI grounding error patterns and identifies that incorrect clicks predominantly land on plausible distractor elements rather than background. Proposes ConClick, a contrastive learning framework trained on 4,292 target-distractor pairs, improving Qwen3-VL-8B grounding accuracy from 54.7% to 66.0% on ScreenSpot-Pro and from 63.3% to 70.2% on OSWorld-G.

• Latent World Action Model for GUI Agents

  • Seungjun Yu, Hojun Choi, Seojeong Park, Jaeyo Shin, Hyunjung Shim
  • 🏛️ Institutions: KAIST
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [world action model], [latent world model], [GUI agent], [screen transition], [OSWorld]
  • 📖 TLDR: Proposes Latent World Action Model (LWAM) for GUI agents, shifting screen transition prediction from expensive pixel reconstruction into a compact latent visual space. LWAM enables efficient forward rollouts and action evaluation, achieving 4.8x faster inference and improved action selection accuracy on OSWorld and AndroidWorld.

• OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools

  • Chenyang Yan, Yingying Zhang, Kanzhi Cheng, Qiushi Sun, Zhengyuan Pan, Zheng Ma, Hang Yan, Yian Wang, Nuo Chen, Jialin Cao, Xingdong Gong, Wenpo Song, Han Chen, Zichen Ding, Fangzhi Xu, Shujian Huang, Xinyu Dai, Yichen Liu, Tiankuo Yao, Bo Wang, Ben Kao, Jianbing Zhang, Lewei Lu, Dahua Lin
  • 🏛️ Institutions: NJU, SenseTime, NUS, HKU, East China Normal University, Xi’an Jiaotong University
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [OpenMobile-2], [MobileGym++], [mobile agent], [app-native tools], [Android], [benchmark]
  • 📖 TLDR: Presents OpenMobile-2, an open mobile agent trained on MobileGym++, which features 35 realistic simulated commercial applications with interactive environments and app-native tools. OpenMobile-2 achieves frontier performance across diverse mobile benchmarks while releasing open training recipes and environment sandboxes.

• Web$hell: A Unified Interface for Agentic Web Access

  • Sunghwan Kim, Deogyong Kim, Sangam Lee, Donguk Kwon, Dongha Lee
  • 🏛️ Institutions: Yonsei University
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [Web$hell], [web agent], [WebContract], [WebCompiler], [web navigation], [Mind2Web]
  • 📖 TLDR: Introduces Web$hell, a unified web access interface that compiles live webpages into addressable information and action objects via WebContract, maintaining them in WebCache. Web$hell unifies broad information gathering and fine-grained browser interaction, improving task success while lowering interaction costs on Mind2Web and web search benchmarks.

• When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks

  • Yisheng Li, Dongming Han, Hongkun Pan, Bingcheng Mao, Zijian Wang, Yunlong Mao, Wei Tong, Sheng Zhong
  • 🏛️ Institutions: NJU, Hithink Research, ZJU
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [visualization], [charts], [misleading visualizations], [web agent], [visual grounding]
  • 📖 TLDR: Introduces a paired benchmark of 280 executable web agent tasks assessing susceptibility to misleading visual charts and deceptive dashboard designs. Demonstrates that frontier multimodal web agents frequently misinterpret truncated axes and inverted scales, making erroneous navigation and purchasing decisions.

• Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

  • Yuqi Huang, Hui Li, Yifei Cao, Zikun Cai, Tianci Xie, Yangfan Gao, Pinxue Guo, Chaoyun Zhang, Bingxuan Li, Rui Zheng, Hang Yan, Ning Liao
  • 🏛️ Institutions: SJTU, Fudan, NTU, ZJU, Shanghai Qiji Zhifeng Co., Ltd
  • 📅 Date: October 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [Work2Skill], [skill induction], [human demonstrations], [GUI agent], [OSWorld]
  • 📖 TLDR: Introduces Work2Skill, a pipeline for inducing modular and parameterized skills from human GUI demonstration recordings. By parsing GUI action trajectories and abstracting environment-specific element locators, Work2Skill improves execution success on complex desktop workflows in OSWorld while reducing demonstration collection costs.

• AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks

  • Xing Han Lù, Dheeraj Vattikonda, Sina Hajimiri, Fatemeh Pesaran Zadeh, Parishad BehnamGhader, Ghazwa Darwiche, Amirhossein Kazemnejad, Christopher Pal, Alexandre Drouin, Siva Reddy
  • 🏛️ Institutions: ServiceNow Research, McGill University, Mila, Polytechnique Montréal, SNU, Université Laval, ÉTS Montréal
  • 📅 Date: October 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [AgentHorizon], [benchmark], [agentic judge], [long-horizon], [computer-use], [OSWorld], [evaluation]
  • 📖 TLDR: Presents AgentHorizon, a benchmark and diagnostic suite evaluating automated agentic judges on multi-application, long-horizon computer-use trajectories from OSWorld and WorkArena. Shows that current VLM judges suffer significant false-positive rates on complex multi-window workflows due to temporal aggregation errors and misread intermediate states.

• DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents

  • Yudong Bai, Yihong Chen, Quanming Yao, Yaqing Wang
  • 🏛️ Institutions: Tsinghua
  • 📅 Date: October 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [DeltaReplay], [memory reuse], [mobile GUI agent], [transition graph], [AndroidWorld]
  • 📖 TLDR: Proposes DeltaReplay, a memory-augmented framework for mobile GUI agents that represents historical trajectories as paths in a state transition graph and reuses them relative to task differences. Evaluated on AndroidWorld, DeltaReplay avoids rigid replay failures and reduces execution steps by dynamically adapting past trajectories to new target parameters.

• Harness Evolution Hits a Ceiling: When Weight Training Should Begin

  • Yuan Tian, Bing Hu, Hao Wang, Binghang Lu, Fang Wu
  • 🏛️ Institutions: Independent Researcher, Purdue University, UC Berkeley, Stanford
  • 📅 Date: October 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [harness evolution], [weight training], [WebArena-Lite], [failure diagnosis], [LoRA]
  • 📖 TLDR: Analyzes the trade-off between self-evolving agent harnesses and weight training for long-horizon agents, showing that harness evolution fixes process failures (loops, budget exhaustion) while weight training is necessary for content failures. Validates the diagnosis rule across DeepPlanning and WebArena-Lite (+0.09 on 117 unseen tasks).

• Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

  • Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim
  • 🏛️ Institutions: NTU, Institute of Science Tokyo, Sony AI
  • 📅 Date: October 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Web], [Desktop]
  • 🔑 Key: [world model], [verifier], [GUI safety], [transition verification], [Android], [WebArena]
  • 📖 TLDR: Formulates GUI agent safety as transition verification rather than static screen inspection, noting that identical screens can be benign or malicious depending on the preceding action. Introduces a world-model-based transition verifier that predicts expected screen changes and flags anomalous or redirected transitions across Android and web benchmarks.

• SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments

  • Tian Jin, Wenhao Wang, Jing Kang, Haoting Shi, Rui Ye, Yijun Lu, Siheng Chen
  • 🏛️ Institutions: SJTU, ZJU
  • 📅 Date: October 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [SoftwareGym], [environment synthesis], [computer-use agents], [desktop software], [scaling], [RL]
  • 📖 TLDR: Introduces SoftwareGym, a platform that scales computer-use agent training by synthesizing diverse desktop software applications and programmatic verifiers. Pre-training agents with reinforcement learning across 1,200+ synthesized software environments substantially boosts out-of-domain transfer performance on OSWorld.

• Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution

  • Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang, Guanhong Tao
  • 🏛️ Institutions: University of Utah
  • 📅 Date: October 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [adversarial attack], [web agent], [visual grounding], [browser execution], [red-teaming]
  • 📖 TLDR: Studies end-to-end visual adversarial attacks targeting multimodal web agents from visual grounding down to browser execution. Demonstrates that imperceptible adversarial image perturbations on webpages consistently hijack agent attention and cause targeted misclicks across Mind2Web and VisualWebArena.

• Secure-CUA: Controlling Untrusted Influence in Computer-Use Agents

  • Sarthak Choudhary, Mihai Christodorescu, Ashish Hooda, Somesh Jha, Tongxin Li, Damien Octeau
  • 🏛️ Institutions: UW-Madison, Google, Google DeepMind
  • 📅 Date: October 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web], [Mobile]
  • 🔑 Key: [Secure-CUA], [computer-use agents], [security], [untrusted content], [information flow], [guardrails]
  • 📖 TLDR: Introduces Secure-CUA, a security framework that isolates untrusted interface content from computer-use agent planning across desktop, web, and mobile environments. By enforcing information-flow constraints on UI elements and restricting actions derived from untrusted inputs, Secure-CUA prevents indirect prompt injection while preserving legitimate automation capabilities.

• AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Model

  • Sarim Hashmi, Mukul Ranjan, Kshitij Mishra, Mikhail Kuznetsov, Praneeth Vepakomma, Nils Lukas
  • 🏛️ Institutions: MBZUAI, Amazon, MIT
  • 📅 Date: October 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [AdvSim2Real], [web agent], [prompt injection], [web world model], [defense], [security]
  • 📖 TLDR: Proposes AdvSim2Real, an adversarial simulation framework that trains web agents against adaptive indirect prompt injection within a web world model sandbox. Pre-training agents against simulated adversarial web pages significantly reduces injection vulnerability on live web benchmarks without harming benign navigation success.

• Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces

  • Yeji Park, Jaeyun Shim, Taesik Gong
  • 🏛️ Institutions: UNIST
  • 📅 Date: October 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [PAIR], [RePAIR], [benchmark], [mobile GUI agent], [personalization], [cross-user reliability]
  • 📖 TLDR: Introduces PAIR, a benchmark evaluating mobile GUI agents across personalized interface states resulting from diverse user histories and configurations. Proposes RePAIR, a reinforcement learning method with personalization-aware interaction rewards that improves agent robustness against UI layout shifts across distinct user accounts.

• nanoMuse: An Open-Source Personal Agent for Every Device You Own

  • Guangyi Liu, Yong Liu, Jiangning Zhang
  • 🏛️ Institutions: ZJU
  • 📅 Date: October 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Desktop]
  • 🔑 Key: [nanoMuse], [personal agent], [cross-device], [phone GUI], [computer GUI], [open-source]
  • 📖 TLDR: Analyzes the architecture of proprietary cross-device personal assistants and presents nanoMuse, an open-source cross-device agent operating on phone and computer screens. It features a shared cross-device session relay, a safety Sentinel gating screen actions, and transparent file-based provenance memory.

• AMBER: Training Long-Horizon Web Agents through Append-Only Memory

  • Chinmay Savadikar, Zhaoyu Zhang, Mingyu Zhao, Shuang Xie, Han Li, Tianfu Wu, Lingyun Wang
  • 🏛️ Institutions: North Carolina State University, Shopify
  • 📅 Date: October 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [AMBER], [append-only memory], [long-horizon], [web agents], [WebArena], [WorkArena]
  • 📖 TLDR: Proposes AMBER (Append-only Memory Bank for Evidence Retention), an architecture that replaces overwrite memory mechanisms with structured append-only evidence recording for long-horizon web navigation. AMBER prevents critical historical evidence and error feedback from being discarded during multi-step execution, boosting task completion across WebArena and WorkArena.

• CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling

  • Yifan Zhang, Yutong Dai, Viraj Prabhu, Zhiyuan Hu, Ran Xu, Zeyuan Chen
  • 🏛️ Institutions: Salesforce AI Research
  • 📅 Date: October 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [CLIFT], [conformal prediction], [self-verification], [test-time scaling], [WebArena], [Mind2Web]
  • 📖 TLDR: Presents CLIFT, a framework applying conformal prediction to provide calibrated step-level self-verification signals for web agents without relying on costly external LLM judges. Used for RL credit assignment and test-time trajectory search, CLIFT yields consistent success rate gains on WebArena and Mind2Web.

• Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?

  • Aadi Chauhan, Arthur Ilyasov
  • 🏛️ Institutions: Stanford
  • 📅 Date: October 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [GUI grounding], [small models], [action type], [Android in the Wild], [Qwen2-VL]
  • 📖 TLDR: Investigates whether small GUI grounding models (Qwen2-VL-2B) should receive action type information explicitly or infer it jointly with coordinates on Android in the Wild. Finds that conditioning coordinate prediction on the action type through decoupled embeddings improves grounding accuracy while reducing parameter overhead.

• Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training

  • Yongxin Ning, Runliang Niu, Qianli Xing, Zhiyi Duan, Qingzu He, Pan Wang, Qi Wang
  • 🏛️ Institutions: Jilin University, OPPO
  • 📅 Date: October 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [Imagine to Act], [world model], [image editing], [data synthesis], [GUI grounding], [agent training]
  • 📖 TLDR: Proposes Imagine to Act, an image-editing world model framework that synthesizes high-fidelity GUI transition trajectories by generating post-action screenshots from localized element manipulations. Fine-tuning vision-language models on the synthesized trajectories substantially improves multi-step execution and grounding performance across Android and desktop GUI benchmarks.

• PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

  • Dheeraj Varghese, Anna Vettoruzzo, Walter Simoncini, Michelle Lorena Acevedo Callejas, Mohammad Mahdi Derakhshani, Kristof Meding, Joaquin Vanschoren, Cees G. M. Snoek
  • 🏛️ Institutions: University of Amsterdam, Eindhoven University of Technology
  • 📅 Date: October 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [PlaySuite], [benchmark], [interactive visual intelligence], [games], [computer-use agents], [VLM]
  • 📖 TLDR: Introduces PlaySuite, a benchmark evaluating interactive visual agents across 5K+ open-source video games (Pygame, HTML5, Godot, Unity) using a closed-loop execution harness and Video-LLM judge. Evaluation of 14 vision-language models and computer-use agents reveals a substantial perception-action gap, with agents failing consistently on dynamic spatial grounding and real-time self-correction.

• Recurrent Latent Visual Search for GUI Grounding

  • Kaiyu Wu, Beichen Zheng, Weiyao Huang, Keze Wang
  • 🏛️ Institutions: Sun Yat-sen University
  • 📅 Date: October 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [Mobile]
  • 🔑 Key: [GUI grounding], [visual search], [recurrent latent], [VLM], [ScreenSpot]
  • 📖 TLDR: Proposes Recurrent Latent Visual Search (RLVS) for GUI grounding, addressing single-step grounding failures on crowded screenshots by iteratively refining latent target representations across recurrent search steps. RLVS improves grounding accuracy across desktop, web, and mobile interfaces without relying on explicit bounding box prompts.

• TeleTune: Evolving Agent Skills From Offline Telemetry

  • Justin Chih-Yao Chen, Elias Stengel-Eskin, Yan Chen, Pol Llado, Scott Counts, Mohit Bansal, Benjamin Van Durme, Harsh Jhamtani, Gaurav Verma
  • 🏛️ Institutions: UNC, Microsoft, UT Austin
  • 📅 Date: October 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [Desktop]
  • 🔑 Key: [TeleTune], [skill induction], [offline telemetry], [WorkArena], [Mind2Web], [skill library]
  • 📖 TLDR: Introduces TeleTune, a framework that induces reusable textual skill libraries for computer-use agents from offline user telemetry logs without goal labels or replayability. Guided by action-prediction error reductions on logged trajectories, TeleTune achieves 77.1% on WorkArena and 80.6% on Online-Mind2Web, outperforming Agent Workflow Memory by 6.7-7.7% while consuming up to 75x fewer tokens than live optimization.

• MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles

  • Ziyang Long, Xinqi Li, Lujing Xing, Hsin-Jung Yang
  • 🏛️ Institutions: Cedars-Sinai Medical Center, UCLA, TUM, Max Delbrück Center
  • 📅 Date: October 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [MedImageOSWorld], [benchmark], [medical imaging], [CT], [MR], [ultrasound], [console GUI]
  • 📖 TLDR: Introduces MedImageOSWorld, a benchmark evaluating GUI agents on simulated CT, MR, and ultrasound console interfaces across seven capability levels using mouse-and-keyboard interactions. Testing eleven open-weight and two proprietary agents reveals that open-weight models achieve only 3.0-25.0% procedural success and collapse on fine-grained planning and closed-loop acquisition control.

• CUAWright: A Minimal Unified Interface for Digital Agents

  • Yadong Lu, Theodore Lee, Yifei Li, Lawrence Keunho Jang, Tianci Xue, Yu Su, Huan Sun, Ahmed Hassan Awadallah
  • 🏛️ Institutions: MSR, NUS, CMU, OSU
  • 📅 Date: October 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [CUAWright], [harness], [terminal interface], [OSWorld], [Mind2Web], [efficiency]
  • 📖 TLDR: Proposes CUAWright, a minimal (~3K LOC) terminal-based execution harness using bash commands and file-system evolution as the interface for digital agents, arguing that GUI-native harnesses limit programmatic operations. On OSWorld 2.0, CUAWright improves partial reward by 33.2% while reducing token cost by 37.5% over the GPT-5.5 baseline, and achieves +4.7% and +44.0% success rate gains on Online-Mind2Web and Odysseys.

• Control OSWorld: An AI Control Environment for GUI Computer Use Agents

  • Maria Sanchez del Rio, Mikhail Terekhov
  • 🏛️ Institutions: EPFL, MATS
  • 📅 Date: October 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [AI control], [OSWorld], [safety], [monitoring], [computer-use agents]
  • 📖 TLDR: Introduces Control OSWorld, an evaluation pairing 318 tasks from OSWorld with 81 harmful side tasks (such as private file exfiltration) that a computer-use agent must complete without detection. Evaluates monitors that detect malicious GUI actions, finding that full-trajectory monitors achieve 97% recall at a 3% false positive rate, whereas step-level monitors suffer lower recall due to reduced context.

• Securing Computer-Use Agents Against Branch Steering Attacks

  • Giulio Zingrillo, Hanna Foerster, Ilia Shumailov, Yiren Zhao, Robert Mullins
  • 🏛️ Institutions: ETH, University of Cambridge, AI Sequrity Company, Imperial College London
  • 📅 Date: October 02, 2026
  • 📑 Publisher: NeurIPS 2026 Workshop on Agents in the Wild
  • 💻 Env: [Web]
  • 🔑 Key: [STEER-Bench], [COBRA], [benchmark], [security], [indirect prompt injection], [branch steering]
  • 📖 TLDR: Studies branch steering attacks, in which untrusted page content pushes a computer-use agent down a hazardous but pre-approved branch of a Dual-LLM plan without injecting explicit instructions, and introduces STEER-Bench (101 tasks across 9 domains) to measure them. Proposes COBRA, which pairs trusted branching plans with ahead-of-time capability constraints on the parameters and destinations each branch may use; it reduces attack success on STEER-Bench to 0% while retaining 97% benign utility.

• WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites

  • Jiangang Han
  • 🏛️ Institutions: Independent Researcher
  • 📅 Date: October 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [WebFovea], [WebRetriever Challenge], [vision-based agent], [harness], [failure analysis]
  • 📖 TLDR: Technical report of WebFovea, a vision-based web agent that placed 2nd in the WebRetriever Challenge 2026 on live websites. Many failures it observed occur in the harness between the model and the browser (action parsing, action execution, result reporting, and observation) rather than in model reasoning; hardening each stage raised the official hidden-set score from 31.0 to 57.0 with the same model.

• When to Compile a Computer-Use Agent? Measuring Payback and Making Compilation Decisions for Token Efficiency

  • Yulong Ming, Jie Xu, Zihan Wu, Xiaohua Jia
  • 🏛️ Institutions: CityU
  • 📅 Date: October 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [PACE], [efficiency], [workflow compilation], [token cost]
  • 📖 TLDR: Studies when it pays to compile a GUI procedure that an agent executes repeatedly into a program, given uncertain compilation cost and unknown future reuse. Proposes PACE, which combines a measurement protocol for payback counts with an online compilation algorithm that bounds total cost to at most 1+ε times that of running every task with the agent.

• AgentBreaker: Evaluating Context-Aware Indirect Prompt Injection Risks in Modern Web Agents

  • Yongbi Son, Changoo Lee, Dongwon Shin, Byoungyoung Lee, Sanghyun Hong, Sooel Son
  • 🏛️ Institutions: KAIST, SNU, Oregon State University
  • 📅 Date: October 01, 2026
  • 📑 Publisher: ISSTA 2026
  • 💻 Env: [Web]
  • 🔑 Key: [AgentBreaker], [security], [indirect prompt injection], [attack], [online-Mind2Web]
  • 📖 TLDR: AgentBreaker is an indirect prompt injection attack framework that composes adversarial phrases tailored to page-specific context and embeds them as HTML elements that web agents process, inducing actions such as clicking attacker-designated elements, posting attacker-provided text, and disclosing internal agent secrets. It reaches attack success rates of 71.7%-100% against five web agents on 60 Online-Mind2Web pages, and the proposed defenses lower the rate to 1.7%.

• Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories

  • Chengguang Gan, Zimeng He, Yoshihiro Tsujii, Ken-ichiro Kobayashi, Hiroki Itoh, Kotaro Funakoshi
  • 🏛️ Institutions: Techtouch, Institute of Science Tokyo
  • 📅 Date: October 01, 2026
  • 📑 Publisher: NeurIPS 2026 Workshop on Who Verifies the Agents? Toward Reliable Agent Development
  • 💻 Env: [Web]
  • 🔑 Key: [WebArena-Lite], [evaluation], [human review], [trajectory evaluation], [failure analysis], [memory]
  • 📖 TLDR: Audits all 165 WebArena-Lite tasks under six evaluation conditions with human review of outcomes and trajectories. Human review recovers successes the automatic evaluator missed, a failure analysis of 102 trajectories identifies recurring error patterns, and a memory-and-analysis mechanism and guide text improve success.

• AutoGUIWorld: Image Generators as Visual World Models for GUI Agent

  • Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng, Haolin Yang, Geyuan Zhu, Sa Zhu, Jianhuan Zhuo, Qiuyong Xiao, Jianhao Ruan, Yiran Peng, Jiayi Zhang, Tian Ye, Xinlei Yu, Tianwen Jiang, Jihong Zhang, Yuyu Luo
  • 🏛️ Institutions: Tencent Hunyuan
  • 📅 Date: October 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [AutoGUIWorld], [model], [world model], [data generation], [trajectory synthesis], [OSWorld], [ScienceBoard]
  • 📖 TLDR: AutoGUIWorld synthesizes GUI interaction trajectories without running the corresponding software: a planner specifies atomic actions and their intended visual consequences, and an image generator iteratively edits the current screenshot to produce the next observation. The 79,266 step-level samples across Ubuntu, Windows, macOS, and Chrome raise Qwen3.5-35B-A3B from 33.0% to 40.8% on OSWorld and from 14.0% to 32.2% on ScienceBoard.

• DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

  • A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong, Marc Pollefeys, Peter W. J. Staar
  • 🏛️ Institutions: ETH, IBM Research, Microsoft
  • 📅 Date: October 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [DeskForge], [dataset], [model], [GUI grounding], [environment synthesis], [long-horizon tasks]
  • 📖 TLDR: DeskForge is a controllable desktop environment that composes real applications under varied states, window layouts, appearances, and resolutions, and fuses screenshots, accessibility trees, and window geometry into dense element annotations. The resulting DeskForge-1M corpus (1.2M annotated observations) improves four fine-tuned vision-language models on five GUI grounding benchmarks and on the long-horizon WebArena-Infinity and OpenApps tasks.

• GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution

  • Geyi Yang, Zikun Qu, Xiang Li, Zhiyong Wang, Min Zhang, Shipei Zeng, Zhongxiang Dai
  • 🏛️ Institutions: CUHK-Shenzhen, Tianjin University, HIT-Shenzhen, East China Normal University, Shenzhen Research Institute of Big Data
  • 📅 Date: October 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [GUI-HARVEST], [harness optimization], [self-improvement], [OSWorld], [WindowsAgentArena]
  • 📖 TLDR: GUI-HARVEST automatically optimizes the executable harness around a frozen GUI model by aligning actions with before-and-after screenshots, treating repeated runs of a task as one evidence unit, and mapping recurring failure patterns to bounded source-code edits. It yields held-out gains on OSWorld-Verified across six backbones and transfers a frozen harness to WindowsAgentArena.

• Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents

  • Beining Wu, Zihao Ding, Jun Huang
  • 🏛️ Institutions: South Dakota State University
  • 📅 Date: October 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [component routing], [self-improvement], [memory], [experience]
  • 📖 TLDR: Studies where a self-improving GUI agent should put its experience, in the weights or in the prompt context, by splitting trajectories into locators, procedures, state facts, and lessons. Locators and lessons are better placed in the weights and procedures and state facts in the context; a rule based on recurrence and state-conditionality routes components accordingly and beats every whole-trajectory baseline.

• Action Conditioned Bisimulation For GUI Agent Memory

  • Hongbo Zhang, Liuyang Song, Quanquan Li, Daqian Yang, Yan Wen, Zhengtao Yao
  • 🏛️ Institutions: PKU, East China Normal University, USC
  • 📅 Date: September 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [bisimulation], [memory], [state abstraction], [MiniWoB], [training-free]
  • 📖 TLDR: Defines when a web agent's memory should treat two pages as the same state using an action-conditioned bisimulation over the empirical state graph a frozen agent builds as it acts: two states merge only when their shared actions lead to agreeing outcomes. Nothing is trained, and replacing an outcome-value memory's merge rule with it raises success on MiniWoB++.

• ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents

  • Yong Du, Tongbo Chen, Zhengxi Lu, Yizhou Liu, Bofan Chen, Tao Jiang, Wenhao Xu, Yongliang Shen
  • 🏛️ Institutions: ZJU, Ant Group
  • 📅 Date: September 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [ComputerSD], [self-distillation], [online reinforcement learning], [GRPO], [OSWorld]
  • 📖 TLDR: ComputerSD is an online self-distillation method that turns real-time feedback from executed GUI transitions into token-level guidance for computer-use agents. A fine-tuned GUI analyzer supplies guidance and a step-level value score, and training jointly optimizes on-policy self-distillation and trajectory-level GRPO, outperforming outcome-only GRPO on OSWorld-Verified.

• In-Context Teacher–Student Guidance for Open-Weight Browser Agents

  • Wéwerthon Silva Cardoso, Ricardo Marcacini
  • 🏛️ Institutions: University of São Paulo
  • 📅 Date: September 2026
  • 📑 Publisher: KDMiLe 2026
  • 💻 Env: [Web]
  • 🔑 Key: [ICLG], [browser agent], [training-free], [teacher-student guidance], [MiniWoB]
  • 📖 TLDR: In-Context Learning Guidance (ICLG) has a stronger teacher model solve representative browser tasks offline and converts its reasoning into reusable prompt-level guidance for open-weight student agents, without fine-tuning. On MiniWoB++ with GPT-5.1 as the teacher and ten open-weight students, the number of solved student-task pairs rises from 208 to 271.

• Learning Reliable GUI Agents under Imperfect Priors

  • Bo Han, Qianyi Wang, Shuai Liu, Xiong Zifan, Changqiao Wu, Yuanfa Li, Pengzhi Gao, Wei Liu, Jian Luan, Heng Qu, Yunpeng Song, Zhongmin Cai
  • 🏛️ Institutions: XJTU, Xiaomi
  • 📅 Date: September 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [imperfect priors], [exploration], [data synthesis], [robustness], [UI state-transition graph]
  • 📖 TLDR: Argues that GUI agents should act correctly under imperfect, drifting priors rather than pursue perfect app knowledge. Structured exploration builds a UI state-transition graph and synthesizes task-trajectory pairs without human annotation, and a noise-aware training strategy injects five types of realistic errors so the agent learns to assess prior reliability before acting.

• OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software

  • Dingyuan Dai, Heli Qi, Lei Liu, Yinxi Li, Baiding Chen, Zijun Dou, Qingcheng Zeng, Qi Kang, Oliver Sun, Eric Wang, Bo Zhou, Haixin Wang, Yufan Du, Shi Bo, Ruihan Lin, Mengqi Yuan, Dunjie Lu, Steven Dillmann, Yiming Shi, Tina Su, Amy Xin, Minghao Liu, Xi Wang, Xu Huang, Ge Zhang, Pengyu Nie, Zhen Yang, Jie Tang, Juanzi Li, Weihao Xuan, Tianyu Liu
  • 🏛️ Institutions: Tsinghua, UCLA, RIKEN AIP, University of Waterloo, CMU, Yale University, Northwestern University, ZJU, UC Berkeley, University of Illinois Chicago, Boston University, HKU, Stanford, New York University, TokenWave.AI, University of Tokyo
  • 📅 Date: September 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [OSWorld-Science], [benchmark], [scientific software], [OSWorld]
  • 📖 TLDR: OSWorld-Science is a benchmark of 146 expert-proposed tasks for computer-use agents on scientific software, covering workflows such as molecular drawing, pathology image analysis, statistical computing, and physical simulation. Execution-based evaluators inspect application states and generated artifacts with partial credit, and 12 vision-language models still struggle even with a strong harness.

• Talk2Agent: Benchmarking Voice Interfaces for Text Agents

  • Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang
  • 🏛️ Institutions: Tsinghua, University of Cambridge
  • 📅 Date: September 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [Talk2Agent], [benchmark], [voice interface], [speech recognition], [OSWorld], [WildClawBench]
  • 📖 TLDR: Talk2Agent benchmarks how well voice interfaces convey human-spoken instructions to computer-use agents, using spoken versions of WildClawBench and OSWorld tasks. It proposes an execution-free, task-conditioned evaluation that measures how much task-relevant information survives transcription, correlating with downstream completion better than WER/CER.

• cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

  • Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh
  • 🏛️ Institutions: CMU
  • 📅 Date: September 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [cua-speedrun], [benchmark], [efficiency], [evaluation], [latency]
  • 📖 TLDR: cua-speedrun provides standardized virtual-machine infrastructure, a common agent interface, and task sets for measuring the speed and cost of computer-use agents across four benchmarks. It shows how reasoning effort, agent harness, and environment latency affect speed, and that most benchmark task sets can be reduced without losing statistical power.

• Absorbed in Inertia: Activation Analysis for Computer-Use Agents

  • Giulio Segalini, Zhi Wen Soi, Jérémie Decouchant, Lydia Chen
  • 🏛️ Institutions: University of Neuchâtel, Delft University of Technology
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [inertia], [activation analysis], [failure analysis], [R3], [training-free]
  • 📖 TLDR: Shows that computer-use agents can exhibit inertia, repeating fruitless actions while recognizing that they are ineffective, and that inertia corresponds to an absorbing region of the underlying model's activation space. Proposes R3 (Reset, Reroute, Restore), which temporarily resets the context trajectory and then restores it, lowering measured inertia by 17-55% across models.

• AdaptArena: Evaluating Test-Time Personalization of Web Agents

  • Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala, Tomás Vergara Browne, Jaewon Moon, Fengyuan Liu, Alexandre Drouin, Siva Reddy, Alexandre Lacoste
  • 🏛️ Institutions: Mila, McGill University, ServiceNow Research, Laval University
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [AdaptArena], [benchmark], [personalization], [preference inference]
  • 📖 TLDR: AdaptArena is a benchmark of 480 tasks for test-time personalization of web agents, where the agent must infer a latent user preference from a relevant historical trajectory. Oracle agents given the true preference succeed on 82.92% of tasks while evaluated agents reach at most 15.62%, and correct preference inference does not guarantee downstream execution success.

• EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?

  • Hongcheng Gao, Hailong Qu, Yu Lei, Henghui Sun, Haoyang Li, Yipeng Wei, Naihao Xue, Xiaohan Yu, Zhuo Tao, Yihe Zang, Yajiao Wang, Jingyi Tang, Yi Li, Jingjing Zhou, Jie Luo, Bohan Zeng, Chengyu Shen, Hao Jiang, Chong Chen, Bowen Qu, Olive Huang, Zeqiang Wang
  • 🏛️ Institutions: Tsinghua, Zhiman Inc., Chongqing University, University of Chinese Academy of Sciences, Shandong University, BUPT, Fudan, Henan Polytechnic University, XJTU, PKU, ZJU
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [EngiWorld], [benchmark], [professional software], [engineering], [CAD]
  • 📖 TLDR: EngiWorld is a benchmark of 1,301 expert-curated tasks across six engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms, with both GUI and CLI interfaces. Domain verifiers check the geometric validity, physical feasibility, and rule compliance of artifacts; the strongest of seven frontier models reaches an EngiScore of 44.3.

• HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

  • Tongbo Chen, Junbo Niu, Zhengxi Lu, Niu Lian, Fei Tang, Yuchen Yan, Yike Hong, Yong Du, Yizhou Liu, Bofan Chen, Yongliang Shen
  • 🏛️ Institutions: ZJU, PKU, Tsinghua
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [HybridCUA], [dataset], [model], [CLI], [reinforcement learning], [verifiable rewards], [OSWorld], [WindowsAgentArena]
  • 📖 TLDR: HybridCUA trains computer-use agents to combine GUI interaction with the command line, using a data pipeline that produces GUI-only, CLI-only, and interleaved trajectories (HybridCUA-8K), followed by supervised fine-tuning and reinforcement learning with CLI-aware rewards. HybridCUA-9B reaches 53.6% on OSWorld, 14.8 points above its base model.

• MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows

  • Mei Wu, Rui Xie, Runyu Zhang, Yuqiang Li, Tianfan Fu, Bo Chen, Kai Yu, Xin Chen, Lu Chen
  • 🏛️ Institutions: SJTU, Suzhou Laboratory, Shanghai AI Laboratory, BIGAI, Shanghai Innovation Institute
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [MatToolBench], [benchmark], [scientific software], [materials science], [OriginPro]
  • 📖 TLDR: MatToolBench is a real-environment benchmark of 204 tasks across 10 professional materials science tools, covering GUI operation, OriginPro scripting, and database queries in a Windows 11 VM, with expert-defined sub-criteria for partial credit. The best model reaches 25% on GUI tasks, and failures stem from missing domain-specific operational knowledge rather than visual grounding alone.

• Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient Execution

  • Hyewon Suh, Thanh Minh Nguyen, Chih-Lun Lee, Darrow Hartman, Lizhao Liu, Xin Eric Wang, Ang Li, Jiachen Yang
  • 🏛️ Institutions: Georgia Tech, Stanford, Simular Research
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [neuro-symbolic computer use], [reusable policies], [efficiency], [OSWorld], [ScienceBoard]
  • 📖 TLDR: Executes recurring computer workflows with a learned neuro-symbolic policy instead of re-planning every run: executable code fixes the stable decisions (ordering, variables, loops, branches) and neural models handle observation-dependent ones such as grounding and state checks. On OSWorld-Verified and ScienceBoard the policies reach the highest Pass^3 among compared methods while cutting per-run cost by 15-217x.

• PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval

  • Bin Kang, Jiarui Ouyang, Li Jiang, Bin Chen, Zhuotao Tian
  • 🏛️ Institutions: University of Chinese Academy of Sciences, HKUST, Harbin Institute of Technology, CUHK, Shenzhen Loop Area Institute
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [PrecogUI], [benchmark], [InterfereBench], [world model], [memory], [long-horizon tasks]
  • 📖 TLDR: PrecogUI moves GUI agents from reactive to proactive decision-making with an experience pool of anomaly and success patterns, a simulator that forecasts the next symbolic UI layout for a candidate action, and a controller that fuses both with closed-loop error correction. It also builds InterfereBench for long-horizon tasks with strong disturbances, where PrecogUI surpasses prior methods.

• SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents

  • Shengtian Yang, Ziyu Xiong, Kaibing Yang, Guangfeng Cai, Yewen Li, Peng Jiang, Gai Kun, Qingpeng Cai, Lei Feng
  • 🏛️ Institutions: Southeast University, Kuaishou Technology
  • 📅 Date: September 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [SCA], [reinforcement learning], [GRPO], [GUI grounding], [credit assignment]
  • 📖 TLDR: Spatial Credit Assignment (SCA) refines group-relative reinforcement learning for GUI agents using the screen coordinates of sampled clicks. It predicts each response's reward from the others in mixed groups, and when every sampled click fails it ranks them by distance to the target, improving grounding and offline action prediction.

• Before Acting, Change the State: Prospective State Intervention for Web Agents under Deceptive Interfaces

  • Ruozhao Yang, Mingfei Cheng, Xiaofei Xie
  • 🏛️ Institutions: Singapore Management University
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [Veer], [dark patterns], [safety], [runtime defense], [TrickyArena], [WebDecept]
  • 📖 TLDR: Treats task-relevant web state as a runtime control target for defending web agents against deceptive interfaces, since a task-valid action can still realize an unauthorized consequence. Veer constructs and verifies a prospective intervention trajectory toward a safe state before the action executes, and it achieves the highest safe task completion on TrickyArena and WebDecept.

• GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

  • Shaoqing Zhang, Kehai Chen, Xuefeng Bai, Zhuosheng Zhang, Pengfei Zhang, Yang Xiang, Min Zhang
  • 🏛️ Institutions: HIT-Shenzhen, Pengcheng Laboratory, SJTU
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Web]
  • 🔑 Key: [GUITAR], [failure analysis], [state transition graph], [offline evaluation], [AndroidControl], [Mind2Web]
  • 📖 TLDR: GUITAR is a state-centric diagnostic framework that maps visually diverse screens to shared functional states in a State Transition Graph and analyzes GUI-agent failures over states and transitions. On AndroidControl and Mind2Web it finds that 60.4% of failures occur in 20% of states, and bottleneck-targeted guidance improves success rate.

• LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

  • Bingo Zhang, Haochuan Lu, Zongjie Li, Genjian Li, Ari Yu Zhang, Chaozheng Wang
  • 🏛️ Institutions: Vera Praxis, Tencent, HKUST, Independent Researcher, CUHK
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [LongPuzzleBench], [benchmark], [long-horizon tasks], [puzzle games], [planning]
  • 📖 TLDR: LongPuzzleBench evaluates GUI agents on 114 levels of six puzzle games played through native GUI actions, where a legal move can make the puzzle unsolvable several moves later. Seven of ten general-purpose agents solve nothing harder than Medium, and diagnostics show agents judge a move by its visible progress rather than the future options it leaves.

• One Rollout Is All You Get: Fully Test-Time Adaptation for GUI Agents

  • Ziqiang Wang, Li Gu, Zhixiang Chi, Linlian Jiang, Zihuan Jiang, Linqiang Guo, Siobhan Reid, Zhi Liu, Yang Wang
  • 🏛️ Institutions: Concordia University, Mila, University of Toronto, Shanghai University
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [Mobile]
  • 🔑 Key: [SOLO], [test-time adaptation], [self-distillation], [continual learning], [WebArena], [VisualWebArena], [MobileWorld]
  • 📖 TLDR: Defines fully test-time adaptation for GUI agents, where a deployed agent gets one attempt per task and no ground truth, and proposes SOLO. A judge selects episodes it deems successful, a proposer-verifier pair relabels failed prefixes, and a small adapter is updated by top-K self-distillation, improving UI-TARS-7B and Qwen3-VL-8B by three to six points on recurring web and mobile task streams.

• PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

  • Yangqin Jiang, Lingrui Xu, Chao Huang
  • 🏛️ Institutions: HKU
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [PhoneCLI], [compilation], [efficiency], [AndroidLab], [AndroidWorld], [training-free]
  • 📖 TLDR: PhoneCLI compiles a mobile app's GUI navigation into callable commands by exploring the app offline into an annotated screen map; online, the agent verifies and replays a command deterministically and falls back to the VLM for open-ended interaction. It needs no app-internal API or training and improves success while cutting steps and tokens on AndroidLab.

• SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety

  • Jianxing Chen, Xiao Yu, Shipra Agrawal, Zhou Yu
  • 🏛️ Institutions: Columbia
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [SCOUT], [safety], [verification], [agent-as-a-judge], [AutoElicit-Bench], [OS-Blind]
  • 📖 TLDR: SCOUT is a two-stage agentic safety verifier for computer-use agents: a rubric generator reasons over the task and trajectory to define safe and complete execution, and a probing agent interacts with the post-execution environment to collect grounded evidence. It outperforms LLM-as-a-judge verifiers on AutoElicit-Bench and OS-Blind, and test-time reflection lowers the unsafe execution rate from 30.2% to 17.2%.

• Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?

  • Tien Tran, Namho Koh, Daiki E. Matsunaga, Ayush Jain, Kee Eung Kim
  • 🏛️ Institutions: KAIST, CMU
  • 📅 Date: September 28, 2026
  • 📑 Publisher: Findings of EMNLP 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [AnyAppBench], [benchmark], [cross-application generalization], [failure analysis], [Android]
  • 📖 TLDR: AnyAppBench is a live Android benchmark with 100 task templates and 520 task-application pairs over 52 applications that keeps the user goal fixed across apps. Across 13 agents, success on the original application does not transfer reliably to new applications with the same goal, and sub-goal decomposition does not close the gap.

• Structured Interaction, Visual Localization, and Robust Execution for Complex Web Tasks: A Technical Report on the WebRetriever Challenge

  • Ziqi Zhang, Shaohui Li, Bing Li
  • 🏛️ Institutions: Beijing Key Laboratory of Multimodal Super-Intelligent Security, People AI
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [WebRetriever Challenge], [technical report], [visual localization], [context management]
  • 📖 TLDR: Technical report of the winning system in the WebRetriever Challenge. It uses semantic webpage information for routine browser operations and invokes vision only when structured representations fall short, with grid-assisted visual localization, hierarchical context management, and fault-aware execution across eight concurrent browser workers; it achieved a 59% pass rate on the official Protocol 3.

• WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents

  • Anton Emelyanov, Maria Tikhonova, Zaven Martirosian, Sergei Averkiev, Alena Fenogenova
  • 🏛️ Institutions: DAIMLD, HSE University, NUST MISIS
  • 📅 Date: September 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [WebPageBench], [benchmark], [event-level verification], [UI variation], [evaluation]
  • 📖 TLDR: WebPageBench verifies each web-agent task from the interface's own event log across six instrumented mock sites, with no judge model, and supports controlled UI variation that re-renders one control while keeping the prompt and success conditions fixed. On its 152-task leaderboard, the gap between what agents declare finished and what the log confirms reaches 41 points.

• Probe to Act: Elevating Browser-Use Agent via Active Visual Probing

  • Keliang Li, Heng Wang, Chen Hu, Daxin Jiang, Hong Chang, Shiguang Shan
  • 🏛️ Institutions: Institute of Computing Technology, CAS, University of Chinese Academy of Sciences, StepFun
  • 📅 Date: September 27, 2026
  • 📑 Publisher: Findings of EMNLP 2026
  • 💻 Env: [Web]
  • 🔑 Key: [P2A], [Set-of-Marks], [memory], [VisualWebArena], [long-horizon tasks]
  • 📖 TLDR: Probe to Act (P2A) lets a browser-use agent issue lightweight probes at decision time that translate DOM handles into pixel evidence and map screen regions back to DOM candidates, keeping only probed or committed observations as memory. It works as a prompting strategy for proprietary models and can be distilled into open-weight models, improving results on three browser-use benchmarks.

• Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

  • Fuyao Zhang, Xuan Wang, Zherui Li, Jiaming Zhang, Longtao Huang, Wei Yang Bryan Lim
  • 🏛️ Institutions: NTU, Alibaba Group
  • 📅 Date: September 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [ExpActivator], [personalization], [memory], [training-free], [proactive agent]
  • 📖 TLDR: Argues that relevance does not imply applicability for personal GUI agents: retrieved history mainly helps the first step of an episode. ExpActivator is a training-free framework that matches each screen to historical states in the backbone's latent space and activates a recurring intent only when time and scenario support it, improving within-trajectory step success on four backbones.

• AgentTell: Behavioural Side-Channel Leakage in Browser-Use Agents

  • Asif Shahriar, Md Nafiu Rahman, Sadif Ahmed, Farig Sadeque, Md Rizwan Parvez
  • 🏛️ Institutions: BRAC University, Qatar Computing Research Institute
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [AgentTell], [benchmark], [privacy], [side-channel leakage], [browser-use agent]
  • 📖 TLDR: AgentTell studies behavioural side-channel leakage, where a browser-use agent's actions reveal a private fact learned on a prior website despite an instruction not to disclose it. Across 9,760 sessions on six backbones, agents reveal the secret through their actions in 61.1% of sessions.

• CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

  • Prince Zizhuang Wang, Chenhao Liang, Zelong Xu, Aojie Yuan, Xiaolin Zhou, Haiyue Zhang, Yue Zhao, Xiyang Hu, Shuli Jiang
  • 🏛️ Institutions: CMU, USC, University of Wisconsin-Madison, Arizona State University, AWS Agentic AI
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [CUA-SWE], [benchmark], [software engineering], [environment], [visual feedback]
  • 📖 TLDR: CUA-SWE is a benchmark, environment, and evaluation pipeline for software engineering with computer use, in which agents modify code, run commands, interact with the running application, and inspect visual feedback within one task. It spans four software engineering domains with deterministic task-specific tests, and includes tasks where required information is available only through the application's interface.

• CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning

  • Xin Yan, Zhengbo Jiao, Jiaqi Liu, Zhenglin Wan, SiYuan Ma, Xuliang Yu, Tianyi Jiang, Chubin Zhang, Pengfei Zhou, Wangbo Zhao, Xingrui Yu, Bo An, Yang You, Ivor Tsang
  • 🏛️ Institutions: A*STAR, HKUST, Beijing Normal University, NUS, NTU, ZJU, PKU
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [Desktop]
  • 🔑 Key: [CUA-Sandbox], [training environment], [reinforcement learning], [efficiency]
  • 📖 TLDR: CUA-Sandbox makes computer-use reinforcement learning cheaper by sharing initialized application runtimes across concurrent rollouts while keeping each trajectory's mutable state in a private capsule with transactional resets and branches. It matches or improves task success relative to Docker with up to 6.20x rollout throughput and 9.2x lower per-environment memory.

• DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis

  • Chuhan Zhang, Qi Xie, Ziyue Wang, Jianing Yin, Yunfan Zhou, Dazhen Deng, Yingcai Wu
  • 🏛️ Institutions: ZJU
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [DashAct], [benchmark], [dashboard analysis], [diagnostic evaluation], [visual grounding]
  • 📖 TLDR: DashAct is a diagnostic benchmark of 357 human-verified trajectories for GUI agents on interactive dashboards. A progressive cascade evaluates end-to-end execution, then restores verified context for next-action prediction, then adds target semantics and a local view for grounding, revealing bottlenecks hidden by end-to-end scores.

• From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs

  • Yangzhe Peng, Xiaoyang Wang, Yiyang Zhao, Lijun Wu, Kun He
  • 🏛️ Institutions: HUST, Shanghai AI Laboratory, ZJU, Shanghai Innovation Institute
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [SurfPRM], [dataset], [process reward model], [reward model], [data synthesis], [WebArena-Lite], [WebPRMBench]
  • 📖 TLDR: SurfPRM synthesizes process preference data for comparative web process reward models by building an Interaction Element Graph from demonstrations and using it to propose grounded contrastive negative actions, raising the share of grounded minimal contrastive pairs from 24.19% to 74.60%. PRMs trained on the resulting data improve on WebPRMBench and guide step-level search on WebArena-Lite.

• PastForward: Faster On-Device GUI Agents via Computational Experience Reuse

  • Taehwan Park, Changmin Lee, Hayeon Lee, Taesik Gong
  • 🏛️ Institutions: UNIST, Meta
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [PastForward], [efficiency], [on-device agents], [speculative decoding], [AndroidWorld]
  • 📖 TLDR: PastForward speeds up on-device GUI agents by reusing computation from earlier task executions: it retrieves prior outputs as multi-token proposals verified in one forward pass, and begins next-step inference from predicted screens while the current action runs. It cuts action-step latency by 1.63-2.36x on AndroidWorld workloads while maintaining task success.

• REBASE: Device-Cloud Experience Coherence for GUI Agents Across App Updates

  • Beining Wu, Jun Huang, Yanxiao Zhao
  • 🏛️ Institutions: South Dakota State University, Virginia Commonwealth University
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [REBASE], [app updates], [device-cloud collaboration], [experience reuse], [robustness]
  • 📖 TLDR: REBASE keeps the device and cloud copies of a mobile GUI agent's recorded experience coherent across app updates: the cloud replays experience on the new version, the device verifies each step, and failures trigger escalating evidence, starting from an accessibility-subtree diff, until a version-keyed patch is derived. On two Jetson devices it restores the success rate of stale experience to that of fresh experience within one episode.

• The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models

  • Dongsheng Liu, Chao Jin, Wenkui Yang, Hejin Wang, Junwei Yang, Zeren Zhang, Ziwei Chen, Huaibo Huang, Jie Cao, Ran He
  • 🏛️ Institutions: University of Chinese Academy of Sciences, CASIA, Huawei Noah's Ark Lab
  • 📅 Date: September 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [StateAliasBench], [benchmark], [world model], [state aliasing], [AndroidWorld]
  • 📖 TLDR: Identifies state aliasing in GUI world models, where the visible interface omits transition-relevant environment state so identical observations can lead to different valid futures. Introduces StateAliasBench to isolate it and a lightweight predictive-state recovery that augments frozen world models, restoring state-sensitive prediction and improving agents on AndroidWorld.

• CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs

  • Zhenhao Zhang, Zhaoyu Fan, Haohan Ying, Jingwen Hu, Hancen Fan, Junhao Zhou, Zitian Chen, Linchao Zhu
  • 🏛️ Institutions: Columbia, ZJU, University of Rochester, UIUC
  • 📅 Date: September 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [CaptchaArena], [CAPTCHA], [dataset], [model], [reinforcement learning]
  • 📖 TLDR: CaptchaArena is a training dataset of 50K execution-verified interactive CAPTCHA puzzles across 20 types and 5 interaction modes, with screenshot-action trajectories, step-by-step reasoning annotations, and pixel-mask annotations. Supervised fine-tuning followed by reinforcement learning on it yields CaptchaAgent, a single 9B policy for all 20 types.

• From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks

  • Yuchen Sun, Chenglin Cai, Gongjie Zhang, Tianyu Xia, Quyu Kong, Panrong Tong, Zhengwen Zeng, Long Chen, Steven Hoi, Chongyang Zhang, Yue Wang
  • 🏛️ Institutions: SJTU, Tongyi Lab
  • 📅 Date: September 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [GUI-Hopper], [deeplink], [hybrid interaction]
  • 📖 TLDR: GUI-Hopper lets a mobile GUI agent jump to a target screen through an app-native deeplink and fall back to taps and swipes for everything else. Deeplinks are found by static analysis, validated on real devices, and stored with descriptions of their landing screens as a verified catalog; using this catalog improves task success in commercial apps on real devices.

• ScreenHaystack: Finding Blind Zones in GUI Grounding

  • Chenyue Li, Xiaoxiao Sun, Yubo Deng, Qinlin Zhao, Serena Yeung-Levy, Yuhui Zhang
  • 🏛️ Institutions: Stanford
  • 📅 Date: September 25, 2026
  • 📑 Publisher: EMNLP 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [ScreenHaystack], [benchmark], [GUI grounding], [ScreenSpot-Pro], [blind zones]
  • 📖 TLDR: ScreenHaystack is a needle-in-a-haystack benchmark that relocates controlled target icons across high-resolution GUI backgrounds, revealing blind zones where grounding accuracy drops sharply for models including Qwen3-VL, UI-TARS, GTA, and UI-Venus. The blind zones transfer to ScreenSpot-Pro, and augmenting training data in them improves accuracy.

• Jev-Mobile: Jev as an Executor for Mobile GUI Agents

  • Linghua Zhang
  • 🏛️ Institutions: Rice University
  • 📅 Date: September 24, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [Jev-Mobile], [Jev], [accessibility tree], [efficiency]
  • 📖 TLDR: Jev-Mobile splits a mobile GUI agent into low-frequency VLM planning and high-frequency execution: the VLM sets local goals, the accessibility tree defines the executable action space, and Jev, a fast typed decision model, selects actions inside it. On the full AndroidWorld suite it reaches 79% task success against 84% for a step-wise VLM baseline, while cutting execution time by 32.7% and model API cost by 73.4% on successful trajectories.

• The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge

  • Alexander Gill, Md Farhan Ishmam, Xuyen Nguyen, Neha Bhat, Parker Henry DeYoung, Fateme Hashemi Chaleshtori, Nathan Stringham, Kenneth Marino, Ana Marasović
  • 🏛️ Institutions: University of Utah
  • 📅 Date: September 24, 2026
  • 📑 Publisher: Findings of EMNLP 2026
  • 💻 Env: [Web]
  • 🔑 Key: [KNOWS], [benchmark], [long-horizon tasks], [artifact generation]
  • 📖 TLDR: KNOWS is a benchmark of open-ended, long-horizon, browser-based tasks in which an agent retrieves information, synthesizes it, and produces an artifact such as a document, presentation, or spreadsheet, scored by evaluators that combine deterministic checks with LLM judgments. The best frontier computer-use agent or browser harness fully succeeds on fewer than 3% of tasks, and failures on visual steps make the produced artifacts unusable.

• CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments

  • Yuxuan Li, Will Epperson, Wesley Deng, Zezhou Huang
  • 🏛️ Institutions: CMU, MSR
  • 📅 Date: September 23, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [CAVEAT], [CAVEAT-Harness], [benchmark], [incentive robustness]
  • 📖 TLDR: CAVEAT is a benchmark of nine marketplace environments and eight steering mechanisms that tests whether computer-use agents keep the user's objective when the environment itself has a stake in the outcome. Agents buy the user-optimal product in 78.6% of matched-control episodes but only 17.3% when steering is enabled, and CAVEAT-Harness, which targets the diagnosed failure modes, raises user-optimal purchasing by 55.0%.

• Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents

  • Chengguang Gan, Yunhao Liang, QingHao Zhang, Shiwen Ni
  • 🏛️ Institutions: Independent Researcher, University of Chinese Academy of Sciences, Pusan National University, Shenzhen University of Advanced Technology
  • 📅 Date: September 23, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [WebMRE], [benchmark], [WebArena], [offline evaluation], [mutual reinforcement]
  • 📖 TLDR: WebMRE is an offline benchmark of 541 tasks and 5,293 steps derived from successful WebArena trajectories that scores a web agent checkpoint deterministically without an environment, with each step pairing a guide sentence with a grounded action. Decoding the guide causally improves action accuracy (forcing the gold guide lifts it from .422 to .684), and the mutual reinforcement effect grows from the 4B to the 9B model.

• Invisible in Space, Visible in Time: Motion Vision CAPTCHA against GUI Agents

  • Zeyu Zhang, Dingyi Rong, Zijian Chen, Zicheng Zhang, Xiongkuo Min, Guangtao Zhai
  • 🏛️ Institutions: SJTU, Shanghai AI Laboratory
  • 📅 Date: September 23, 2026
  • 📑 Publisher: ACM MM 2026
  • 💻 Env: [Web]
  • 🔑 Key: [MVCAP], [MVCAP-Bench], [benchmark], [CAPTCHA]
  • 📖 TLDR: Motion Vision CAPTCHA (MVCAP) hides target semantics in motion-defined foreground structures that can only be recovered by temporal segregation from a dynamic background, at three levels from coherent to biological motion. On MVCAP-Bench, a browser-based benchmark of 600 live instances, humans reach 99.6% while the best GUI agent reaches 16.8%, close to six-way chance.

• Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents

  • Yan Zhang, Daiqing Wu, Huawen Shen, Liang Li, Gang Cao, Zhi Gong, Wei Dai, Xiaode Zhang, Can Ma, Yu Zhou
  • 🏛️ Institutions: Institute of Information Engineering, Tencent, Tsinghua, University of Chinese Academy of Sciences, Nankai University
  • 📅 Date: September 23, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [GUI-SD-v2], [self-distillation], [model]
  • 📖 TLDR: GUI-SD-v2 extends on-policy self-distillation from GUI grounding to multi-turn GUI agents in two stages: it first strengthens the self-teacher's ability to follow privileged guidance, then selectively distills step-specific reasoning and memory guidance. It outperforms existing on-policy self-distillation baselines and the evaluated state-of-the-art methods on AndroidWorld and MobileWorld in Pass@1 and Pass@3.

• Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction

  • Yan Zhang, Pei Fu, Daiqing Wu, Huawen Shen, Ruoceng Zhang, Shaojie Zhang, Jiahui Yang, Yu Zhou, Can Ma, Zhenbo Luo, Jian Luan
  • 🏛️ Institutions: Institute of Information Engineering, Xiaomi, University of Chinese Academy of Sciences, Nankai University
  • 📅 Date: September 22, 2026
  • 📑 Publisher: EMNLP 2026
  • 💻 Env: [Mobile], [Web]
  • 🔑 Key: [MaP], [masked trajectory prediction], [multi-task], [model]
  • 📖 TLDR: MaP (Masked Trajectory Prediction) casts multi-turn GUI interaction as a trajectory and defines training objectives by masking and predicting its components, unifying step-wise decision-making, state-action alignment, and long-horizon planning under one objective. A role-aware adapter routes tokens to specialised representation spaces, and MaP outperforms direct mixture training on five GUI navigation benchmarks.

• How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation

  • Yunxiang Li, Xixin Wu, Helen Meng
  • 🏛️ Institutions: CUHK
  • 📅 Date: September 21, 2026
  • 📑 Publisher: EMNLP 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [PACE], [confidence estimation], [GUI grounding], [ScreenSpot-Pro]
  • 📖 TLDR: GUI agents emit click coordinates as digit tokens, and Place-Aware Coordinate Entropy (PACE) weights each digit's Shannon entropy by its place value to estimate per-click confidence in a single forward pass. On ScreenSpot-Pro and ScreenSpot-v2 it wins AUROC and selective accuracy on all primary comparisons, matching or outperforming K-sample baselines at a fraction of the cost.

• OSWorld-Pro: Process-based Evaluation for Computer Use Agents

  • Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, Karan Sapra, Andrew Tao, Jan Kautz, Yi Dong
  • 🏛️ Institutions: NVIDIA
  • 📅 Date: September 21, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [OSWorld-Pro], [benchmark], [process evaluation], [OSWorld]
  • 📖 TLDR: OSWorld-Pro is a set of over 300 tasks with over 2,800 subgoals, grounded in more than 67,000 human annotations, that evaluates computer-use agents on the process rather than only the end state, using human-aligned LLM judges. The top performer, Claude Opus 5, reaches 75.7% against 83.4% on OSWorld, and the analysis surfaces process failures such as subgoal-irrelevant actions and click mistakes.

• Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions

  • Xunjian Yin, Tianchen Guan, Jinao Wang, Weili Cao, Daisy Xinlei Lin, Royce Cheng-Yue, Keagan Long, Kyle Wong, Bhuwan Dhingra, Xiangjun Wang, Shuyan Zhou
  • 🏛️ Institutions: Duke University, Amazon
  • 📅 Date: September 20, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [BreakingWeb], [benchmark], [browser-use agent], [robustness], [failure analysis]
  • 📖 TLDR: BreakingWeb builds harder browser-use tasks from tasks that agents already solve, by pairing each base task with a deterministic environment intervention at one web stack layer that keeps the instruction and the backend success criterion unchanged. Its 519 clean/intervention task pairs over seven self-hosted websites lower agent pass rates by 22.9% on average, and 75% of the failures of six browser-use agents end with a declared success although the required change never happened.

• MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning

  • Changyue Jiang, Jiayi Wang, Xin Wen, Jiarun Dai, Geng Hong, Xudong Pan
  • 🏛️ Institutions: Fudan, Shanghai Innovation Institute
  • 📅 Date: September 19, 2026
  • 📑 Publisher: USENIX Security 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [MATE], [MATEBench], [benchmark], [security], [model]
  • 📖 TLDR: MATE is a policy-conditioned auditor that reads a mobile agent's trajectory together with a natural-language security policy, decides whether the trajectory violates it, and explains why, so policies can change without retraining. It is trained on more than 140K synthesized trajectories built from hundreds of apps, is evaluated on the released MATEBench, and audits AutoGLM and Mobile-Agent trajectories on real devices with over 95% accuracy.

• MintAct: A Unified Visual Agent for Digital Environments

  • Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy, Kaixin Ma, Vishwesh Kirthivasan, Oğuzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan
  • 🏛️ Institutions: Apple
  • 📅 Date: September 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Desktop], [Web]
  • 🔑 Key: [model], [MintAct], [GUI grounding], [reinforcement learning], [unified agent]
  • 📖 TLDR: MintAct is a family of 2B/4B/8B vision-language models that unifies GUI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use in a single agent. Backed by an asynchronous RL infrastructure that hosts hundreds of concurrent environment instances, it matches per-domain specialists across all these capabilities and reaches 48.9 on OSWorld-Verified.

• RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

  • Shuai Bai, Jiayong Deng, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou
  • 🏛️ Institutions: Alibaba Group
  • 📅 Date: September 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [benchmark], [dataset], [RecreationWorld], [RecreationBench], [hybrid computer use], [execution-grounded reward]
  • 📖 TLDR: RecreationWorld is a five-platform environment suite (Ubuntu, macOS, Windows, Android, Web) for hybrid computer-use agents that interleave GUI interaction with coding, using a running reference application as an oracle for hidden behavioral tests. Models trained on its generated trajectories improve across five out-of-distribution coding and hybrid benchmarks, and on the held-out 250-task RecreationBench the leading agent reaches 58.1% overall while passing all programmatic tests on only 2.8% of tasks.

• A Dual-Process Perspective on Nudge Susceptibility in LLM-Based GUI Agents

  • Haya Halimeh, Sascha Kaltenpoth, Kevin Bösch, Oliver Müller
  • 🏛️ Institutions: Paderborn University
  • 📅 Date: September 17, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [safety], [digital nudging], [dual-process theory], [choice architecture]
  • 📖 TLDR: This study tests whether LLM-based GUI agents are susceptible to digital nudges, running 21,600 simulations of 3,600 agents across six frontier models in a randomized online-shopping experiment. Agents proved vulnerable to both automatic and reflective nudges, and extended reasoning did not confer robustness: it reduced susceptibility to default nudges while heightening susceptibility to social-influence nudges.

• EconSkills: Studying Skill Transfer and Retrieval for Web Agents on Live Economic Data

  • Yinzhu Quan, Zefang Liu
  • 🏛️ Institutions: Georgia Tech, Capital One
  • 📅 Date: September 17, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [EconSkills], [skill library], [skill transfer], [retrieval], [EconWebArena]
  • 📖 TLDR: EconSkills distills verified EconWebArena trajectories into parameterized standard operating procedures for retrieving live economic data, each recording scope, navigation steps, verification checks, and recovery steps with instance values replaced by placeholders. Matched skills improve success over no-skill prompting and abstraction beats replaying raw trajectories, but at library scale retrieval is only competitive with the no-skill baseline because approximate matches on uncovered tasks offset the gains.

• Affora: A Design System for Agent-Friendly Interfaces

  • Jin Gao
  • 🏛️ Institutions: Independent Researcher
  • 📅 Date: September 16, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [Affora], [design system], [agent-friendly interface], [interface affordance]
  • 📖 TLDR: Affora is a design system for making software interfaces legible to computer-use agents without constraining their visual design for human users, grounded in three controlled studies of component implementations, visual variation, and interaction-design principles. It reports that agent performance depends on the interaction meaning exposed through the interface representation, and that substantial visual variation remains possible when that meaning is preserved.

• RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection

  • Liyang Fan, Xinping Bi, Yitai Li, Shuaimin Li, Hui Li, Min Yang
  • 🏛️ Institutions: Shenzhen University, Shenzhen University of Advanced Technology, Shenzhen Institutes of Advanced Technology, CAS
  • 📅 Date: September 16, 2026
  • 📑 Publisher: ACM MM 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [RankGround], [GroundRanker], [GUI grounding], [reranking], [crop selection]
  • 📖 TLDR: RankGround achieves high-resolution GUI grounding with a single VLM call per query by first using GroundRanker, a lightweight multimodal reranker, to pick the most promising crop from a dense candidate set. Ranking supervision is constructed from existing grounding datasets under a strict containment criterion and trained with a pointwise-then-listwise curriculum, yielding 1.4x faster inference and 5.5% average accuracy gains over the second-best method across backbones and screen scales.

• The Verifiable Action Card: Trustworthy Human-in-the-Loop Control for Secure Autonomous Agents

  • Hasnain Irshad, Anam Mughees, Neelam Mughees, Abdullah Mughees, Imtiaz Ali Soomro
  • 🏛️ Institutions: Sir Syed CASE Institute of Technology, University of Engineering and Technology Lahore, National Textile University, KFUPM
  • 📅 Date: September 16, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [security], [prompt injection], [human-in-the-loop], [agentic browser]
  • 📖 TLDR: The Verifiable Action Card reconstructs approval information from the ground-truth pending browser action and trusted intent provenance, renders it out-of-band in trusted browser chrome, and binds the approval to the exact action re-verified at dispatch. On a 24-scenario benchmark covering confused-deputy attacks, dialog forging, indirect prompt injection, and provenance evasion, attack success falls from 68-100% without the mechanism to 0% on every evaluated model, with 78% legitimate-task completion and no false blocks.

• ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software

  • Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan, Yuying Zhao, Eugene Siow
  • 🏛️ Institutions: Accenture
  • 📅 Date: September 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [ERPBench], [enterprise software], [state-grounded evaluation]
  • 📖 TLDR: ERPBench evaluates screenshot-only computer-use agents on a live, reproducible ERP system and scores each task against ground-truth values in the underlying database rather than on-screen appearance. Across six closed and open-source agents it shows that strong general GUI performance does not transfer to enterprise reliability: some agents save a form in up to 85% of runs but write the correct value in as few as 3%.

• EchoPath: Execution-Level Replayable Memory for GUI Agents

  • Yao Zhao, Aditya Shanmugham, Swastik Roy, Yanxun Xu
  • 🏛️ Institutions: JHU, Amazon AGI
  • 📅 Date: September 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [EchoPath], [memory], [trajectory replay], [enterprise automation]
  • 📖 TLDR: EchoPath converts artifact-validated GUI trajectories into standardized, parameter-controlled callable memories that a host agent invokes only when they can be deterministically replayed in the current runtime. An image-based target-reaiming algorithm treats stored coordinates as visual evidence and corrects them against the current screen before execution, reducing median token cost by more than 90% and median execution time by about 60% on real computer-use tasks.

• Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents

  • Bofan Chen, Boxuan Zhang, Fei Tang, Zhengxi Lu, Yong Du, Tongbo Chen, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
  • 🏛️ Institutions: ZJU, UESTC
  • 📅 Date: September 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Desktop]
  • 🔑 Key: [framework], [EvoSkill-GUI], [skill evolution], [failure recovery], [training-free adaptation]
  • 📖 TLDR: EvoSkill-GUI makes GUI agent skills revisable at deployment time instead of fixed beforehand, representing each skill as a multi-file package holding retrieval metadata, executable plans, backup localization, failure-recovery rules, and failure cases. A reflect-revise-reuse loop lets the executor patch skills mid-rollout while an isolated critic diagnoses failed trajectories, improving multiple base models without any training by up to 16.2% on MobileWorld, 6.0% on AndroidWorld, and 10.5% on OSWorld.

• When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents

  • Heng Li, Fulin Zhao, Zhe Geng, Zhiyuan Yao, Wei Yuan, Xiapu Luo
  • 🏛️ Institutions: PolyU, Huazhong University of Science and Technology
  • 📅 Date: September 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [security], [UI desynchronization], [repackaging attack], [human oversight]
  • 📖 TLDR: This paper identifies human-agent UI desynchronization, where one mobile UI state presents materially different information to a human viewer, bound by occlusion and luminance contrast, than to an agent consuming screenshots and accessibility metadata. An automated framework builds deployable repackaged APKs that exploit the gap without runtime adaptation, reaching 77.9% and 66.9% average misleading rates across five mobile-agent frameworks and three backbones on 546 tasks, while a 186-participant study finds the perturbations hard for people to notice.

• AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation

  • Shengjie Jin, Zelong Sun, Hengbo Xu, Yanbiao Ma, Zhiwu Lu
  • 🏛️ Institutions: Renmin University of China
  • 📅 Date: September 14, 2026
  • 📑 Publisher: ECCV 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [AnchorGUI], [memory], [cross-trial distillation], [prediction error], [AndroidWorld]
  • 📖 TLDR: AnchorGUI addresses an informational asymmetry in GUI navigation, where expected transitions compress into lightweight text but unexpected outcomes need preserved screenshots as causal evidence, through a Cognitive State Anchor that turns trajectories into explicit prediction-error signals. Its asymmetric memory drives both in-episode correction and cross-trial credit assignment, reaching 57.3% on AndroidWorld with 2.4x fewer tokens per step and 69.2% after cross-trial distillation.

• CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design

  • Zihan Dong, Yuanzhe Liu, Zhiyuan Ma, Qishi Zhan, Dehan Kong, Guohao Li, Kaixin Li
  • 🏛️ Institutions: Georgia Tech, North Carolina State University, Marquette University, Saros Lab, CAMEL-AI.org, Eigent.AI, NUS
  • 📅 Date: September 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [CADWorld], [computer-aided design], [long-horizon task], [FreeCAD]
  • 📖 TLDR: CADWorld is a 200-task benchmark for long-horizon computer use in FreeCAD, spanning 11 mechanical-CAD workflow categories from sketching and part modeling to CAM, FEM, and technical drawing. Agents act through screenshots and GUI actions while success is decided by executable checks over saved FreeCAD artifacts, and the strongest of seven agents reaches 17.5% against an 87.0% expert reference pass rate.

• Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

  • Yizhou Liu, Fei Tang, Yuchen Yan, Zhengxi Lu, Songqin Nong, Tao Jiang, Wenhao Xu, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen
  • 🏛️ Institutions: ZJU, Ant Group
  • 📅 Date: September 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [test-time adaptation], [negative learning], [CANL], [ScreenSpot-Pro]
  • 📖 TLDR: This work introduces a label-free test-time training paradigm for GUI grounding built on two observations: coordinate-token confidence indicates correctness better than full-sequence confidence, and in sparse GUI coordinate spaces negative samples give more reliable signal than potentially noisy pseudo-positives. The resulting Confidence-Anchored Negative Learning reaches 92.1% on ScreenSpot-V2 and 33.8% on ScreenSpot-Pro, an 8.9-point absolute gain over the base model.

• RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments

  • Sibo Zhu, Shicheng Fan, Xinyue Wang, Wenyi Wu, Kun Zhou, Biwei Huang
  • 🏛️ Institutions: Aether AI, UC San Diego, University of Illinois Chicago
  • 📅 Date: September 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [RSIAgent], [recursive self-improvement], [memory construction], [OSWorld]
  • 📖 TLDR: RSIAgent is a training-free multi-agent framework in which curriculum, actor, and verifier agents autonomously explore a new environment and accumulate reusable causal knowledge linking actions, conditions, and consequences into a memory that is then frozen for downstream reuse. Using a broad-then-deep exploration strategy, it lifts strong open-source models on OSWorld-v2 and Agent's Last Exam enough to outperform frontier closed-source models.

• CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time

  • Nitish Kovuru, Prateek Jannu
  • 🏛️ Institutions: Coasty Research Lab
  • 📅 Date: September 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [CoArena], [live evaluation], [Bradley-Terry model], [pairwise comparison]
  • 📖 TLDR: CoArena proposes a real-time evaluation platform for computer-use agents in which real users submit tasks, two systems execute them concurrently in identical sandboxed desktops, and a public leaderboard is refit from blind pairwise judgments. The paper's contribution is a formal account of real-time evaluation as five measurable properties plus the full Bradley-Terry rating methodology; its five-system worked example with 211 votes is labeled illustrative rather than a measurement of a deployed system.

• PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI Agents

  • Qihang Cen, Tianshuo Cong, Da Song, Xinlei He, Jiaxing Song, Ke Xu, Qi Li
  • 🏛️ Institutions: Tsinghua, Shandong University, Wuhan University
  • 📅 Date: September 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dataset], [privacy], [PriMobiBench], [MobiLeak], [user profiling]
  • 📖 TLDR: PriMobiBench is a benchmark for visual privacy leakage in screenshot-driven mobile GUI agents, paired with MobiLeak, a dataset of execution traces from 16 apps covering 25 privacy attributes across 2,960 embedded privacy instances. VLMs extract sensitive on-screen information with up to 82.5% success and infer user profiles from aggregated visual evidence about 70% of the time; masking privacy-sensitive but task-irrelevant UI elements before cloud processing cuts profiling success by up to 58% at roughly 8% performance loss.

• AutoTailor: Automatic, User-Aligned Capability Selection and Adaptation for Web Agents

  • Xinyun Cao, Adriana Szekeres, Fazle Elahi Faisal
  • 🏛️ Institutions: UMich, MSR
  • 📅 Date: September 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [AutoTailor], [tool synthesis], [capability selection], [WebArena]
  • 📖 TLDR: AutoTailor builds and maintains a compact set of MCP APIs derived from web trajectories, filtering 1,283 unrefined APIs down to 87 offline by granularity, redundancy, and usage likelihood, then to 33 online through dynamic reselection driven by observed coverage gaps. On 106 WebArena Postmill tasks the resulting set with ReAct fallback reaches 90.6% correctness versus 87.5% for ReAct alone, while cutting average request-token cost by 57.8% and latency by 29.4%.

• BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents

  • Tong Ye, Kunyang Han, Guozhi Wang, Longqiang Luo, Zhifeng Ding, Yongxiang Zhang, Xiaolei Shen, Yuxuan Zhang, Zhuping Zhang, Tao Xu, Yue Pan, Yucheng Zhao, Yupei Hu, Yuanjiang Ouyang, Danfeng Shen, Runqi Lin, Hongda Cai, Zhaoxiong Wang, Mengjia Yan, Yingjie Zhong, Chen Zhou, Zeyu Zhang, Xuwen Zhu, Penggang Shi, Mingcheng Luo, Ziyang Wu, Min Jin, Mingfu Shen, Zairong Xu, Fan Zhang, Hao Wang, Liang Liu, Zhulin Xie, Lijun Yao, Xiao Liang, Liangmin Wen, Liqiang Feng, Feilong Wu, Min Hu, Min Chen, Guanjing Xiong, Xiaohu Ruan, Xiaoxin Chen
  • 🏛️ Institutions: vivo AI Lab
  • 📅 Date: September 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [real-device training], [self-improvement], [AndroidWorld], [BlueLM-GUI]
  • 📖 TLDR: BlueLM-GUI is a 35B-A3B native mobile GUI agent trained through continual pre-training, SFT, and agentic RL run on hundreds of real phones instead of sandboxes, with a pipeline that salvages failed trajectories into supervision. It reaches 87.4 on MobileGUI-VBench and 84.9 on AndroidWorld.

• Token Efficient Task Execution via Application Behavior Modeling for Web Agents

  • Alexandru Ianta, Eleni Stroulia
  • 🏛️ Institutions: University of Alberta
  • 📅 Date: September 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [OdoBot], [token efficiency], [application behavior model], [demonstration learning]
  • 📖 TLDR: OdoBot is a web-agent architecture that builds a behavioral model of the target application from successful task demonstrations and uses it to cut the token cost of execution. On 45 tasks in the Canvas learning management system it consumes 44% and 80% fewer tokens than Agent-E and WebVoyager respectively while also exceeding WebVoyager's task success rate.

• VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets

  • Yu Bai, Yukai Miao, Dawei Wang, Li Chen, Yanyu Ren, Yuqian Shi, Dan Li, Ying Xiong, Chengqiu Tan, Run Zhou, Li Li
  • 🏛️ Institutions: Zhongguancun Laboratory, Tsinghua, Network Management Center, China Mobile
  • 📅 Date: September 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web], [General GUI]
  • 🔑 Key: [benchmark], [verbal reinforcement learning], [trial budget], [exploration-exploitation], [VRL-Bench]
  • 📖 TLDR: VRL-Bench is a harness for fairly comparing verbal-reinforcement-learning and trial-and-error-memory methods on computer control under a fixed trial budget, evaluated on MiniWoB and WebShop. It finds reflection can reduce success under tight budgets and proposes VEX², an exploration-exploitation scheduler.

• GraphDroid: Asynchronous LLM-Based Mobile App GUI Testing via History-Aware Exploration and Hybrid Intent Fulfillment

  • Xiaolei Li, Jialun Cao, Zhijian Hou, Yuzhi Zhao, Yepang Liu, Shing-Chi Cheung
  • 🏛️ Institutions: HKUST, SUSTech, CityU
  • 📅 Date: September 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [GUI testing], [GraphDroid], [asynchronous exploration], [bug detection]
  • 📖 TLDR: GraphDroid is an intent-driven Android GUI testing framework that uses a cluster-based memory over visited states to identify uncovered functionality, generates test intents asynchronously so exploration never blocks, and delegates simple intents to a lightweight heuristic while reserving the LLM for complex ones. Across 41 real apps it achieves up to 36.4% higher code coverage at under one eighth the cost of the best pure-LLM baseline, exposing 19 bugs of which seven were previously unknown.

• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

  • Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang
  • 🏛️ Institutions: Beihang University, Beijing Institute of Technology, Baidu Inc.
  • 📅 Date: September 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Desktop]
  • 🔑 Key: [benchmark], [cross-device], [task composition], [JarvisGUI]
  • 📖 TLDR: JarvisGUI is a dynamic benchmark for GUI agents on cross-device workflows spanning Android, Windows, and Ubuntu, formulating tasks as typed input-output transformations so multi-step cross-device workflows can be composed and evaluated automatically.

• LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents

  • Zhangxuan Gu, Haoxing Chen, Qi Qin, Yi Xin, Kai Gan, Lin Liu, Long Cui, Xiaomei Wang, Beitong Zhou, Yunzhu Zhang, Zhengwen Zeng, Changlong Gao, Weizhi Chen, Rongchao Zhang, Haoyuan Wu, Shuheng Shen, Changhua Meng, Weiqiang Wang, Jianguo Li, Zhenzhong Lan
  • 🏛️ Institutions: Inclusion AI, Venus Team, Westlake University
  • 📅 Date: September 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Desktop], [Web]
  • 🔑 Key: [model], [LLaDA-UI], [diffusion language model], [block-wise decoding], [mixture of experts]
  • 📖 TLDR: LLaDA-UI is a 16.7B-parameter MoE block-wise diffusion vision-language GUI agent, built by aligning a native-resolution vision encoder with the LLaDA2.0-mini-base diffusion backbone and then fine-tuning on mobile, desktop, web, and grounding data. It substantially outperforms Qwen2.5-VL-7B and surpasses Qwen3-VL-8B on four of six reported GUI benchmarks, establishing block-wise diffusion as a practical generative paradigm for GUI agents.

• TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents

  • Yuhao Wang, Mu Qiao, Xindong Zhang, Yunzhi Zhuge, Lei Zhang, Huchuan Lu
  • 🏛️ Institutions: Dalian University of Technology, OPPO Research Institute, PolyU
  • 📅 Date: September 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [efficiency], [visual token pruning], [admission control], [TRACE]
  • 📖 TLDR: TRACE is a training-free visual-token pruning method for GUI agents that treats pruning as an irreversible admission decision, combining a layout-derived interaction prior with instruction relevance and feature novelty to keep operable regions under tight token budgets.

• APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

  • Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang, Yichen Gong
  • 🏛️ Institutions: Central China Normal University, Agentic Labs, Acrab AI
  • 📅 Date: September 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [reproducibility], [simulated apps], [APPSim-Bench]
  • 📖 TLDR: APPSim-Bench provides 557 tasks across 17 high-frequency Chinese and English apps, rebuilt as controllable simulated apps that preserve interaction logic while removing stochasticity from ads, recommendations, and accounts, and evaluates 19 GUI agents on them.

• FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?

  • Jingpu Yang, Fengxian Ji, Jinri Guo, Tianhao Li, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, Preslav Nakov, Zhuohan Xie
  • 🏛️ Institutions: Wuhan University, MBZUAI, Northeastern University, Zhongguancun Academy
  • 📅 Date: September 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [General GUI]
  • 🔑 Key: [benchmark], [financial agents], [autonomous benchmark construction], [FinCUABuild]
  • 📖 TLDR: FinCUABuildBench evaluates whether agents can autonomously construct computer-use-agent evaluation tasks for financial scenarios, via 576 construction requests over 24 financial workflows with a task-qualification mechanism for the resulting benchmarks.

• AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents

  • Zhihao Liu, Hongyu Sun, Zhiyuan Fu, Xiaonan Duan, Jice Wang, Shangru Zhao, Weizhi Meng, Wuxin Yang, Yangfan Zhou, Yuqing Zhang
  • 🏛️ Institutions: Hainan University, University of Chinese Academy of Sciences, Lancaster University
  • 📅 Date: September 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [General GUI]
  • 🔑 Key: [safety], [security], [adversarial patch], [prompt injection], [AgentHijack]
  • 📖 TLDR: AgentHijack is an end-to-end evaluation of image-triggered command injection against computer-use agents: adversarial patches deployed on real pages are tested across five GUI-agent/VLM backends over 600 online cases, reaching 84.5% trigger success rate and 20.3% end-to-end attack success rate.

• Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions

  • Juyong Lee, Woogyeol Jin, Kimin Lee
  • 🏛️ Institutions: KAIST
  • 📅 Date: September 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [tool actions], [self-generation], [DroidTool]
  • 📖 TLDR: DroidTool augments Android GUI agents with a hybrid GUI-plus-tool action space by auto-generating Python tools over app state through an agentic propose/implement/test/repair workflow with relational cross-tool tests, improving both proficiency and efficiency.

• Selective Knowledge Control for Continual GUI Agent Learning over Application Streams

  • Zirui Shang, Xin Shu, Yang Liu, Zhi Gao, Xinxiao Wu, Lifeng Fan
  • 🏛️ Institutions: Beijing Institute of Technology, BIGAI, Wuhan University, Shenzhen MSU-BIT University
  • 📅 Date: September 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [continual learning], [catastrophic forgetting], [neuron-level gradient control]
  • 📖 TLDR: This work proposes activation-conditioned selective knowledge control, a neuron-level gradient surgery method that protects highly-activated MLP neurons so GUI agents can learn new applications over a continual stream without interfering with knowledge from earlier ones.

• CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

  • Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang
  • 🏛️ Institutions: SJTU, ZJU
  • 📅 Date: September 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [environment], [hybrid GUI and CLI], [CUA-Universe]
  • 📖 TLDR: CUA-Universe is an environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments over shared application state, targeting agents that coordinate visual inspection with command-line throughput.

• ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults

  • Weide Zhan, Qumu Shaqu, Yuanqing Liu, Peng Zhang, Jiahao Liu, Kam Him Lam, Ning Gu, Zhan Hu, Tun Lu
  • 🏛️ Institutions: Fudan
  • 📅 Date: September 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [accessibility], [older adults], [ElderBench]
  • 📖 TLDR: ElderBench collects 249 naturally elicited smartphone tasks from older adults across 20 applications, characterizing how elderly instructions (indirect speech, referential ambiguity, under-specification) diverge linguistically from existing GUI benchmark instructions.

• From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents

  • Longtao Hu, Xiao Liang, Linchao Zhu
  • 🏛️ Institutions: University of Electronic Science and Technology of China, ZJU
  • 📅 Date: September 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [General GUI]
  • 🔑 Key: [skill library], [online evolution], [procedural memory]
  • 📖 TLDR: This work converts computer-use-agent trajectories and evaluator feedback into a persistent, versioned library of reusable procedures through online skill evolution, running each iteration against a frozen library snapshot to isolate the library's incremental value.

• Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents

  • Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng, Zheng Wu, Yansi Li, Chuanbiao Song, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang
  • 🏛️ Institutions: SJTU, Ant Group
  • 📅 Date: September 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [conflict detection], [termination], [feasibility verification], [CONFLICTGUI]
  • 📖 TLDR: CONFLICTGUI benchmarks instruction-internal and instruction-vs-GUI-context conflicts, exposing execution-biased overcompliance in GUI agents; CONFLICTGUARD is an inference-time feasibility-verification framework that aligns termination decisions with action generation.

• Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails

  • Suyoung Lee, Myungsub Choi
  • 🏛️ Institutions: Tynapse
  • 📅 Date: September 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [safety], [security], [prompt injection], [guardrail evaluation], [Mind2Web-Injection]
  • 📖 TLDR: Mind2Web-Injection provides 9,954 instruction-screenshot pairs with pixel-exact evidence boxes and image-side counterfactuals, measuring whether a web-agent guardrail VLM actually localizes the on-screen injected text it claims to detect, rather than only judging accuracy of the verdict.

• Discriminative World Models for Web Agents

  • Kelvin Li, Dhruv Pendharkar, Anish Pahilajani, Chuyi Shang, Leon Oks, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Roei Herzig
  • 🏛️ Institutions: UC Berkeley, MIT-IBM Watson AI Lab, Cal Poly San Luis Obispo, Xero
  • 📅 Date: September 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [world model], [discriminative training], [state prediction]
  • 📖 TLDR: This work replaces supervised next-state prediction with predicted-state matching, so a web world model's predicted HTML/AXTree state must discriminate the true resulting state from states reached by alternative actions, aligning the world model with the downstream ranker or process reward model.

• Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization

  • Bizhe Bai, Jiakang Yuan, Hongming Wu, Xinyue Wang, Jie Ren, Siyao Chen, Yuchen Ya, Fan Bai, Pai Peng, Huafeng Qin, Tao Chen
  • 🏛️ Institutions: Fudan, Shanghai Innovation Institute, Chongqing Technology and Business University
  • 📅 Date: September 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [survey], [efficiency], [observation], [memory], [runtime optimization]
  • 📖 TLDR: This survey organizes GUI-agent efficiency research along observation, context/memory, action, and planner/system axes, arguing that real-world deployment depends on efficiency as much as on task success rate.

• Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervision

  • Sitong Pan, Yipeng Shen, Yilin Lu, Caiwen Ding, Lu Cheng, Qianwen Wang
  • 🏛️ Institutions: University of Minnesota, Purdue University, Pennsylvania State University
  • 📅 Date: September 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [runtime monitoring], [risk prediction], [observable trajectories]
  • 📖 TLDR: This work does prefix-level risk prediction for web agents using only observable trajectory signals with no access to token logits, combining macro cross-step features with micro intention/action/state-change consistency features, labeled by the first uncorrected critical error.

• OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations

  • Yixiong Xiao, Lang An, Hucheng Yang, Pinxue Ma, Yongquan Chen, Jingjia Cao, Yusai Zhao, Ting Wang, Ting Liu, Siqi Bao, Jingbo Zhou, Hua Wu
  • 🏛️ Institutions: Baidu Inc., Ningxia Electric Power Engineering Co., Ltd.
  • 📅 Date: September 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [standard operating procedures], [human demonstrations], [OmegaUse-SOP]
  • 📖 TLDR: OmegaUse-SOP is a human-in-the-loop system that turns demonstrations of professional computer use into standard operating procedures for GUI agents, targeting domain-specific workflows with implicit knowledge and software-specific conventions.

• WiP: Characterizing and Defending Against Mobile-Agent-Driven MFA Automation

  • Yimeng Liu, Hua Huang
  • 🏛️ Institutions: University of California, Merced
  • 📅 Date: September 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [safety], [security], [multi-factor authentication], [risk signal]
  • 📖 TLDR: This work-in-progress paper identifies "factor collapse" -- a mobile GUI agent completing all authentication factors inside one autonomous environment, completing 10/10 authorized MFA workflows -- and proposes a motion-based Android risk signal for separating human from agent-driven logins.

• Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications

  • Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan, Vikas Ashok
  • 🏛️ Institutions: Stony Brook University, Old Dominion University
  • 📅 Date: September 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [accessibility], [blind users], [failure taxonomy], [OLLA]
  • 📖 TLDR: A three-week diary study with 8 blind users of OLLA, a screen-reader-accessible computer-use-agent prototype, collects 1,258 commands across 12 applications; GPT-5 reaches 52.5% success, with a failure taxonomy spanning grounding, planning, constraint tracking, and termination.

• SCAFFOLD: Self-Improving Web Agents via Recursive Parametric Skill Abstraction

  • Bowei He, Xiaokun Zhang, Meng Ding, Xue Liu
  • 🏛️ Institutions: MBZUAI, McGill University, CityU, UMass Boston
  • 📅 Date: August 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [self-improvement], [skill abstraction], [hierarchical skills], [SCAFFOLD]
  • 📖 TLDR: SCAFFOLD induces parametric, executable skills from successful trajectories under a multi-instance abstraction constraint and maintains a recursively composed hierarchy where higher-level skills invoke lower-level ones, for visual web agents.

• SIR: Self-improving Red-teaming for Compute Use Agents

  • Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho
  • 🏛️ Institutions: CUHK, IBM Research, University of Zagreb, Radboud University
  • 📅 Date: August 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [General GUI]
  • 🔑 Key: [safety], [security], [red-teaming], [prompt injection], [SIR]
  • 📖 TLDR: SIR is a black-box indirect-prompt-injection attack that composes stealthy injections from a reusable principle library and self-improves over rounds, arguing that fixed hand-written injection benchmarks underestimate adaptive adversaries against computer-use agents.

• When and What to Teach: Budget-Aware Online Adaptation for Web Agents

  • Jianwei Zhang, Sihan Cao, Pengcheng Zheng, Ya Wen, Pei Ke, Kuien Liu, Shen Gao, Wei Dong, Yang Yang, Chaoning Zhang
  • 🏛️ Institutions: University of Electronic Science and Technology of China, Institute of Software, Chinese Academy of Sciences, Xi'an University of Architecture and Technology
  • 📅 Date: August 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [online adaptation], [budget-aware teaching], [distillation]
  • 📖 TLDR: Score-Guided Online Teaching with Budgeted Trajectory Trimming does post-deployment online teaching of a lightweight local web agent by a stronger teacher, avoiding budget spent on unresolvable episodes and redundant turns.

• ActReal: System-Level Mobile Agents Challenge Mobile Automation Detection

  • Mingshuo Wang, Hanqing Guo, Huining Li, Yuliang Fu, Jing Xu, Chenhan Xu
  • 🏛️ Institutions: North Carolina State University, Indiana University
  • 📅 Date: August 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [security], [automation detection], [sensor spoofing], [ActReal]
  • 📖 TLDR: ActReal shows a privileged system-level mobile agent can jointly synthesize time-aligned touch and six-axis IMU signals, defeating automation detectors that rely on touch trajectories, timing, and touch-IMU physical coupling.

• GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

  • Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong
  • 🏛️ Institutions: Yale University, ZJU, China University of Geosciences, DAMO Academy, Alibaba Group, Tongji University, UC San Diego
  • 📅 Date: August 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [world model], [contextual consistency], [GUI-CC]
  • 📖 TLDR: GUI-CC evaluates GUI world models as multi-step environments rather than one-step next-screen predictors, via an offline reference-action track over real mobile trajectories and an online agent-loop track that stresses contextual consistency over long horizons.

• Learning Simple Test-Time Environments for LLM Web Agents

  • Junxuan Li, Zijun Liu, Ziyi Huang, Peng Li, Yuzhou Liu, Ming Yan, Yang Liu
  • 🏛️ Institutions: Jilin University, Tsinghua, Institute for AI Industry Research, Tsinghua, Beijing Jiaotong University, Tongyi Lab, Alibaba Group
  • 📅 Date: August 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [test-time adaptation], [environment decomposition], [label-free]
  • 📖 TLDR: Test-Time Environment Decomposition is a label-free method with trial steps that let a web agent decompose a complex environment observation into sub-modules and adapt its behavior during inference, without additional training.

• CURA: Certified Runtime Alarms for Computer-Use Agents

  • Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi
  • 🏛️ Institutions: University of Illinois Chicago, Intel Labs, Capital One AI Labs
  • 📅 Date: August 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [runtime monitoring], [false-success detection], [sequential testing], [CURA]
  • 📖 TLDR: Documenting that 90% of an OSWorld pipeline's failures end with a false success claim, CURA proposes an external monitor reading only harness-visible telemetry, with no access to model internals or extra LLM calls, that turns a running trajectory into a sequential statistical test.

• Iron: Intent-Aligned and Retrospective Dual Learning Framework for Enhancing Generalist Virtual Agents

  • Jiahe Ying, Wendong Bu, Kaihang Pan, Bingchen Miao, Siyu Chen, Wen Wang, Xueming Jiang, Juncheng Li, Siliang Tang
  • 🏛️ Institutions: Fudan, ZJU
  • 📅 Date: August 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [framework], [reward shaping], [cycle-consistency], [Iron]
  • 📖 TLDR: Iron is an annotation-efficient GUI-agent training framework using a stepwise cycle-consistent reward to align low-level actions with high-level intents, plus retrospective reuse of failed trajectories.

• ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

  • Rui Xie, Lu Chen
  • 🏛️ Institutions: X-LANCE Lab, Shanghai Jiao Tong University, BIGAI
  • 📅 Date: August 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [structured state], [semantic actions], [ASIL]
  • 📖 TLDR: ASIL argues screenshot-and-click interaction is state-incomplete and brittle, and introduces an agent-native interaction layer exposing software via structured JSON observations and code-executable semantic actions, instantiated across 15 applications with a 300-task benchmark.

• UI-Venus-2 Technical Report

  • Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jianrong Jiang, Jianguo Li, Runze Li, Jinzhen Lin, Zhenyu Ma, Changhua Meng, Han Peng, Xinyu Qiu, Shuheng Shen, Zhongyi Shui, Weiqiang Wang, Ming Wen, Zhuoer Xu, Hang Yan, Kaiwen Yang, Ruilin Yao, Nanjun Yu, Zhengwen Zeng, Lianrui Zhang, Yunzhu Zhang, Zhe Zhao, Beitong Zhou
  • 🏛️ Institutions: Ant Group
  • 📅 Date: August 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [model], [foundation agent], [reward verification], [UI-Venus-2]
  • 📖 TLDR: UI-Venus-2 is a general-purpose foundation GUI agent for mobile, web, and desktop under a unified closed-loop reasoning-action framework, scaling environments (170+ multilingual mobile apps plus native desktop OSes), task construction, and reward verification.

• WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning

  • Yu Han, Tianwen Qian
  • 🏛️ Institutions: East China Normal University
  • 📅 Date: August 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reinforcement learning], [world model], [reasoning], [WM-R1]
  • 📖 TLDR: WM-R1 trains mobile GUI agents with a world model substituting for the real Android environment during RL rollouts, embedding the world model into the reasoning process so the agent predicts the consequences of candidate actions before acting.

• LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents

  • Weiming Li, Helen Paik, Yulei Sui
  • 🏛️ Institutions: University of New South Wales
  • 📅 Date: August 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [local deployment], [control state], [small models], [LocalLSTC]
  • 📖 TLDR: Replacing GPT-5 with Qwen3.5-9B drops average OSWorld SR-100 from 60.9% to 37.7% with control failures in 91.6% of failed trajectories; LocalLSTC is a training-free architecture that makes persistent control state explicit to stabilize locally deployed GUI agents.

• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

  • Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe
  • 🏛️ Institutions: FAIR at Meta
  • 📅 Date: August 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Desktop]
  • 🔑 Key: [benchmark], [safety], [trustworthiness], [disambiguation], [ADeptS-Bench]
  • 📖 TLDR: ADeptS-Bench is a dual-stream trustworthiness benchmark for computer-use agents on mobile and desktop: a Safety stream with paired benign/malicious tasks and threats embedded in the visual interface, and a Disambiguation stream testing whether agents seek clarification; no model exceeds 80% success while staying under 30% attack success.

• Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

  • Guo Gan, Yilun Zhao, Cong Chen, Jinbiao Wei, Tingyu Song, Zheyuan Yang, Lin Fu, Hong Zhou
  • 🏛️ Institutions: ZJU, Yale University, University of Chinese Academy of Sciences, Tongji University
  • 📅 Date: August 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [AnTrap], [benchmark], [reinforcement learning], [safety], [security], [GRPO], [AndroidWorld]
  • 📖 TLDR: AnTrap is an AndroidWorld-based benchmark that injects controllable runtime anomalies, organized into four categories with ten subcategories, into Android GUI-agent trajectories. Testing 16 leading GUI agents reveals substantial robustness degradation under these anomalies, and adversarial RL training can address many single-step traps but struggles with deep contextual failures such as state deadlock.

• BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes

  • Fei Tang, Huawen Shen, Zhiqiong Lu, Zhengxi Lu, Pengyuan Lyu, Chengquan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
  • 🏛️ Institutions: ZJU, Tencent
  • 📅 Date: August 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [data synthesis], [parallel sandboxes], [BrowserForge]
  • 📖 TLDR: BrowserForge generates pixel-based web-interaction training data at scale by driving many browser sandboxes in parallel over the open web, rather than staying bound to predefined site lists or tutorial sources.

• Reflection with Action-Induced Visual Differences for Desktop GUI Agents

  • Yijie Ma, Chaoyue Niu, Fan Wu, Guihai Chen
  • 🏛️ Institutions: SJTU
  • 📅 Date: August 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [reflection], [outcome verification], [Planner-Operator-Reflector]
  • 📖 TLDR: Evidence-First Reflection is a two-stage reflector for the Planner-Operator-Reflector agent loop that decouples detection of action-induced visual differences from outcome verification, targeting dense desktop GUIs where state changes are subtle.

• Task-Adaptive Rubrics for GUI Reward Modeling

  • Tao Xiong, Xavier Hu, Wenkai Wang, Qinzhuo Wu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, Shengyu Zhang
  • 🏛️ Institutions: ZJU, MiLM Plus, Xiaomi
  • 📅 Date: August 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reward model], [task-adaptive rubric], [outcome verification], [reinforcement learning], [AdaptRubric]
  • 📖 TLDR: AdaptRubric makes GUI reward verification task-adaptive by routing each instruction to a task family for coarse rubric retrieval and then generating instance-level checks for concrete values, scopes, and constraints. It improves offline reward F1 by 3.6 points over the matched baseline average and raises downstream task success by 4.23 points in online reinforcement learning.

• WebMCP-Phalanx: Enforcing and Characterizing Trust Boundaries for Browser-Integrated LLM Agents

  • Lin-Fa Lee, Yi-Yu Chang, Kuo-Hui Yeh
  • 🏛️ Institutions: National Yang Ming Chiao Tung University
  • 📅 Date: August 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [safety], [security], [WebMCP], [trust boundary], [prompt injection]
  • 📖 TLDR: WebMCP-Phalanx is a dual-layer runtime for the W3C WebMCP proposal that binds each page-exposed tool to its registering principal with cryptographic capability credentials, addressing subject-attribution spoofing, uncontrolled tool lifecycles, and semantic prompt injection.

• GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

  • Long Zhang, Yuhan Chen, Chaoran Zhang, Wanxia Cao, Kun Huang, Pengzhi Gao, Wei Liu, Jian Luan, Chenliang Li, Lixin Zou
  • 🏛️ Institutions: Wuhan University, Xiaomi
  • 📅 Date: August 24, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reinforcement learning], [reward model], [data synthesis], [goal-state anchors], [GSAR]
  • 📖 TLDR: GSAR addresses the limited diversity of GUI-agent training environments and the unreliability of scalable evaluators with self-evolving task synthesis and goal-state anchors. The anchors identify task-relevant UI elements in successful states to provide accurate rewards, achieving over 90% offline trajectory-verification accuracy and improving agents on AndroidWorld and a new benchmark.

• CONTRAMEM: Learning Self-Evolving Procedural Memory from Contrasting Multi-Model Trajectories

  • Zheyuan Deng, Binghang Lu, Hanqi Feng, Shirley Huang, Dianzhuo Wang, Yuanda Xu, Zhiwei Zhang, Yige Sun, Changhong Mou, Runyu Zhang, Yuexing Hao, Barnabas Poczos, Xiaomin Li
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 23, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [procedural memory], [self-evolution], [multi-model contrastive learning]
  • 📖 TLDR: CONTRAMEM learns self-evolving procedural memory for autonomous computer-use agents by contrasting trajectories produced by multiple different backbone models on the same tasks, extracting procedures that generalize across models rather than overfitting to one model's idiosyncrasies.

• CausalCache: Conditional High-Fidelity Restoration for Long-Horizon GUI Agents

  • Jiaxuan Luo, Zhanfeng Liao, Jiayao Teng, Yuan Wang
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 23, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [memory], [screenshot caching], [long-horizon tasks], [CausalCache]
  • 📖 TLDR: CausalCache is a budgeted screenshot-fidelity restoration method for long-horizon GUI trajectories, conditionally restoring high-fidelity visual context only where it causally affects the agent's next decision instead of caching every screenshot at full resolution.

• Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents

  • Chengyang Gu, Le Zhang, Jingbo Zhou, Yize Chen, Yu Shi, Siqi Bao, Zheng-Fan Wu, Hua Wu, Hui Xiong
  • 🏛️ Institutions: HKUST(GZ), Baidu Inc., University of Alberta
  • 📅 Date: August 22, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reinforcement learning], [reward gradient], [trajectory length], [GRPO]
  • 📖 TLDR: This work identifies a reward-gradient misalignment in GRPO-style GUI-agent RL caused by trajectory length and proposes length-aware contrastive learning that credits partial progress instead of collapsing every trajectory to a binary success/failure signal.

• GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding

  • Md Abrar Jahin, Md Rizwan Parvez
  • 🏛️ Institutions: USC, Qatar Computing Research Institute
  • 📅 Date: August 22, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [spatial reasoning], [diagnostic benchmark], [GUI-Primitives]
  • 📖 TLDR: GUI-Primitives is a 994-item contrastive spatial-relation benchmark that isolates spatial-reasoning failures in vision-language GUI grounding from other sources of grounding error.

• Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

  • Qijia Chen, Giulio Jacucci
  • 🏛️ Institutions: University of Helsinki
  • 📅 Date: August 22, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Web]
  • 🔑 Key: [GUI grounding], [lexical coupling], [embedding analysis]
  • 📖 TLDR: This work shows that sentence-embedding similarity used for GUI-element grounding is confounded by lexical coupling -- recovery of the visible on-screen label -- rather than genuine semantic understanding, across both mobile and web interfaces.

• Spine-Branch Coordination for Multi-agent Computer Use

  • Mian Zhang, Manasi Sharma, Sheng Zhang, Minglai Yang, Kejian Shi, Ying Liu, Zhiyu Zoey Chen, Daniel Yue Zhang
  • 🏛️ Institutions: Scale AI, JHU, University of Texas at Dallas
  • 📅 Date: August 22, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [multi-agent], [coordination], [parallel virtual machines]
  • 📖 TLDR: This work proposes spine-branch coordination for multi-agent computer use, splitting a long-horizon task between a persistent coordinating spine agent and short-lived branch agents that each execute in a parallel virtual machine.

• Training Needs Trustworthy Worlds: Verified Synthetic Web Environments for Agent Learning

  • Chenghao Zhang, Canran Xiao, SaiSai Hu, Dan Roth
  • 🏛️ Institutions: University of Pennsylvania, Sun Yat-sen University, Pace University
  • 📅 Date: August 22, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [synthetic environments], [data verification], [training data]
  • 📖 TLDR: This work argues that synthetic web environments used to train agents must themselves be verified for trustworthiness, and proposes a pipeline that checks generated environments and trajectories for correctness before they are used as training data.

• Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation

  • Pengshuai Yang, Zijing Gao, Xue Yu, Benhui Zhuang, Bo Yuan, Junlan Feng
  • 🏛️ Institutions: Jiutian Research, China Mobile
  • 📅 Date: August 21, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [automated evaluation], [trajectory evaluation], [step-level reasoning]
  • 📖 TLDR: This work automates mobile-agent trajectory evaluation by reasoning about the consequence of each step individually and aggregating step-level judgments into a trajectory-level verdict, rather than scoring only the final outcome.

• Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

  • Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu
  • 🏛️ Institutions: Tsinghua, Alibaba Group
  • 📅 Date: August 21, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [real-world scenarios], [GMA]
  • 📖 TLDR: GMA benchmarks general mobile assistants in challenging real-world scenarios, positioned against prior AndroidWorld/MobileWorld-style benchmarks by emphasizing scenario realism over curated task sets.

• Inducing Task Models from Computer-Use Traces

  • Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
  • 🏛️ Institutions: Stanford, CMU
  • 📅 Date: August 20, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [task model induction], [screenshot traces], [symbolic representation]
  • 📖 TLDR: This work induces symbolic task models from raw computer-use traces -- screenshots plus mouse and keyboard events -- turning unstructured demonstration recordings into reusable, inspectable procedures.

• ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents

  • Tianchen Guan, Xinlei Lin, Royce Cheng-Yue, Xiangjun Wang, Shuyan Zhou
  • 🏛️ Institutions: Duke University, Amazon AGI SF Lab
  • 📅 Date: August 18, 2026
  • 📑 Publisher: COLM 2026
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [component-level evaluation], [failure diagnosis], [ComponentBench]
  • 📖 TLDR: ComponentBench targets the under-instrumented middle layer between long-horizon workflow benchmarks and atomic GUI-grounding tests, with a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified, component-centered tasks on modern web UIs.

• MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps

  • Sujin Chen, Lijun Li, Tianyi Du, Jing Shao
  • 🏛️ Institutions: Shanghai AI Laboratory
  • 📅 Date: August 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [safety], [security], [environmental injection], [MobileWorldSafety]
  • 📖 TLDR: MobileWorldSafety benchmarks Android GUI-agent safety against environmental injection -- indirect prompt injection and adversarial instructions delivered through everyday mobile channels such as notifications, pop-ups, and shared content.

• CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

  • Linqiang Guo, Li Gu, Zihuan Jiang, Zhixiang Chi, Siobhan Reid, Ziqiang Wang, Yuanhao Yu, Wei Liu, Yang Wang, Tse-Hsun (Peter) Chen
  • 🏛️ Institutions: Concordia University, Mila, University of Toronto, McMaster University
  • 📅 Date: August 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [CoAdapt-GUI], [test-time adaptation], [workflow context], [GRPO], [AndroidWorld-Generalization]
  • 📖 TLDR: Mobile GUI agents stay brittle on applications absent from source training, so CoAdapt-GUI performs test-time adaptation that jointly updates a structured workflow context and the policy from the agent's own target-app rollouts and rewards. The workflow context retains transferable procedures, failure modes, and verification rules while excluding app-bound source details, and policy adaptation uses a task-context-matched group-relative objective over a LoRA adapter on a frozen VLM. It reaches 45.0% on AndroidWorld-Generalization against 37.5% for the reported Policy-Only TTA baseline, and lifts AndroidWorld Plus from 38.6% to 52.9%.

• Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

  • Yuke Li, Xuehan Hou
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [layout-aware matching], [coordinate hallucination]
  • 📖 TLDR: A regression-free GUI grounding pipeline lets a frozen MLLM turn an instruction into a structured visual description, then matches that description against layout-prior candidates. The paper reports gains on ScreenSpot-Pro and Mind2Web while avoiding coordinate-regression fine-tuning.

• Software Engineering for and with GUI Agent

  • Shengcheng Yu, Yuchen Ling, Junyang Xing, Quan Zhou, Chunrong Fang, Zhenyu Chen
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [survey], [software engineering], [reliability], [lifecycle]
  • 📖 TLDR: This survey reviews 336 GUI-agent papers through a software-engineering lens, covering architectures, evaluation, lifecycle concerns, and deployment gaps. It identifies recurring weaknesses in recovery, safety enforcement, auditability, testing, and maintenance.

• AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents

  • Weikai Xu, Yunren Feng, Haoxiang Lei, Kun Huang, Yuxuan Liu, Kang Zhao, Xiaolin Hu, Shuo Shang, Bo An
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [world model], [executable HTML], [mobile agents], [AppDeltaWorld]
  • 📖 TLDR: AppDeltaWorld models mobile GUI transitions as constrained executable HTML updates rather than unconstrained next-screen generation. It uses retrieved app structures and predicted next-screen content to support a mobile-agent training environment and test-time adaptation.

• Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents

  • Jiaming Wei, Zekun Wu, Adriano Koshiyama, Maria Perez-Ortiz
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [observation routing], [multimodal observation], [WebArena], [VisualWebArena]
  • 📖 TLDR: This paper measures when web agents should use text, pixels, or both. Repeated runs show that apparent oracle routing gains are heavily affected by execution noise; learned routers do not consistently beat a fixed mode, while unsolved-task routing can still reduce cost without lowering success.

• StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection

  • Zhuoxin Zhan, Akbar Rafiey, Avery Ma, Leila Pishdad, Layla El Asri
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [safety], [indirect prompt injection], [StepJack]
  • 📖 TLDR: StepJack studies indirect prompt injection chains in which a harmful goal is split into seemingly innocuous steps across linked pages. Its 480-example benchmark evaluates whether computer-use agents follow these multi-step attacks and releases the attack-generation pipeline.

• StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

  • Linqiang Guo, Wei Liu, Li Gu, Yang Wang, Tse-Hsun (Peter) Chen
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reflection], [UI transitions], [structured prediction], [StepReflect]
  • 📖 TLDR: StepReflect recasts after-action reflection for mobile GUI agents as structured transition prediction over paired visual evidence. Its staged training recipe produces a local reflection model that improves several evaluated agent configurations while reducing dependence on repeated frontier-model calls.

• The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents

  • Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Wen Li
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [hindsight distillation], [mobile agents], [privileged information], [GHD]
  • 📖 TLDR: Gated Hindsight Distillation uses the next screenshot as training-only privileged information to rescore an on-policy student's actions. It distills the signal only when that view recovers a missed demonstrated action, improving mobile-agent task success over the compared GRPO baseline.

• LoginTrap: Uncovering Task-Agnostic Phishing-Style Indirect Prompt Injection Attacks against LLM-based Web Agents

  • Longtao Guo, Zelin Zhang, Kaifeng Huang, Yang Shi
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [security], [indirect prompt injection], [authentication], [LoginTrap]
  • 📖 TLDR: LoginTrap is a black-box, task-agnostic attack that uses page-specific indirect instructions to make a controlled login flow appear necessary. The paper evaluates end-to-end credential-oriented attack success across web-agent configurations and defenses.

• GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

  • Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao
  • 🏛️ Institutions: CityU, Tencent Jarvis Lab, Westlake University
  • 📅 Date: August 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI-Lens], [GUI grounding], [coarse-to-fine cropping], [coordinate priming], [visual verification]
  • 📖 TLDR: GUI-Lens turns GUI grounding into an iterative visual-search process: OCR and detected components provide coordinate references, a general-purpose VLM selects progressively enlarged crops, and separate verification gates reject bad crop or click proposals. Across four grounding benchmarks and three VLM backends, the paper reports gains of up to 24.9 percentage points, with GPT-5.5 reaching 87.9% on ScreenSpot-Pro.

• Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents

  • Siqi Fan, Minghao Li, Xiaoqian Ma, Wenhui Tan, Xiusheng Huang, Juntong Wu, Liujie Zhang, Shuo Shang, Weihang Chen
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [MCP], [tool use], [multimodal context], [OSWorld-MCP]
  • 📖 TLDR: This work examines hybrid computer-use agents that can choose screenshots or text tools on OSWorld-MCP. It identifies an adoption gap in tool use and shows that training with a compatible post-tool observation policy can reduce image-context cost while improving the reported operating point.

• Qwen-CUA: Native Computer Use for (almost) Everything

  • Dunjie Lu, Shuai Bai, Tianyi Bai, Sicheng Fan, Chang Gao, Jian Guan, Feng Hu, Mianqiu Huang, Xingyang Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Ning Li, Dayiheng Liu, Shixuan Liu, Zheng Liu, Que Shen, Bowen Wang, Junli Wang, Chencan Wu, Rui Xie, Tianbao Xie, Zhihui Xie, Haiyang Xu, An Yang, Tao Yu, Wenzhen Yuan, Xi Zhang, Zhenru Zhang, Mingkang Zhu, Zhaoqing Zhu, Yizhong Cao, Kai Dang, Binyuan Hui, Kaixin Li, Junyang Lin, Haiquan Wang, Zekun Wang, Yiheng Xu, Fan Yan, Mengqi Yuan, Danyang Zhang, Jiajun Zhang, Zhipeng Zhang, Fan Zhou
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [model], [computer use], [online reinforcement learning], [verifiable tasks], [Qwen-CUA]
  • 📖 TLDR: Qwen-CUA is a screenshot-only native computer-use agent that controls software through keyboard and mouse events without DOM, accessibility, or task-specific APIs. It combines large-scale verifiable task construction, long-horizon trajectory optimization, and visual-history management, and evaluates across desktop and browser computer-use benchmarks.

• CUADebug: Diagnosing and Repairing Computer-Use Agent Failures

  • Weijia Zhang, Kunlun Zhu, Zeyi Liu, Yinting Chen, Tianyi Ma, Jiateng Liu, Jiaxun Zhang, Bingxuan Li, Xiangru Tang, Heng Ji, Jiaxuan You
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [debugging], [benchmark], [error diagnosis], [CUAErrorBench]
  • 📖 TLDR: CUADebug introduces a computer-use failure taxonomy, an OSWorld failure benchmark with human annotations, and an active debugger that inspects suspect before-and-after steps. Its structured diagnoses identify root-cause actions and corrective strategies for subsequent execution.

• MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation

  • Hang Yan, Zhangxuan Gu, Beitong Zhou, Jiaxuan Chen, Runze Li, Yusong Hu, Shuheng Shen, Changhua Meng
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [model fusion], [action distillation], [cross-platform agents], [MAGA]
  • 📖 TLDR: MAGA consolidates domain-specific GUI agents into a cross-platform policy using structured action distillation. It emphasizes incorrectly generated actions while suppressing invalid or unnecessary teacher signals, improving average task success over the compared fusion baselines.

• OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

  • Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [benchmark], [reward model], [trajectory evaluation], [OSReward]
  • 📖 TLDR: OSReward evaluates vision-language reward models for computer-use trajectories across platforms, including a hard subset and fine-grained efficiency/alignment assessment. It also releases OS-Shepherd, an open corpus and reward-model family for lower-cost trajectory judgment.

• Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

  • Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen, Lei Wang, Gaole Dai, Pengxiang Li, Liangyu Chen, Yue Wang, Steven Hoi
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [model], [online reinforcement learning], [real-device training], [Qwen-UI-Agent]
  • 📖 TLDR: Qwen-UI-Agent is a cross-platform foundation GUI agent spanning mobile, computer, and browser use. It combines sandbox and real-device runtimes, mixed GUI/CLI actions, batched action generation, an automated data flywheel, and long-horizon online RL.

• OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

  • Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [office suite], [economic grounding], [OmegaUse-OfficeVal]
  • 📖 TLDR: OmegaUse-OfficeVal benchmarks agents on 100 long-horizon office-suite tasks with code-based verifiers and task-level human labor and price signals. The economic annotations enable comparisons of deliverable quality, inference cost, and execution time against human work.

• Scaling GUI Agents with Visual State Transitions

  • Xiangyan Liu, Kaixin Li, Haonan Wang, Biao Wu, Meng Fang, Longxu Dou, Chao Du, Michael Qizhe Shieh, Tianyu Pang
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [pretraining], [visual state transitions], [world model], [STP]
  • 📖 TLDR: State Transition Pretraining jointly learns inverse dynamics and forward GUI dynamics from visual state transitions before instruction-conditioned trajectory fine-tuning. The paper evaluates the resulting models on desktop and mobile agent benchmarks and releases its official implementation.

• SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

  • Yang Wan, Zhenhao Zhang, Jierui Wang, Linchao Zhu
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [reward model], [reinforcement learning], [trajectory evaluation], [SeekJudge]
  • 📖 TLDR: SeekJudge uses specialized Condense, Ground, Seek, and Analyze roles to judge whether long computer-use trajectories satisfy their instructions. Its distilled shared-backbone design supplies step-level reward signals intended to make model-based supervision practical for online RL.

• StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

  • Yan Yang, Xiangru Jian, Ziyang Luo, Zirui Zhao, Yutong Dai, Ziji Shi, Hanshu Yan, Jun Hao Liew, Silvio Savarese, Junnan Li
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 24, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [program state], [multi-agent harness], [verification], [StateAct]
  • 📖 TLDR: StateAct is a code-first computer-use harness that lets a main agent directly inspect and edit program state, reserving GUI interaction for subgoals that require it. An independent finish gate validates results, and fresh subagents keep long-horizon context focused.

• SEE: Structure-aware Exploring and Exploiting for Long-horizon GUI Agent Trajectory Synthesis

  • Zhuohang Fan, Beichen Zhang, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 20, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [trajectory synthesis], [UI transition graph], [data generation], [SEE]
  • 📖 TLDR: SEE builds UI transition graphs through structured exploration, then synthesizes diverse long-horizon mobile trajectories with graph-based planning and controlled sampling. The method targets coverage of rare transitions while avoiding spurious interaction cycles.

• EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

  • Yaohan Yang, Minglei Shi, Borui Zhang, Jie Zhou, Jiwen Lu
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [state transitions], [visual question answering], [EvoGUI]
  • 📖 TLDR: EvoGUI turns normalized web GUI trajectories into temporal ordering, inverse action/value prediction, and successor-state discrimination tasks. The benchmark is constructed from Mind2Web and WebLINX to isolate state-transition understanding from end-to-end agent success.

• SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

  • Xue Yu, Bo Yuan, Pengshuai Yang, Kailin Zhao, Hong Hu, Junlan Feng
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 17, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [safety], [world model], [risk assessment], [SeerGuard]
  • 📖 TLDR: SeerGuard combines instruction screening with pre-execution action risk assessment for mobile GUI agents. Its safety-augmented world model jointly predicts likely next states and action risk so the agent can reject harmful actions before execution.

• SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL

  • Bowen Lv, Xiao Liu, Yanyu Ren, Hanyu Lai, Bohao Jing, Hanchen Zhang, Yanxiao Zhao, Shuntian Yao, Jie Tang, Yuxiao Dong
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [reinforcement learning], [verifiable task synthesis], [training framework], [SCALECUA]
  • 📖 TLDR: SCALECUA couples Docker-based verifiable task synthesis with frontier sampling and a visual-context segmentation training system for online CUA reinforcement learning. The released framework covers task generation, RL training, and OSWorld/ScienceBoard evaluation.

• StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure

  • Wenyi Wu, Sibo Zhu, Kun Zhou, Aayush Salvi, Zixuan Song, Biwei Huang
  • 🏛️ Institutions: UC San Diego, Aether AI
  • 📅 Date: July 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [long-horizon tasks], [state tracking], [verification], [failure recovery], [StructAgent]
  • 📖 TLDR: StructAgent organizes long-horizon computer use around a unified, evidence-backed representation of task progress shared by its planner, actor, and verifier. Verifier-gated state transitions support checkpointing, evidence-based completion, and targeted recovery across OSWorld-Verified and other digital environments.

• WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

  • Wei Dong, Tianyu Fu, Zhe Yu, Hanning Wang, Anyang Su, Zhizhou Fang, Yuyang Chen, Shuo Wang, Minghui Wu, Ping Jiang, Zhen Lei, Chenxu Zhao
  • 🏛️ Institutions: Unknown
  • 📅 Date: July 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [web agent evaluation], [navigation evaluation], [WebRetriever]
  • 📖 TLDR: WebRetriever provides 1,550 web-agent tasks across 800 websites and evaluates navigation, knowledge-assisted interaction, and end-to-end information extraction. Its NavEval judge incorporates richer interaction context than screenshots alone for fine-grained evaluation.

• Xiaomi-GUI-0 Technical Report

  • Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, Niu Lian, Fazhan Liu, Hui Liu, Heng Qu, Qinzhuo Wu, Zhehao Yu, Tongbo Chen, Shiqi Cui, Anan Du, Shukai Jia, Yuanfa Li, Wei Liu, Yike Liu, Wenchao Lu, Zhenbo Luo, Haoyuan Sun, Jiatong Sun, Cheng Tan, Yajie Wang, Changqiao Wu, Tao Xiong, Jiahui Yang, Yuxuan Yuan, Ruoceng Zhang, Shaojie Zhang, Jian Zhu, Jian Luan, Cong Zou
  • 🏛️ Institutions: Xiaomi
  • 📅 Date: June 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [real-device training], [real-world deployment], [error recovery], [Xiaomi-GUI-0]
  • 📖 TLDR: Xiaomi-GUI-0 trains and evaluates a native multimodal mobile GUI agent in a real-device-dominant closed loop instead of relying only on offline trajectories and simulated environments. Its multi-source data, error-driven flywheel, and progressive SFT/step-RL/agentic-RL pipeline reach 72.0% on the RealMobile benchmark and 78.9% on AndroidWorld, while improving abnormal-state recognition and recovery.

• AOHP: An Open-Source OS-Level Agent Harness for Personalized, Efficient and Secure Interaction

  • Shanhui Zhao, Jiacheng Liu, Guohong Liu, Jichao Yan, Jialei Ye, Yuhao Yang, Hao Wen, Shizuo Tian, Yizhen Yuan, Yuxuan Chen, Yunxin Liu, Ju Ren, Ya-Qin Zhang, Chao Huang, Yao Guo, Yuanchun Li
  • 🏛️ Institutions: Tsinghua, PKU, HKU
  • 📅 Date: June 22, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [AOHP], [framework], [agent-native OS], [information-flow tracking]
  • 📖 TLDR: AOHP is an open-source OS-level agent harness built on AOSP that treats agents as first-class OS actors, introducing personalized service composition, efficient agent interfaces, and secure information-flow tracking while preserving the existing Android software and hardware ecosystem. On tasks covering key OS-agent capabilities it improves task completion by 21.12 points and cuts token cost by 51.55% versus baseline, while enforcing security-policy compliance.

• Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents

  • Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kunwoong Kim, Woomin Song, Chanseok Jeon, Jinwoo Shin
  • 🏛️ Institutions: Unknown
  • 📅 Date: June 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [visual memory], [failure modes], [experiential memory], [grounding error]
  • 📖 TLDR: This paper studies how visual memory affects GUI agents by classifying failures into cognitive failure, visual state misunderstanding, hidden operation blindness, and grounding error. It finds that full-image memory can reduce state-level failures while worsening action-level failures such as hidden operation blindness and grounding errors.

• WebChallenger: A Reliable and Efficient Generalist Web Agent

  • Jayoo Hwang, Xiaowen Zhang, Vedant Padwal
  • 🏛️ Institutions: Independent
  • 📅 Date: June 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [web agent], [PageMem], [selective observation], [site memory], [compound actions], [WebChallenger]
  • 📖 TLDR: WebChallenger is a web-agent framework built around PageMem, a structured page representation that supports selective observation, persistent site memory, and compound action workflows. Without additional training, it combines a 32B LLM and 7B VLM to improve performance across WebArena, VisualWebArena, Online-Mind2Web, and WorkArena.

• Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields

  • Liya Zhu, Jingzhe Ding, Jian Zhang, Jianbo Xue, Shihao Liang, Ge Zhang, Yi Zhu, Duju Zeng, Xiang Gao, Qingshui Gu, Mailun Gao, Huimin Che, Yan Zhao, Peiheng Zhou, Haojun Wang, Chaobo Xian, Lili Le, Chi Wu, Yiwei Liu, Shengda Long, Jiale Yang, Fangzhi Xu, Sijin Wu, Haodong Duan, Chao He, Zhaojian Li, Minchao Wang, Huan Zhou, Jiani Hou, Chuqian Yu, Weiran Shi, Hongwan Gao, Jiamin Chen, Guanhong Chen, Tingqin Luo, Kaiyuan Zhang, Zhixin Yao, Qing Hua, Yuhao Jiang, Jin Chen, Pu Chen, Zhenyu Hu, Xingyu Li, Zhengxuan Jiang, Meng Cao, Tianfeng Long, Haozhe Wang, Mingzhang Wang, Yichen Zhang, Yiming Dai, Chenchen Zhang, Jiaying Wang, Xinying Liu, Xingzu Liu, Lingling Zhang, Xinjie Chen, Yujia Qin, Wangchunshu Zhou, Zhiyong Wu, Yang Liu, Jiaheng Liu, Lei Zhang, Shen Yan, Wenhao Huang, Zaiyuan Wang, Xiaolong Chang
  • 🏛️ Institutions: Unknown
  • 📅 Date: June 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [long-horizon tasks], [professional workflows], [Workflow-GYM]
  • 📖 TLDR: Workflow-GYM is a benchmark for long-horizon computer-use tasks in professional software environments. It evaluates whether agents can complete domain-specific workflows through GUIs and reports that current state-of-the-art models still struggle with end-to-end professional tasks.

• Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms

  • Jiashu Yao, Heyan Huang, Daiqing Wu, Wangke Chen, Huaxi Ai, Haoyu Wen, Zeming Liu, Yuhang Guo
  • 🏛️ Institutions: BIT, THU, Beihang University
  • 📅 Date: June 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dynamic interface], [short-video platform], [observation control], [LivingScreen]
  • 📖 TLDR: LivingScreen is a benchmark for "living-screen-native" GUI agents that must act on continuously updating interfaces such as short-video platforms, where on-screen content changes between agent actions. Evaluating frontier models, it finds none reaches human cost-accuracy performance and identifies over- and under-observation as key failure modes, motivating better observation-control capabilities.

• Online Skill Learning for Web Agents via State-Grounded Dynamic Retrieval

  • Jiaxi Li, Ke Deng, Yun Wang, Jingyuan Huang, Yucheng Shi, Qiaoyu Tan, Jin Lu, Ninghao Liu
  • 🏛️ Institutions: University of Georgia, Tencent America, New York University, PolyU
  • 📅 Date: June 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [SGDR], [online skill learning], [dynamic retrieval], [executable skills], [WebArena]
  • 📖 TLDR: SGDR converts successful WebArena trajectories into verified, executable text-code subprocedures using sliding windows, then retrieves a diverse skill set at every step from both the current page state and task goal rather than once per task. It raises average success to 37.5% with GPT-4.1 and 24.3% with Qwen3-4B, improvements of 3.6 and 2.2 percentage points over the strongest compared baseline.

• Demo2Tutorial: From Human Experience to Multimodal Software Tutorials

  • Zechen Bai, Zhiheng Chen, Yiqi Lin, Kevin Qinghong Lin, Difei Gao, Xiangwu Guo, Xin Wang, Mike Zheng Shou
  • 🏛️ Institutions: NUS
  • 📅 Date: June 02, 2026
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [software tutorials], [task graphs], [planning], [Demo2Tutorial]
  • 📖 TLDR: Demo2Tutorial converts screen recordings and interaction logs into structured multimodal software tutorials with parsed actions, intents, and hierarchical task graphs. The paper evaluates tutorial generation quality and shows that the resulting representations improve downstream GUI-agent planning and generalization.

• Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models

  • Athar Parvez, Muhammad Jawad Mufti, Muqaddas Gull, Omar Hammad
  • 🏛️ Institutions: Unknown
  • 📅 Date: June 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [UI annotation], [mobile UI understanding], [workflow decomposition], [MUIAnno]
  • 📖 TLDR: This paper studies automated mobile UI annotation for UI understanding, accessibility, testing, dataset construction, and GUI agents. It decomposes annotation into context-aware stages with structured prompts and schema-constrained outputs, improving precision on expert-annotated MUIAnno mobile screens.

• STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models

  • Yuhang Han, Wenzheng Yang, Yujie Chen, Xiangqi Jin, Yaojie Zhang, Siteng Huang, Linfeng Zhang
  • 🏛️ Institutions: SJTU, HKUST(GZ), ZJU
  • 📅 Date: June 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [efficiency], [KV cache compression], [vision-language model], [training-free], [STaR-KV]
  • 📖 TLDR: STaR-KV is a training-free KV cache compression framework for GUI vision-language models, whose cache grows linearly with interaction steps. It combines subspace-aware spatial scoring with temporal-stability re-weighting to retain salient tokens, reducing memory while preserving accuracy across GUI agent benchmarks.

• GUI-C²: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

  • Junlong Li, Chao Hao, Lap-Pui Chau, Yi Wang
  • 🏛️ Institutions: PolyU
  • 📅 Date: May 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [reinforcement learning], [difficulty-aware data curation], [coarse-to-fine refinement], [GUI-C²]
  • 📖 TLDR: GUI-C² targets data efficiency and inference overhead in RL-based GUI grounding by combining GUI-D, a difficulty-scored data curation pipeline, with area-gated coarse-to-fine visual refinement. Its improvement-aware stage rewards and simplified decision process achieve 46.4% on ScreenSpot-Pro with a 3B model while adding minimal inference time.

• UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents

  • Yuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [lightweight agents], [knowledge graph], [app exploration], [UI-KOBE]
  • 📖 TLDR: UI-KOBE improves lightweight mobile GUI agents by autonomously exploring apps and building reusable app knowledge graphs. At runtime, the agent uses the graph to identify UI states and choose executable transitions, reducing reliance on large vision-language models for long-horizon planning.

• GUI Agents for Continual Game Generation

  • Yixu Huang, Bo Li, Na Li, Zhe Wang, Kaijie Chen, Haonan Ge, Qingyi Si, Yuanzhe Shen, Ruihan Yang, Guangjing Wang, Hongcheng Guo
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [game generation], [PlaytestArena], [Play2Code], [browser games]
  • 📖 TLDR: This paper studies GUI agents as playtesters for continual browser-game generation. It introduces PlaytestArena, where a GUI agent plays generated games against rubrics, and Play2Code, a loop where game and GUI agents use play feedback to improve generated games.

• AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications

  • Yifan Sui, Xin Huang, Hongbing Li, Fang Xu, Jiahe Lv, Haolong Yan, Yeqing Shen, Litao Liu, Zhimin Fan, Ziyang Meng, Jia Wang, Junbo Qi, Kaijun Tan, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Osamu Yoshie
  • 🏛️ Institutions: BUPT, StepFun, Waseda University
  • 📅 Date: May 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [mobile GUI agent], [closed-source apps], [trajectory evaluation], [AndroidDaily], [GRADE]
  • 📖 TLDR: AndroidDaily is a verifiable benchmark of 350 daily-use tasks across 94 commercial, closed-source Android apps for evaluating mobile GUI agents. It introduces GRADE, an evaluator that judges agents by tracking the visual trajectory against observable external guidelines rather than internal app state, reaching 87.37% agreement with human judgment.

• Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation

  • Heng Qu, Yike Liu, Renren Jin, Wenzong Zhang, Pengzhi Gao, Wei Liu, Jian Luan
  • 🏛️ Institutions: MiLM Plus, Xiaomi
  • 📅 Date: May 26, 2026
  • 📑 Publisher: ICML 2026 (Poster)
  • 💻 Env: [Mobile]
  • 🔑 Key: [dataset], [benchmark], [data scaling], [reasoning], [HyperTrack], [GUIEvalKit]
  • 📖 TLDR: This study examines data scaling, benchmarking, and reasoning for VLM-based mobile GUI agents. It introduces HyperTrack, a dataset of more than 16,000 tasks across over 650 Chinese mobile apps, and GUIEvalKit for unified offline evaluation, finding that reinforcement-based fine-tuning is especially effective for out-of-domain generalization.

• MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research

  • Dingbang Wu, Rui Hao, Haiyang Wang, Shuzhe Wu, Han Xiao, Zhenghong Li, Bojiang Zhou, Zheng Ju, Zichen Liu, Lue Fan, Zhaoxiang Zhang
  • 🏛️ Institutions: CASIA, PKU, CUHK
  • 📅 Date: May 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [framework], [reinforcement learning], [simulation platform], [sim-to-real], [MobileGym]
  • 📖 TLDR: MobileGym is a browser-hosted Android-like simulation platform for mobile GUI agent research that represents full environment state as structured JSON, enabling deterministic state-based judging, snapshot/reset/fork, side-effect detection, and highly parallel rollouts for online RL. Its MobileGym-Bench provides 416 parameterized task templates over 28 apps, and GRPO training on Qwen3-VL-4B-Instruct improves a 256-task test set by +12.8 points with 95.1% sim-to-real gain retention on a real-device subset.

• ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis

  • Guohong Liu, Jialei Ye, Pengzhi Gao, Wei Liu, Jian Luan, Yunxin Liu, Yuanchun Li
  • 🏛️ Institutions: AIR, Tsinghua, University of Electronic Science and Technology of China, MiLM Plus, Xiaomi
  • 📅 Date: May 24, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [benchmark], [environment synthesis], [verifiable rewards], [long-horizon tasks], [ScaleWoB]
  • 📖 TLDR: ScaleWoB uses coding agents to synthesize high-fidelity, backend-free interactive environments with verifiable rewards for GUI-agent evaluation and training across mobile, desktop, and automotive interfaces. Its release covers more than 100 environments and 1,000 tasks, including a 120-task mobile benchmark where current agents average 27.92% success and only 17.82% on long-horizon tasks, compared with 92.08% for humans.

• MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

  • Ziyun Zeng, Hang Hua, Bocheng Zou, Mu Cai, Rogerio Feris, Jiebo Luo
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [memory control], [long-horizon tasks], [MementoGUI], [MementoGUI-Bench]
  • 📖 TLDR: MementoGUI is a plug-in memory framework for long-horizon GUI agents. It introduces MementoCore for online memory selection, compression, and retrieval, plus MementoGUI-Bench and metrics for evaluating semantic action matching, task progress, and memory consistency.

• SE-GA: Memory-Augmented Self-Evolution for GUI Agents

  • Shilong Jin, Lanjun Wang, Zhuosheng Zhang
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 16, 2026
  • 📑 Publisher: ICML 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [memory], [self-evolution], [AndroidWorld], [SE-GA]
  • 📖 TLDR: SE-GA is a memory-augmented self-evolving GUI agent framework for multi-step tasks. It combines Test-Time Memory Extension with a Memory-Augmented Self-Evolution training pipeline and reports improved results on ScreenSpot, AndroidControl-High, and AndroidWorld.

• Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

  • Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, Hao Tian
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [pretraining], [trajectory synthesis], [dataset], [WildGUI], [Video2GUI]
  • 📖 TLDR: Video2GUI is an automated pipeline for extracting grounded GUI interaction trajectories from unlabeled Internet videos. It constructs WildGUI, a dataset of 12 million interaction trajectories across more than 1,500 applications and websites, and uses it for generalized GUI-agent pretraining.

• Executable Agentic Memory for GUI Agent

  • Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, Ju Ren
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [memory], [knowledge graph], [planning], [EAM]
  • 📖 TLDR: Executable Agentic Memory (EAM) represents reusable GUI routines as a structured knowledge graph. It uses state-aware exploration, action-group mining, and value-guided graph search to shift GUI planning from repeated free-form model calls toward retrieval and execution over learned routines.

• How Mobile World Model Guides GUI Agents?

  • Weikai Xu, Kun Huang, Yunren Feng, Jiaxing Li, Yuhan Chen, Yuxuan Liu, Zhizheng Jiang, Heng Qu, Pengzhi Gao, Wei Liu, Jian Luan, Xiaolin Hu, Bo An
  • 🏛️ Institutions: NTU, MiLM Plus, Xiaomi, Renmin University of China, Wuhan University, Xiamen University
  • 📅 Date: May 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [world model], [model-based planning], [data synthesis], [test-time guidance]
  • 📖 TLDR: This study compares delta-text, full-text, diffusion-image, and renderable-code mobile world models and evaluates how their generated rollouts guide GUI agents. Renderable code gives strong in-distribution fidelity and training supervision, while text feedback is more robust for online out-of-distribution execution; generated trajectories also improve end-to-end training, but do not preserve the original data distribution.

• LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning

  • Yubin Wu, Zicheng Cai, Liping Ning, Hua Wang, Zhi Chen, Yaohua Tang, Hao Chen
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [lightweight agents], [reinforcement learning], [distillation], [LiteGUI]
  • 📖 TLDR: LiteGUI proposes an SFT-free training paradigm for compact on-device vision-language GUI agents. It uses Guided On-policy Distillation with oracle trajectories and dynamic retrieval to improve small GUI-agent performance while reducing hallucination and cognitive misalignment.

• Planner Matters! An Efficient and Unbalanced Multi-agent Collaboration Framework for Long-horizon Planning

  • Wenyi Wu, Sibo Zhu, Kun Zhou, Biwei Huang
  • 🏛️ Institutions: UC San Diego
  • 📅 Date: May 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [long-horizon planning], [multi-agent system], [planner-centric reinforcement learning], [compute allocation], [Planner Matters]
  • 📖 TLDR: Planner Matters decomposes long-horizon computer use into a planner, actor, and memory manager, then finds that performance depends disproportionately on planner capacity. Its planner-centric reinforcement learning updates only the planner with trajectory-level VLM-judge rewards while leaving execution and memory components fixed.

• A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction

  • Michito Takeshita, Takuro Kawada, Takumi Ohashi, Shunsuke Kitada, Hitoshi Iyatomi
  • 🏛️ Institutions: Hosei University
  • 📅 Date: May 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [accessibility tree], [observation compression], [efficiency], [A11y-Compressor]
  • 📖 TLDR: A11y-Compressor reformats linearized accessibility-tree observations into compact, structured representations for GUI agents, addressing the cost and noise of raw a11y trees. The compression cuts input tokens to 22% of the original while improving OSWorld task success rate by an average of 5.1 percentage points.

• WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments

  • Jinchao Li, Yunxin Li, Chenrui Zhao, Zhenran Xu, Baotian Hu, Min Zhang
  • 🏛️ Institutions: HIT-Shenzhen
  • 📅 Date: April 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [process-centric], [cross-application], [Windows], [WindowsWorld]
  • 📖 TLDR: WindowsWorld targets the gap that existing GUI benchmarks focus on isolated single-application tasks, presenting a process-centric suite of 181 cross-application desktop tasks (avg 5.0 sub-goals across 17 applications, 78% multi-application). Evaluated computer-use agents fall below 21% success on multi-application tasks, substantially trailing single-application performance and exposing weak workflow-level coordination.

• Step-level Optimization for Efficient Computer-use Agents

  • Jinbiao Wei, Kangqi Ni, Yilun Zhao, Guo Gan, Arman Cohan
  • 🏛️ Institutions: Unknown
  • 📅 Date: April 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [efficiency], [step-level optimization], [model routing], [computer-use agents]
  • 📖 TLDR: This paper targets the cost and latency of computer-use agents that call large multimodal models at nearly every GUI step. It argues that long-horizon trajectories have heterogeneous step difficulty and studies step-level optimization so routine actions can be handled by cheaper policies while high-risk moments receive stronger compute.

• Training Computer Use Agents to Assess the Usability of Graphical User Interfaces

  • Alice Gao, Weixi Tong, Rishab Vempati, Katharina Reinecke, R. Benjamin Shapiro, Tianyi Zhang, Jason Wu
  • 🏛️ Institutions: UW, Purdue University
  • 📅 Date: April 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [model], [usability], [interaction flow prioritization], [uxCUA]
  • 📖 TLDR: uxCUA frames automated GUI usability assessment as a learned agent task by (i) prioritizing important interaction flows, (ii) executing them with human-like interactions, and (iii) predicting a numerical usability score. The trained agent outperforms larger general-purpose LLMs at scoring usability and generating actionable critiques on both experimental and production interfaces.

• AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

  • Hongxin Li, Xiping Wang, Jingran Su, Zheng Ju, Yuntao Chen, Qing Li, Zhaoxiang Zhang
  • 🏛️ Institutions: UCAS, CASIA, PolyU, Shanghai AI Laboratory
  • 📅 Date: April 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [GUI functionality understanding], [region semantics], [state prediction], [AutoGUI-v2]
  • 📖 TLDR: AutoGUI-v2 unifies region-level semantics, element grounding, and state prediction into 2,753 tasks spanning six operating systems, addressing the bifurcation between black-box task-completion and shallow grounding benchmarks. Open-source models excel at functional grounding while commercial models do better at functionality description, but all struggle with complex interaction logic in uncommon actions.

• Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks

  • Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried, Ruslan Salakhutdinov
  • 🏛️ Institutions: CMU
  • 📅 Date: April 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [long-horizon], [multi-site], [rubric evaluation], [Odysseys]
  • 📖 TLDR: Odysseys targets the saturation of short single-site web-agent benchmarks by curating 200 realistic long-horizon multi-site workflows graded with 1,225 rubric items. The benchmark exposes large gaps between frontier computer-use agents and human performance on extended cross-site reasoning and persistent task state.

• VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

  • Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie
  • 🏛️ Institutions: UCSC, CMU, UNC, Salesforce, UC Berkeley
  • 📅 Date: April 23, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [early stopping], [recovery], [VLAA-GUI]
  • 📖 TLDR: VLAA-GUI tackles two recurring failure modes of autonomous GUI agents — premature task termination and unproductive action loops — with a modular framework that decides when to Stop, Recover, and Search. The system reaches 77.5% on OSWorld and 61.0% on WindowsAgentArena, top-performing on both with multiple LLM backbones.

• Human-Guided Harm Recovery for Computer Use Agents

  • Christy Li, Sky CH-Wang, Andi Peng, Andreea Bobu
  • 🏛️ Institutions: Unknown
  • 📅 Date: April 20, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [safety], [harm recovery], [human preferences], [recovery planning]
  • 📖 TLDR: This paper formalizes harm recovery for computer-use agents: steering an agent from a harmful post-execution state back to a safe one aligned with human preferences. It builds a user-study-derived rubric, collects pairwise recovery judgments, and uses a reward model to rank recovery plans.

• GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models

  • Yangyue Wang, Harshvardhan Sikka, Yash Mathur, Tony Zhou, Jinu Nyachhyon, Pranav Guruprasad
  • 🏛️ Institutions: Fig AI, Manifold Research
  • 📅 Date: April 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [GUI grounding], [domain randomization], [robustness], [GUI-Perturbed]
  • 📖 TLDR: GUI-Perturbed introduces a controlled perturbation framework that independently varies visual scenes and instructions to probe grounding robustness beyond static benchmarks. Models reporting >85% on standard benchmarks lose 27-56 points when spatial reasoning is required, exposing systematic brittleness rather than genuine grounding ability.

• UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

  • Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
  • 🏛️ Institutions: ZJU
  • 📅 Date: April 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [training-free], [uncertainty quantification], [adaptive zoom], [UI-Zoomer]
  • 📖 TLDR: UI-Zoomer treats the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate activates zoom-in only when needed, while an uncertainty-driven module picks per-instance crop sizes via variance decomposition. The training-free method improves GUI grounding by 4.2-13.4% on three benchmarks and is compatible with multiple model architectures.

• See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback

  • Himangi Mittal, Gaurav Mittal, Nelson Daniel Troncoso, Yu Hu
  • 🏛️ Institutions: Microsoft, CMU
  • 📅 Date: April 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [multi-turn], [visual feedback], [editing-level grounding], [See-Point-Refine]
  • 📖 TLDR: See-Point-Refine reframes GUI grounding as an iterative loop where the agent points, observes visual feedback, and refines, targeting editing-level grounding in dense coding interfaces that require sub-pixel accuracy. The multi-turn formulation improves grounding precision over single-shot baselines by recovering from initial errors using closed-loop visual evidence.

• ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

  • Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
  • 🏛️ Institutions: ZJU
  • 📅 Date: April 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [framework], [reinforcement learning], [reward model], [GiGPO], [ClawGUI]
  • 📖 TLDR: ClawGUI provides an open-source full-stack GUI agent framework with three components: ClawGUI-RL (online RL training infrastructure for parallel virtual environments and real devices using GiGPO + Process Reward Model), ClawGUI-Eval (standardized evaluation across 6 benchmarks with 95.8% reproduction), and ClawGUI-Agent (multi-OS deployment via 12+ chat platforms). The trained ClawGUI-2B outperforms MAI-UI-2B by 6 points on MobileWorld.

• CocoaBench: Evaluating Unified Digital Agents in the Wild

  • Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, Haoxiang Zhang, Junli Wang, Hexi Jin, Boyuan Zheng, Kun Zhou, Yu Wang, Feng Yao, Licheng Liu, Yijiang Li, Zhifei Li, Zhengtao Han, Pracha Promthaw, Tommaso Cerruti, Xiaohan Fu, Ziqiao Ma, Jingbo Shang, Lianhui Qin, Julian McAuley, Eric P. Xing, Zhengzhong Liu, Rupesh Kumar Srivastava, Zhiting Hu
  • 🏛️ Institutions: UC San Diego, MBZUAI, UMich, UC Berkeley, ETH, University of Cambridge, Gray Swan AI
  • 📅 Date: April 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [unified digital agents], [long-horizon tasks], [visual grounding], [CocoaBench], [CocoaAgent]
  • 📖 TLDR: CocoaBench evaluates unified digital agents on long-horizon tasks requiring flexible composition of vision, search, and coding. Tasks are specified by an instruction and an automatic evaluation function, enabling reliable scalable evaluation across agent infrastructures. The best-evaluated system reaches only 45.1%, exposing gaps in reasoning, tool use, and visual grounding.

• WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark

  • Peng Yuan, Yuyang Yin, Yuxuan Cai, Zheng Wei
  • 🏛️ Institutions: Unknown
  • 📅 Date: April 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [dataset], [automated environment generation], [WebForge], [WebForge-Bench]
  • 📖 TLDR: WebForge automates browser agent benchmark construction via a four-agent Plan-Generate-Refine-Validate pipeline that produces interactive, self-contained web environments without human annotation. It releases WebForge-Bench (934 tasks across 7 domains and 3 difficulty levels) with seven-dimensional difficulty control that enables systematic capability profiling beyond aggregate scores.

• The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents

  • Xuwei Ding, Skylar Zhai, Linxin Song, Jiate Li, Taiwei Shi, Nicholas Meade, Siva Reddy, Jian Kang, Jieyu Zhao
  • 🏛️ Institutions: USC, McGill University, Mila
  • 📅 Date: April 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [safety], [security], [unintended attacks], [OS-BLIND]
  • 📖 TLDR: OS-BLIND benchmarks computer-use agents under unintended attack scenarios where benign instructions trigger harmful outcomes through environmental context. Most agents exceed 90% attack success rate, and even safety-aligned Claude 4.5 Sonnet reaches 73%. Existing safety defenses activate only initially and fail to re-engage during execution, especially when subtask decomposition obscures harmful intent.

• The Amazing Agent Race: Strong Tool Users, Weak Navigators

  • Zae Myung Kim, Dongseok Lee, Jaehyung Kim, Vipul Raheja, Dongyeop Kang
  • 🏛️ Institutions: University of Minnesota
  • 📅 Date: April 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [DAG puzzles], [navigation], [tool use], [Wikipedia], [AAR]
  • 📖 TLDR: The Amazing Agent Race introduces 1,400 DAG-puzzle legs that require fork-merge tool chains over Wikipedia, distinguishing navigation from tool-use ability. The best agent reaches only 37.2%, with navigation errors dominating (27-52% of trials) while tool-use errors stay below 17%, revealing a navigation blind spot invisible to linear benchmarks.

• CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation

  • Yushi Feng, Junye Du, Qifan Wang, Zizhan Ma, Qian Niu, Yutaka Matsuo, Long Feng, Lequan Yu
  • 🏛️ Institutions: HKU, CUHK, University of Tokyo
  • 📅 Date: April 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [safety], [benchmark], [conformal risk control], [Phone-Harm], [CORA]
  • 📖 TLDR: CORA reformulates safety as selective action execution: a Guardian model estimates action-conditional risk, Conformal Risk Control calibrates an execute/abstain boundary under a user-specified risk budget, and a Diagnostician proposes interventions for rejected actions. A Goal-Lock mechanism resists visual injection. Phone-Harm benchmark with step-level harm labels is also released.

• EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning

  • Tiantian He, Yihang Chen, Keyue Jiang, Ka Yiu Lee, Kaiwen Zhou, Kun Shao, Shuai Wang
  • 🏛️ Institutions: Huawei Noah's Ark Lab
  • 📅 Date: April 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [MCP], [self-evolving], [experience learning], [hybrid policy], [EE-MCP]
  • 📖 TLDR: EE-MCP frames computer-use agent design as hybrid policy learning that balances GUI interaction and MCP API calls, with an automated pipeline for environment generation, trajectory collection, and gap-driven task synthesis. An experience bank of LLM-learned rules enables inference-time improvement: distillation wins on MCP-dominant tasks (+17.8pp) while the experience bank excels on GUI-intensive tasks (+10.0pp).

• HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks

  • Suhana Bedi, Ryan Welch, Ethan Steinberg, Michael Wornow, Taeil Matthew Kim, Haroun Ahmed, Peter Sterling, Bravim Purohit, Qurat Akram, Angelic Acosta, Esther Nubla, Pritika Sharma, Michael A. Pfeffer, Sanmi Koyejo, Nigam H. Shah
  • 🏛️ Institutions: Stanford
  • 📅 Date: April 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [healthcare], [HealthAdminBench], [EHR], [long-horizon tasks]
  • 📖 TLDR: HealthAdminBench evaluates computer-use agents on healthcare administration via 4 realistic GUI environments (EHR, two payer portals, fax) and 135 expert-defined tasks decomposed into 1,698 subtasks. The best agent (Claude Opus 4.6 CUA) reaches only 36.3% end-to-end despite 82.8% subtask success, exposing a large gap to real-world reliability.

• Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection

  • Wenkui Yang, Chao Jin, Haisu Zhu, Weilin Luo, Derek Yuen, Kun Shao, Huaibo Huang, Junxian Duan, Jie Cao, Ran He
  • 🏛️ Institutions: UCAS, CASIA, Huawei, ShanghaiTech University
  • 📅 Date: April 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [safety], [red teaming], [visual grounding], [UI injection]
  • 📖 TLDR: This paper proposes Semantic-level UI Element Injection, a red-teaming method that overlays safety-aligned UI elements onto screenshots to misdirect GUI agents' visual grounding. Using a modular Editor-Overlapper-Victim pipeline with iterative search, optimized attacks improve attack success rate by up to 4.4x over random injection and transfer across models.

• ClawBench: Can AI Agents Complete Everyday Online Tasks?

  • Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, Huaisong Zhang, Xian Wu, Yi Lu, Minyi Lei, Kai Zou, Huifeng Yin, Ping Nie, Liang Chen, Dongfu Jiang, Wenhu Chen, Kelsey R. Allen
  • 🏛️ Institutions: UBC, Vector Institute, CMU, UWaterloo, SJTU, ZJU, HKUST, Tsinghua
  • 📅 Date: April 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [realistic website], [long-horizon tasks], [ClawBench]
  • 📖 TLDR: ClawBench evaluates browser agents on 283 everyday online tasks (V1 153 + V2 130) across 163 live production websites spanning purchases, bookings, and job applications. A lightweight interception layer blocks final submissions for safe evaluation while preserving end-to-end interaction. Its two-stage interception and judge protocol exposes a persistent gap in real-world web automation.

• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation

  • Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, Wenqi Zhang, Xu Tan, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
  • 🏛️ Institutions: ZJU, Apple, Tencent
  • 📅 Date: April 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [personalization], [proactive agents], [KnowU-Bench]
  • 📖 TLDR: KnowU-Bench is an online benchmark for personalized mobile agents on Android emulation with 42 general, 86 personalized, and 64 proactive tasks. It hides user profiles from the agent and forces genuine preference inference through multi-turn dialogues. Even frontier models fall below 50% under vague instructions requiring preference inference.

• MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

  • Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, Ranjay Krishna
  • 🏛️ Institutions: AI2, UW, UNC
  • 📅 Date: April 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [model], [dataset], [MolmoWeb], [open-source]
  • 📖 TLDR: MolmoWeb is a family of fully open multimodal web agents (4B and 8B) trained on MolmoWebMix (100K+ synthetic trajectories and 30K+ human demonstrations). Operating as screenshot-only visual-language action policies without HTML or accessibility tree access, it achieves SOTA on WebVoyager, Online-Mind2Web, and DeepShop, outperforming larger closed models like GPT-4o.

• Preference Redirection via Attention Concentration: An Attack on Computer Use Agents

  • Dominik Seip, Matthias Hein
  • 🏛️ Institutions: University of Tübingen
  • 📅 Date: April 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [security], [safety], [attack], [adversarial patch], [attention manipulation], [PRAC]
  • 📖 TLDR: PRAC is a novel attack on Computer Use Agents that redirects model attention toward a stealthy adversarial patch to alter internal preferences rather than directly manipulating outputs. The attack influences product selection on online shopping platforms and generalizes across fine-tuned variants of the same backbone, highlighting risks for CUAs built on open-weight models.

• Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems

  • Maria Movin, Claudia Hauff, Aron Henriksson, Panagiotis Papapetrou
  • 🏛️ Institutions: Stockholm University, Spotify
  • 📅 Date: April 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [evaluation], [trace-level analysis], [user simulation], [search systems], [behavioral alignment]
  • 📖 TLDR: This paper presents a trace-level evaluation framework comparing human and GUI-agent behavior across task outcome, query formulation, and navigation in a production audio-streaming search application. With 39 participants and a state-of-the-art GUI agent on 10 multi-hop search tasks, the agent matches task success but follows search-centric, low-branching strategies versus humans' content-centric exploration.

• Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions

  • Guo Gan, Yuxuan Ding, Cong Chen, Yuwei Ren, Yin Huang, Hong Zhou
  • 🏛️ Institutions: Unknown
  • 📅 Date: April 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reinforcement learning], [reward model], [training efficiency], [process reward model], [Android Coach]
  • 📖 TLDR: Android Coach shifts online RL training from Single State Single Action to Single State Multiple Actions by learning a critic that estimates action values and integrating a process reward model with group-wise advantage estimation. It improves UI-TARS-1.5-7B by 7.5% on AndroidLab and 8.3% on AndroidWorld with 1.4x higher training efficiency than PPO and GRPO.

• GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

  • Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, Mike Zheng Shou
  • 🏛️ Institutions: NUS, Oxford
  • 📅 Date: April 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [game agent], [computer-use control], [semantic control], [GameWorld]
  • 📖 TLDR: GameWorld is a standardized browser-based benchmark for multimodal game agents with 34 games and 170 tasks. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.

• What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

  • Songze Li, Xiaoke Guo, Tianqi Liu, Biao Yi, Zhaoyan Gong, Zhiqiang Liu, Huajun Chen, Wen Zhang
  • 🏛️ Institutions: ZJU
  • 📅 Date: April 08, 2026
  • 📑 Publisher: Findings of ACL 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [benchmark], [UILoop], [UI comprehension]
  • 📖 TLDR: UILoop treats GUI reasoning as a cyclic Screen-UI elements-Action process, enabling MLLMs to explicitly learn the localization, semantic functions, and usage of key UI elements. It introduces UI Comprehension-Bench (26K samples) and achieves state-of-the-art GUI reasoning performance with improved interpretability.

• Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

  • Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu, Huanmin Xu, Qiutong Pan, Haiwei Wang
  • 🏛️ Institutions: Unknown
  • 📅 Date: April 07, 2026
  • 📑 Publisher: ACL 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [reinforcement learning], [GRPO], [action verification], [self-correction], [VeriGUI], [AndroidControl]
  • 📖 TLDR: VeriGUI treats action-effect verification as a first-class RL objective to handle non-deterministic GUI environments with network delays, rendering lags, and system failures. A Thinking-Verification-Action-Expectation framework identifies failures; two-phase training with Robust SFT and GRPO using asymmetric verification rewards reduces failure loops. A new Robustness Benchmark built on AndroidControl evaluates failure recognition and correction.

• Gym-Anything: Turn any Software into an Agent Environment

  • Pranjal Aggarwal, Graham Neubig, Sean Welleck
  • 🏛️ Institutions: CMU
  • 📅 Date: April 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [dataset], [Gym-Anything], [CUA-World], [long-horizon tasks]
  • 📖 TLDR: Gym-Anything converts any software into an interactive computer-use environment via multi-agent setup and audit. It produces CUA-World with 10K+ long-horizon tasks spanning medical science, astronomy, and enterprise systems, plus CUA-World-Long with tasks requiring 500+ steps, far exceeding existing benchmarks.

• MAESTRO: Adapting GUIs and Guiding Navigation with User Preferences in Conversational Agents with GUIs

  • Sangwook Lee, Sang Won Lee, Adnan Abbas, Young-Ho Kim, Yan Chen
  • 🏛️ Institutions: Virginia Tech, NAVER AI Lab
  • 📅 Date: April 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [personalization], [MAESTRO], [preference], [GUI adaptation]
  • 📖 TLDR: MAESTRO extends GUI agents from task execution to decision support by maintaining a shared preference memory. It provides Preference-Grounded GUI Adaptation (augment, sort, filter, highlight) and Preference-Guided Workflow Navigation that detects preference conflicts and proposes backtracking.

• WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

  • Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique, Kassem Fawaz
  • 🏛️ Institutions: UW-Madison
  • 📅 Date: April 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [security], [privacy], [WebSP-Eval]
  • 📖 TLDR: WebSP-Eval is the first framework evaluating web agents on user-facing website security and privacy tasks such as cookie preferences, privacy settings, and session revocation. Across 200 task instances on 28 websites, agents fail more than 45% on tasks with stateful UI elements like toggles and checkboxes.

• GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis

  • Yuwen Zhai, Runze Li, Liang Wang, Nian Shi, Liwu Xu, Wei Zhang, Ran Lin, Bo Xu, Benlei Cui
  • 🏛️ Institutions: Unknown
  • 📅 Date: April 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [evaluation], [trajectory diagnosis], [hierarchical diagnosis], [error analysis], [GUIDE]
  • 📖 TLDR: GUIDE decomposes GUI agent trajectory evaluation into three sequential stages — trajectory segmentation, subtask diagnosis, and structured error analysis — mirroring the compositional structure of GUI tasks. Evaluated on 932 industrial e-commerce trajectories, AGENTREWARDBENCH, and AndroidBench, it improves accuracy by up to 5.35 points over baselines while producing diagnostic insights.

• IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents

  • Rongqian Chen, Yu Li, Zeyu Fang, Sizhe Tang, Weidong Cao, Tian Lan
  • 🏛️ Institutions: George Washington University
  • 📅 Date: April 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [reward model], [model], [plan-aware reward], [contrastive alignment], [margin ranking], [OSWorld], [IntentScore]
  • 📖 TLDR: IntentScore is a plan-aware reward model for computer-use agents trained from 398K offline GUI interaction steps across three OSes, using contrastive alignment and margin ranking objectives. It achieves 97.5% pairwise discrimination and, when used as a re-ranker for Agent S3 on OSWorld, improves task success rate by 6.9 points.

• The Art of Building Verifiers for Computer Use Agents

  • Corby Rosset, Pratyusha Sharma, Andrew Zhao, Miguel Gonzalez-Fernandez, Ahmed Awadallah
  • 🏛️ Institutions: MSR
  • 📅 Date: April 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [reward model], [verification], [CUAVerifierBench]
  • 📖 TLDR: Presents lessons from building Universal Verifier for web agent trajectories, based on four principles: meaningful rubrics, separated process/outcome rewards, controllable vs. uncontrollable failure distinction, and divide-and-conquer context management. Reduces false positive rates to near zero compared to WebVoyager (45%+) and WebJudge (22%+).

• AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents

  • Yunhao Feng, Yifan Ding, Yingshui Tan, Xingjun Ma, Yige Li, Yutao Wu, Yifeng Gao, Kun Zhai, Yanming Guo
  • 🏛️ Institutions: Unknown
  • 📅 Date: April 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [safety benchmark], [harmful behavior], [multi-step risk], [AgentHazard]
  • 📖 TLDR: AgentHazard is a benchmark for harmful behavior in computer-use agents. It contains 2,653 instances across risk categories and attack strategies, pairing harmful objectives with locally legitimate steps to test whether agents recognize and interrupt unsafe multi-step behavior.

• The Tool Illusion: Rethinking Tool Use in Web Agents

  • Renze Lou, Baolin Peng, Wenlin Yao, Qianhui Wu, Hao Cheng, Suman Nath, Wenpeng Yin, Jianfeng Gao
  • 🏛️ Institutions: Penn State, MSR
  • 📅 Date: April 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [empirical study], [tool use], [WebArena]
  • 📖 TLDR: An extensive controlled study across diverse tool sources, backbone models, tool-use frameworks, and evaluation benchmarks to determine whether tools provide consistent gains for web agents. Findings revise some prior conclusions and complement others with broader evidence.

• GPA: Learning GUI Process Automation from Demonstrations

  • Zirui Zhao, Jun Hao Liew, Yan Yang, Wenzhuo Yang, Ziyang Luo, Doyen Sahoo, Silvio Savarese, Junnan Li
  • 🏛️ Institutions: Salesforce AI Research
  • 📅 Date: April 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [training-free], [process automation], [long-horizon tasks], [GPA], [robotic process automation]
  • 📖 TLDR: GPA is a vision-based GUI process automation system that enables fast and stable process replay from a single demonstration. Using Sequential Monte Carlo-based localization and readiness calibration, it achieves higher success rates with 10x faster execution than Gemini 3 Pro on long-horizon GUI tasks, running entirely locally without cloud LLMs.

• HippoCamp: Benchmarking Contextual Agents on Personal Computers

  • Zhe Yang, Shulin Tian, Kairui Hu, Shuai Liu, Hoang-Nhat Nguyen, Yichi Zhang, Zujin Guo, Mengying Yu, Zinan Zhang, Jingkang Yang, Chen Change Loy, Ziwei Liu
  • 🏛️ Institutions: NTU
  • 📅 Date: April 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [contextual agent], [personal files], [multimodal file management], [HippoCamp]
  • 📖 TLDR: HippoCamp evaluates contextual agents on personal computers by modeling individual user profiles and requiring search over massive personal file collections to support context-aware reasoning, going beyond app-isolated benchmarks. The benchmark emphasizes multimodal file management as a first-class capability for personal-computing assistants.

• Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants

  • Deepak Nathani, Cheng Zhang, Chang Huan, Jiaming Shan, Yinfei Yang, Alkesh Patel, Zhe Gan, William Yang Wang, Michael Saxon, Xin Eric Wang
  • 🏛️ Institutions: UC Santa Barbara, Apple
  • 📅 Date: April 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [proactive agents], [Pare]
  • 📖 TLDR: Pare models digital apps as finite state machines with stateful navigation to enable realistic active user simulation for proactive agents. Pare-Bench provides 143 diverse tasks spanning communication, productivity, scheduling, and lifestyle apps to test context observation, goal inference, and intervention timing.

• When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigation

  • Henry Peng Zou, Chunyu Miao, Wei-Chieh Huang, Yankai Chen, Yue Zhou, Hanrong Zhang, Yaozu Wu, Liancheng Fang, Zhengyao Gu, Zhen Zhang, Kening Zheng, Fangxin Wang, Yi Nian, Shanghao Li, Wenzhe Fan, Langzhou He, Weizhi Zhang, Xue Liu, Philip S. Yu
  • 🏛️ Institutions: UIC, McGill University, MBZUAI, UCSB, USC
  • 📅 Date: April 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [interruptibility], [InterruptBench], [WebArena]
  • 📖 TLDR: The first systematic study of interruptible agents in long-horizon web navigation. It formalizes three interruption types (addition, revision, retraction) and introduces InterruptBench derived from WebArena-Lite, showing that handling mid-task user interruptions remains challenging for current LLMs.

• PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent

  • Hongyi Nie, Xunyuan Liu, Yudong Bai, Yaqing Wang, Yang Liu, Quanming Yao, Zhen Wang
  • 🏛️ Institutions: Northwestern Polytechnical University, Tsinghua, PKU
  • 📅 Date: March 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dataset], [personalization], [PSPA-Bench]
  • 📖 TLDR: PSPA-Bench evaluates personalization in smartphone GUI agents with 12,855+ personalized instructions across 10 daily-use scenarios and 22 mobile apps. Even the strongest of 11 benchmarked agents performs poorly under personalized settings, highlighting gaps in reasoning, perception, and long-term memory.

• Terminal Agents Suffice for Enterprise Automation

  • Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, Sai Rajeswar
  • 🏛️ Institutions: ServiceNow
  • 📅 Date: March 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [enterprise automation], [terminal agents], [empirical study]
  • 📖 TLDR: This paper shows that a coding agent equipped only with a terminal and filesystem can match or outperform GUI-driven and MCP tool-augmented agents for enterprise automation tasks across ServiceNow, GitLab, and ERPNext, arguing that simple programmatic API interfaces combined with strong foundation models suffice.

• WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale

  • Shuyan Zhou
  • 🏛️ Institutions: Duke University
  • 📅 Date: March 2026
  • 📑 Publisher: Blog Post
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [dataset], [environment synthesis], [verifiable rewards], [reinforcement learning], [WebArena], [WebArena-Infinity]
  • 📖 TLDR: WebArena-Infinity automates the generation of high-authenticity web environments with verifiable tasks from static artifacts like user manuals, using a multi-agent pipeline of coding and browser-use agents. It produces 10 environments with 1,260 tasks and 2,070 trajectories. Agents achieve notably lower success rates than on manually built benchmarks, suggesting the generated tasks capture meaningful complexity.

• GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation

  • Rui Xie, Zhi Gao, Chenrui Shi, Zirui Shang, Lu Chen, Qing Li
  • 🏛️ Institutions: SJTU, State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing Institute of Technology
  • 📅 Date: March 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [video retrieval], [automatic annotation], [domain bias], [training-free], [OSWorld], [GUIDE]
  • 📖 TLDR: GUIDE is a training-free add-on for desktop GUI agents that retrieves relevant tutorial videos, turns them into planning and grounding annotations, and injects that expertise into existing agents without changing model parameters. On OSWorld, it improves multiple agent families while also reducing execution steps.

• Rethinking Token Pruning for Historical Screenshots in GUI Visual Agents: Semantic, Spatial, and Temporal Perspectives

  • Daiqiang Li, Zihao Pan, Zeyu Zhang, Ronghao Chen, Huacan Wang, Honggang Chen, Haiyun Jiang
  • 🏛️ Institutions: Sichuan University, Sun Yat-sen University, Australian National University, PKU, University of Chinese Academy of Sciences
  • 📅 Date: March 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [token pruning], [historical screenshots], [random pruning], [recency effect], [empirical study]
  • 📖 TLDR: This paper studies token pruning for historical GUI screenshots and finds that background regions still carry useful state-transition cues, random pruning preserves spatial structure surprisingly well, and allocating larger token budgets to recent screenshots keeps performance nearly unchanged while reducing cost.

• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding

  • Shrinidhi Kumbhar, Haofu Liao, Srikar Appalaraju, Kunwar Yashraj Singh
  • 🏛️ Institutions: Arizona State University, Amazon (AWS Agentic AI)
  • 📅 Date: March 27, 2026
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [diffusion models], [hybrid masking], [bounding-box prediction], [LLaDA-V], [cross-platform]
  • 📖 TLDR: This paper adapts the discrete diffusion model LLaDA-V to GUI grounding and proposes a hybrid masking schedule for bounding-box prediction. Across web, desktop, and mobile benchmarks, the diffusion model outperforms its linear-masked variant and remains competitive with autoregressive VLMs.

• Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

  • Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, Jie Tang
  • 🏛️ Institutions: Tsinghua, Zhipu
  • 📅 Date: March 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [website development], [agent verification], [UI-to-code], [full-stack development], [Vision2Web]
  • 📖 TLDR: Vision2Web is a hierarchical benchmark for visual website development that spans static UI-to-code, interactive frontend reproduction, and full-stack website construction. It evaluates coding agents with workflow-based verification using a GUI agent verifier and a VLM judge, and shows that current models still struggle badly on full-stack tasks.

• GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks

  • Saelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin, Jae Won Cho, Yale Song, Juho Kim
  • 🏛️ Institutions: KAIST, CMU, Oxford, Konkuk University, Google, SkillBench
  • 📅 Date: March 26, 2026
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [collaborative assistance], [behavior state detection], [intent prediction], [help prediction], [think-aloud data], [GUIDE]
  • 📖 TLDR: GUIDE studies collaborative GUI assistance rather than pure task automation, using 67.5 hours of think-aloud recordings from 120 novice users across 10 software applications. It benchmarks behavior-state detection, intent prediction, and help prediction, and shows that current multimodal models still struggle to infer what users are doing and when intervention would be useful.

• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing

  • Fanheng Kong, Jingyuan Zhang, Yang Yue, Chenxi Sun, Yang Tian, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Jun Du, Wenchong Zeng, Han Li, Kun Gai
  • 🏛️ Institutions: Northeastern University, Kuaishou Technology
  • 📅 Date: March 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [automated testing], [defect detection], [checklist generation], [latent logical defects], [WebTestBench], [WebTester]
  • 📖 TLDR: WebTestBench studies end-to-end automated web testing rather than ordinary task completion, decomposing the problem into checklist generation and defect detection across diverse web applications. Its WebTester baseline shows that current systems still struggle with test completeness, latent logical defects, and long-horizon reliability.

• CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents

  • Xiangru Jian, Shravan Nayak, Kevin Qinghong Lin, Aarash Feizi, Kaixin Li, Patrice Bechard, Spandana Gella, Sai Rajeswar
  • 🏛️ Institutions: ServiceNow, University of Waterloo, Mila, Université de Montréal, McGill University, Oxford, NUS
  • 📅 Date: March 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [dataset], [video demonstrations], [desktop workflows], [grounding dataset], [VideoCUA], [GroundCUA], [UI-Vision], [CUA-Suite]
  • 📖 TLDR: CUA-Suite is a large-scale desktop-agent data ecosystem centered on continuous expert video rather than sparse screenshots. It combines VideoCUA, UI-Vision, and GroundCUA to provide 55 hours of demonstrations, dense grounding annotations, and evaluation data across 87 professional desktop applications where current foundation action models still fail frequently.

• Towards Automated Crowdsourced Testing via Personified-LLM

  • Shengcheng Yu, Yuchen Ling, Chunrong Fang, Zhenyu Chen, Chunyang Chen
  • 🏛️ Institutions: TUM, National Key Laboratory for Novel Software Technology, NJU
  • 📅 Date: March 25, 2026
  • 📑 Publisher: FSE 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [GUI testing], [crowdsourced testing], [persona-guided testing], [bug finding], [PersonaTester]
  • 📖 TLDR: PersonaTester automates crowdsourced GUI testing by injecting empirically derived tester personas into LLM agents. On 15 mobile apps, it reproduces more diverse testing behaviors than non-persona baselines and triggers more crashes and functional bugs.

• UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience

  • Zichuan Lin, Feiyu Liu, Yijun Yang, Jiafei Lyu, Yiming Gao, Yicheng Liu, Zhicong Lu, Yangbin Yu, Mingyu Yang, Junyou Li, Deheng Ye, Jie Jiang
  • 🏛️ Institutions: Tencent Hunyuan
  • 📅 Date: March 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reinforcement learning], [self-evolving], [failed trajectory learning], [RFT], [GRSD], [AndroidWorld], [UI-Voyager]
  • 📖 TLDR: UI-Voyager is a self-evolving mobile GUI agent that learns from failed trajectories instead of manual annotations. Its two-stage training combines rejection fine-tuning with group-relative self-distillation to turn successful rollouts into dense corrective supervision, yielding 81.0% Pass@1 on AndroidWorld with a 4B model.

• AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents

  • Yutao Luo, Haotian Zhu, Shuchao Pang, Zhigang Lu, Tian Dong, Yongbin Zhou, Minhui Xue
  • 🏛️ Institutions: Nanjing University of Science and Technology, Macquarie University, Western Sydney University, HKU, CSIRO Data61
  • 📅 Date: March 24, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [backdoor attack], [notification trigger], [security], [contrastive learning], [Remote Action Execution], [AgentRAE]
  • 📖 TLDR: AgentRAE is a backdoor attack against screenshot-based mobile GUI agents that uses benign-looking notification icons as triggers for remote action execution. Its contrastive-pretraining plus poisoning pipeline preserves clean performance, exceeds 90% attack success over ten mobile operations, and evades eight representative defenses.

• CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

  • Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang
  • 🏛️ Institutions: UIUC
  • 📅 Date: March 23, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [CAPTCHA], [reasoning-action data generation], [self-correction], [native GUI agent], [ReCAP]
  • 📖 TLDR: ReCAP is a native GUI agent specialized for interactive CAPTCHA solving. It builds a seven-type CAPTCHA environment, generates large-scale reasoning-action trajectories plus self-correction data from failed attempts, and improves success from about 30% to 80% without sacrificing general GUI-agent performance.

• Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos

  • Shoubin Yu, Lei Shu, Antoine Yang, Yao Fu, Srinivas Sunkara, Maria Wang, Jindong Chen, Mohit Bansal, Boqing Gong
  • 🏛️ Institutions: Google DeepMind, UNC
  • 📅 Date: March 23, 2026
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [egocentric video], [LLM-as-a-judge], [web planning], [Ego2Web], [Ego2WebJudge]
  • 📖 TLDR: Ego2Web is a benchmark that couples egocentric first-person videos with web tasks requiring real-world visual understanding before online interaction. It also introduces Ego2WebJudge, an LLM-as-a-judge evaluator with about 84% agreement with humans, and shows large headroom for current agents.

• ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents

  • Zijian Lu, Yiping Zuo, Yupeng Nie, Xin He, Weibei Fan, Lianyong Qi, Shi Jin
  • 🏛️ Institutions: Nanjing University of Posts and Telecommunications
  • 📅 Date: March 20, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [agent skill], [contract], [program repair], [verification], [cross-model transfer]
  • 📖 TLDR: ContractSkill turns draft web-agent skills into executable artifacts with explicit contracts, enabling deterministic verification, local fault localization, and patch-based repair instead of full skill rewrites, improving skill reliability on VisualWebArena and MiniWoB while preserving transfer across models.

• AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents

  • Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, Hanqian Li, Yu Huang, Song Dai, Yonghua Hei, Yue Ding, Xiang Li, Shikang Wang, Chengdong Xu, Jingqi Liu, Xueying Ma, Zhiwen Zheng, Xiaofei Zhang, Bincheng Wang, Nichen Yang, Jie Wu, Lihua Tian, Chen Li, Xuming Hu
  • 🏛️ Institutions: XJTU, HKUST(GZ), HKUST, CityU, University of Technology Sydney, Tianjin University, Fudan, Shandong University, CAS, Sun Yat-sen University, Northwestern Polytechnical University
  • 📅 Date: March 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [agent memory], [long-horizon tasks], [Anchored State Memory], [AndroTMem-Bench], [AndroTMem]
  • 📖 TLDR: AndroTMem studies interaction memory in long-horizon Android GUI agents through a 1,069-task benchmark designed to require carrying forward critical intermediate state. It introduces Anchored State Memory, which stores causally linked state anchors and improves completion rates by 5%-30.16% over replay and summary baselines across 12 agents.

• OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

  • Zehao Li, Zhenyu Wu, Yibo Zhao, Bowen Yang, Jingjing Xie, Zhaoyang Liu, Zhoumianze Liu, Kaiming Jin, Jianze Liang, Zonglin Li, Feng Wu, Bowen Zhou, Zun Wang, Zichen Ding
  • 🏛️ Institutions: USTC, Shanghai AI Laboratory, CUHK MMLab, HKUST, NUS
  • 📅 Date: March 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reward model], [reinforcement learning], [critic framework], [OmniGUIRewardBench], [milestone decomposition], [OS-Themis]
  • 📖 TLDR: OS-Themis is a scalable critic framework for GUI reward modeling that breaks trajectories into verifiable milestones and audits the evidence chain before issuing a verdict. It improves AndroidWorld training and filtering loops and introduces OmniGUIRewardBench as a cross-platform benchmark for GUI outcome rewards.

• AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

  • Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, Borui Zhang, Jiwen Lu
  • 🏛️ Institutions: Lenovo Research
  • 📅 Date: March 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [instruction refinement], [adaptive zoom], [GRPO], [bounding-box prediction], [AdaZoom-GUI]
  • 📖 TLDR: AdaZoom-GUI targets two concrete GUI-grounding bottlenecks: ambiguous natural-language instructions and tiny UI elements in high-resolution screenshots. It combines instruction rewriting with a conditional second-stage zoom-in pass and reports state-of-the-art grounding performance among comparable model sizes.

• FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repair

  • Ruize Ma, Yilei Jiang, Shilin Zhang, Zheng Ma, Yi Feng, Vincent Ng, Zhi Wang, Xiangyu Yue, Chuanyi Li, Lewei Lu
  • 🏛️ Institutions: NJU, SenseTime, CUHK, University of Texas at Dallas
  • 📅 Date: March 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [multimodal program repair], [software engineering], [program repair], [failure memory], [SWE-bench Multimodal], [FailureMem]
  • 📖 TLDR: FailureMem is a multimodal automated program repair framework for settings where repair requires joint reasoning over code, issue text, and GUI screenshots. It combines hybrid workflow-agent control, region-level visual grounding, and a Failure Memory Bank that converts failed repair attempts into reusable guidance, improving resolved rate over GUIRepair on SWE-bench Multimodal.

• WebPII: Benchmarking Visual PII Detection for Computer-Use Agents

  • Nathan Zhao
  • 🏛️ Institutions: Stanford
  • 📅 Date: March 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [privacy], [PII detection], [visual redaction], [e-commerce UI], [WebPII], [WebRedact]
  • 📖 TLDR: WebPII is a benchmark for detecting personally identifiable information in web screenshots, centered on e-commerce interfaces where agents may expose sensitive content during browsing and form completion. It extends the PII taxonomy to transaction-level identifiers and partially filled forms, and pairs the benchmark with WebRedact to show practical privacy-preserving deployment.

• GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

  • Yang Li, Yuchen Liu, Haoyu Lu, Zhiqiang Xia, Hongzhen Wang, Kaiyang Han, Changpeng Yang, Jinyang Wu, Jiaming Xu, Runyu Shi, Ying Huang
  • 🏛️ Institutions: HyperAI Team, Xiaomi
  • 📅 Date: March 16, 2026
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [chinese], [hierarchical evaluation], [physical-device evaluation], [GUI-CEval]
  • 📖 TLDR: GUI-CEval is the first comprehensive Chinese benchmark for mobile GUI agents, spanning 201 apps across four device types with a hierarchical two-level evaluation structure (atomic abilities and application-level tasks) along five dimensions (perception, planning, reflection, execution, evaluation), revealing that most MLLMs still struggle with reflective decision-making and post-action self-evaluation.

• Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents

  • Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen
  • 🏛️ Institutions: McGill University, AMD, Red Hat
  • 📅 Date: March 16, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [security], [guardrail], [visual confused deputy], [TOCTOU], [grounding errors], [dual-channel contrastive classification]
  • 📖 TLDR: This paper reframes perception failures in GUI agents as a security problem rather than just a performance issue, formalizing the visual confused deputy where misperceived UI state causes privileged actions on the wrong target. It then proposes a dual-channel guardrail that separately checks the visual target and the agent's textual reasoning to block unsafe executions.

• Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective

  • Mohamed Aghzal, Gregory J. Stein, Ziyu Yao
  • 🏛️ Institutions: George Mason University
  • 📅 Date: March 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [failure analysis], [hierarchical planning], [PDDL], [replanning], [grounding]
  • 📖 TLDR: This paper analyzes web-agent failures through a three-layer hierarchy of high-level planning, low-level execution, and replanning rather than relying only on end-to-end success. It finds that structured PDDL plans improve strategic planning over natural-language plans, but that execution and grounding remain the dominant reliability bottlenecks.

• Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements

  • Ziwei Liu, Tao Feng, Borui Kang, Yanbing Yang, Jun Luo
  • 🏛️ Institutions: Sichuan University, Tsinghua, NJU, NTU
  • 📅 Date: March 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [training-free], [inference scaling], [ZoomUI], [instruction rewriting], [progressive zooming]
  • 📖 TLDR: ZoomUI is a training-free GUI grounding method built on the idea that complex interfaces can be decomposed into simpler visual elements that generic MLLMs already understand. It rewrites instructions into element-level visual descriptions and progressively zooms onto candidate UI regions, reaching or surpassing fine-tuned baselines without additional training.

• AI Planning Framework for LLM-Based Web Agents

  • Orit Shahnovsky, Rotem Dror
  • 🏛️ Institutions: University of Haifa
  • 📅 Date: March 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [AI planning], [evaluation metrics], [WebArena], [trajectory analysis], [planning taxonomy]
  • 📖 TLDR: This paper maps common LLM-based web-agent designs to classical planning paradigms such as BFS, best-first tree search, and DFS, then argues that trajectory-level metrics are needed alongside raw success rate. Using 794 human-labeled WebArena trajectories, it shows that different agent architectures optimize different dimensions of performance.

• Adaptive Vision-Language Model Routing for Computer Use Agents

  • Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen
  • 🏛️ Institutions: vLLM Semantic Router Project, MBZUAI, McGill University, AMD, Red Hat
  • 📅 Date: March 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [VLM routing], [cost-accuracy tradeoff], [semantic routing], [OpenClaw], [AVR], [guardrail escalation]
  • 📖 TLDR: AVR inserts a semantic routing layer between a computer-use agent and a pool of VLMs, selecting the cheapest model that satisfies a target reliability threshold for each action. It projects up to 78% inference-cost reduction while staying close to all-large-model performance, and can escalate risky actions when combined with the Visual Confused Deputy guardrail.

• HATS: Hardness-Aware Trajectory Synthesis for GUI Agents

  • Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen
  • 🏛️ Institutions: HIT-Shenzhen, NUS, CNRS@CREATE, Shenzhen Loop Area Institute, Huawei Noah's Ark Lab
  • 📅 Date: March 12, 2026
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [Web], [Mobile]
  • 🔑 Key: [trajectory synthesis], [semantic ambiguity], [hardness-aware exploration], [alignment-guided refinement], [WebArena], [AndroidWorld]
  • 📖 TLDR: HATS synthesizes GUI-agent training trajectories by modeling action hardness as semantic ambiguity and combining hardness-driven exploration with alignment-guided refinement, improving data quality and downstream performance on both WebArena and AndroidWorld.

• CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents

  • Marta Sumyk, Oleksandr Kosovan
  • 🏛️ Institutions: Ukrainian Catholic University
  • 📅 Date: March 11, 2026
  • 📑 Publisher: HEAL @ CHI 2026 Workshop
  • 💻 Env: [Desktop]
  • 🔑 Key: [evaluation], [VLM judge], [agent-as-a-judge], [calibration], [meta-evaluation], [CUAAudit]
  • 📖 TLDR: CUAAudit studies vision-language models as autonomous judges of desktop-agent task success from observable interactions alone. Across multiple operating-system benchmarks, it finds that even strong VLM auditors degrade on harder environments and disagree substantially with one another, highlighting limits of model-based auditing.

• Hybrid Self-evolving Structured Memory for Computer-Use Agents

  • Sibo Zhu, Wenyi Wu, Kun Zhou, Stephen Wang, Biwei Huang
  • 🏛️ Institutions: UC San Diego, Abel.ai
  • 📅 Date: March 11, 2026
  • 📑 Publisher: Findings of ACL 2026
  • 💻 Env: [Web]
  • 🔑 Key: [memory], [graph-based retrieval], [self-evolving memory], [HyMEM], [multi-hop retrieval]
  • 📖 TLDR: HyMEM is a graph-based memory system for GUI agents that couples symbolic nodes with continuous trajectory embeddings, supports multi-hop retrieval, and updates itself over time. It substantially boosts open-source 7B/8B GUI agents and can match or surpass stronger closed-source models on several benchmarks.

• Safe and Scalable Web Agent Learning via Recreated Websites

  • Hyungjoo Chae, Jungsoo Park, Alan Ritter
  • 🏛️ Institutions: Georgia Tech
  • 📅 Date: March 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [training environment], [VeriEnv], [synthetic environment], [programmatically verifiable rewards], [self-evolution]
  • 📖 TLDR: VeriEnv uses language models to clone real-world websites into executable synthetic environments with deterministic, programmatically verifiable rewards. This makes web-agent training safer and more scalable, and the paper shows agents trained in recreated sites can generalize to unseen websites and benefit from scaling the environment pool.

• SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environments

  • Syed Yusuf Ahmed, Shiwei Feng, Chanwoo Bae, Calix Barrus Xiangyu Zhang
  • 🏛️ Institutions: Purdue University, University of Texas at San Antonio
  • 📅 Date: March 10, 2026
  • 📑 Publisher: ICSE 2026
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [testing framework], [bug finding], [specialist agents], [real-world evaluation], [multimodal testing], [SpecOps]
  • 📖 TLDR: SpecOps is a fully automated testing framework that uses four specialist agents to generate cases, set up environments, execute tasks, and validate outcomes for real-world software agents. Across five deployed agents spanning CLI tools, web apps, and browser extensions, it finds 164 true bugs with 0.89 F1 while keeping each test under eight minutes and under $0.73.

• Video-Based Reward Modeling for Computer-Use Agents

  • Linxin Song, Jieyu Zhang, Huanxin Sheng, Taiwei Shi, Gupta Rahul, Yang Liu, Ranjay Krishna, Jian Kang, Jieyu Zhao
  • 🏛️ Institutions: USC, University of Washington, MBZUAI, Amazon AGI
  • 📅 Date: March 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [reward model], [dataset], [execution video], [trajectory evaluation], [spatiotemporal token pruning], [ExeVR-53k], [ExeVRM]
  • 📖 TLDR: This paper studies reward modeling from execution video rather than agent internals, introducing the ExeVR-53k dataset and an execution-video reward model that predicts success from keyframes plus the user instruction. The model scales evaluation across Ubuntu, macOS, Windows, and Android, outperforming strong proprietary models while providing finer temporal attribution.

• AgentOS: From Application Silos to a Natural Language-Driven Data Ecosystem

  • Rui Liu, Tao Zhe, Dongjie Wang, Zijun Yao, Kunpeng Liu, Yanjie Fu, Huan Liu, Jian Pei
  • 🏛️ Institutions: University of Kansas, Clemson University
  • 📅 Date: March 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [operating system], [natural user interface], [agent kernel], [skill modules], [knowledge discovery], [AgentOS]
  • 📖 TLDR: AgentOS proposes replacing application silos and traditional desktops with a natural-language-driven operating system centered on an Agent Kernel and modular skill components. The paper frames this system as a knowledge discovery problem involving intent mining, workflow automation, recommender systems, and personal knowledge graphs.

• OSExpert: Computer-Use Agents Learning Professional Skills via Exploration

  • Jiateng Liu, Zhenhailong Wang, Rushi Wang, Bingxuan Li, Jeonghwan Kim, Aditi Tiwari, Pengfei Yu, Denghui Zhang, Heng Ji
  • 🏛️ Institutions: UIUC, Stevens Institute of Technology
  • 📅 Date: March 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [exploration], [skill learning], [benchmark], [action primitives], [GUI-DFS], [OSExpert-Eval]
  • 📖 TLDR: OSExpert studies how computer-use agents can learn professional software skills through exploration, introducing GUI-DFS to verify unit functions, discover action primitives, and compose them into longer workflows. The learned skill library improves performance on OSExpert-Eval by about 20% and closes roughly 80% of the efficiency gap to human experts.

• PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents

  • Yuxiang Chai, Shunye Tang, Han Xiao, Rui Liu, Hongsheng Li
  • 🏛️ Institutions: CUHK MMLab, Nankai University, Huawei Research
  • 📅 Date: March 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [benchmark], [proactive recommendation], [intent inference], [continuous screenshots], [PIRF], [multithreaded trajectories]
  • 📖 TLDR: PIRA-Bench studies proactive GUI assistance, where an agent infers user intent from continuous visual streams instead of waiting for explicit commands. It focuses on long, noisy, interleaved trajectories with user-profile context, and introduces PIRF as a memory-aware baseline for proactive intent recommendation.

• SecAgent: Efficient Mobile GUI Agent with Semantic Context

  • Yiping Xie, Song Chen, Jingxuan Xing, Wei Jiang, Zekun Zhu, Yingyao Wang, Pi Bu, Jun Song, Yuning Jiang, Bo Zheng
  • 🏛️ Institutions: Taobao & Tmall Group of Alibaba
  • 📅 Date: March 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [semantic context], [dataset], [benchmark], [history summarization], [chinese mobile apps], [SecAgent]
  • 📖 TLDR: SecAgent is a 3B mobile GUI agent that summarizes history screenshots and actions into concise semantic context, reducing computation while preserving task-relevant information. It also introduces a human-verified Chinese mobile GUI dataset and benchmark, and reaches performance comparable to 7B-8B models through supervised and reinforcement fine-tuning.

• SlowBA: An efficiency backdoor attack towards VLM-based GUI agents

  • Junxian Li, Tu Lan, Haozhen Tan, Yan Meng, Haojin Zhu
  • 🏛️ Institutions: SJTU
  • 📅 Date: March 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [backdoor attack], [response latency], [reward-level backdoor injection], [popup trigger], [security], [SlowBA]
  • 📖 TLDR: SlowBA studies a backdoor attack on VLM-based GUI agents that targets response efficiency rather than action correctness, using realistic pop-up triggers to induce excessively long reasoning chains. Its two-stage reward-level backdoor injection stays stealthy, preserves task accuracy, and substantially increases latency under trigger conditions.

• Generalization in Online Reinforcement Learning for Mobile Agents

  • Li Gu, Zihuan Jiang, Zhixiang Chi, Huan Liu, Ziqiang Wang, Yuanhao Yu, Glen Berseth, Yang Wang
  • 🏛️ Institutions: Mila, Concordia University, Université de Montréal, CIFAR AI Chair, University of Toronto, McMaster University
  • 📅 Date: March 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reinforcement learning], [benchmark], [generalization], [AndroidWorld-Generalization], [GRPO], [test-time adaptation]
  • 📖 TLDR: This paper studies generalization in online RL for mobile agents, introducing AndroidWorld-Generalization to measure transfer to unseen instances, templates, and applications. Its open RL training system shows gains over supervised fine-tuning on unseen instances, but also highlights that unseen templates and apps remain much harder without additional adaptation.

• Enhancing Web Agents with a Hierarchical Memory Tree

  • Yunteng Tan, Zhi Gao, Xinxiao Wu
  • 🏛️ Institutions: Beijing Institute of Technology
  • 📅 Date: March 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [memory], [hierarchical memory tree], [cross-website generalization], [planner-actor decomposition], [HMT], [Mind2Web]
  • 📖 TLDR: This paper proposes Hierarchical Memory Tree, which separates task intent, reusable stages, and action patterns to decouple planning from page-specific execution. The resulting planner-actor setup improves web-agent generalization on Mind2Web and WebArena, especially in cross-website and cross-domain settings.

• TimeWarp: Evaluating Web Agents by Revisiting the Past

  • Md Farhan Ishmam, Kenneth Marino
  • 🏛️ Institutions: University of Utah
  • 📅 Date: March 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [evolving interfaces], [plan distillation], [TimeWarp], [TimeTraj], [behavior cloning]
  • 📖 TLDR: TimeWarp evaluates web agents under interface drift by recreating multiple historical UI versions of the same environments. The paper shows current agents are brittle to design changes and introduces TimeTraj, which distills plans across versions to improve robustness.

• WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces

  • Sicheng Fan, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, Dehan Kong
  • 🏛️ Institutions: Fudan, iMean AI
  • 📅 Date: March 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [dataset], [benchmark], [triple alignment], [human annotation], [dual mid-training], [WebChainBench]
  • 📖 TLDR: WebChain is a large human-annotated dataset of real-world web interaction traces with aligned visual, structural, and action supervision. The paper also proposes Dual Mid-Training, which separates spatial grounding from planning and improves performance on WebChainBench and other public web-agent benchmarks.

• WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents

  • Sicheng Fan, Qingyun Shi, Shengze Xu, Shengbo Cai, Tieyong Zeng, Li Ling, Yanyi Shang, Dehan Kong
  • 🏛️ Institutions: Fudan, iMean AI, CUHK, Tsinghua
  • 📅 Date: March 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [reinforcement learning], [synthetic data], [environment synthesis], [task generation], [embodiment potential], [WebFactory]
  • 📖 TLDR: WebFactory presents a closed-loop training pipeline that compresses LLM latent internet knowledge into grounded web-agent behavior through synthetic environment generation, task generation, trajectory collection, and decomposed-reward RL. It matches agents trained on comparable amounts of human data while using synthetic data from only 10 websites.

• Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks

  • Haoyu Liu, Dingcheng Li, Lukas Rutishauser, Zeyu Zheng
  • 🏛️ Institutions: UC Berkeley, IEOR & BAIR, Google, Google DeepMind
  • 📅 Date: March 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [security], [safety], [defense], [adversarial training], [cross-modal attack], [DOM injection], [DMAST], [GRPO]
  • 📖 TLDR: DMAST is a three-stage adversarial safety training pipeline for multimodal web agents that jointly reasons over screenshots and accessibility trees. It targets cross-modal DOM injection attacks, substantially reducing adversarial risk while improving efficiency on out-of-distribution MiniWob++ tasks.

• CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning

  • Zhenquan Yao, Zitong Huang, Yihan Zeng, Jianhua Han, Hang Xu, Chun-Mei Feng, Jianwei Ma, Wangmeng Zuo
  • 🏛️ Institutions: Harbin Institute of Technology, Huawei Noah's Ark Lab, University College Dublin, PKU
  • 📅 Date: March 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [continual learning], [reinforcement learning], [GRPO], [gradient surgery], [policy entropy], [AndroidControl-CL], [CGL]
  • 📖 TLDR: CGL studies continual GUI learning under app updates, combining supervised adaptation with reinforcement fine-tuning to retain prior interaction skills. It uses policy-entropy-guided SFT weighting and gradient surgery against GRPO anchor gradients, and introduces AndroidControl-CL to benchmark continual adaptation without catastrophic forgetting.

• K²-Agent: Co-Evolving Know-What and Know-How for Hierarchical Mobile Device Control

  • Zhe Wu, Donglin Mo, Hongjin Lu, Junliang Xing, Jianheng Liu, Yuheng Jing, Kai Li, Kun Shao, Jianye Hao, Yuanchun Shi
  • 🏛️ Institutions: Tsinghua, Huawei Noah's Ark Lab, Institute of Automation, CAS
  • 📅 Date: February 28, 2026
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Mobile]
  • 🔑 Key: [hierarchical control], [declarative knowledge], [procedural knowledge], [SRLR], [C-GRPO], [K²-Agent]
  • 📖 TLDR: K²-Agent separates mobile control into a high-level declarative reasoner and a low-level procedural executor, then co-evolves both through self-refinement and curriculum-guided RL. The design reaches 76.1% on AndroidWorld and transfers well to ScreenSpot-v2 and AITW.

• M^2: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrieval

  • Dawei Yan, Haokui Zhang, Guangda Huzhang, Yang Li, Yibo Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Ying Li, Wei Dong, Chunhua Shen
  • 🏛️ Institutions: Northwestern Polytechnical University, Alibaba Group, Xi'an University of Architecture and Technology, ZJU
  • 📅 Date: February 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [memory augmentation], [trajectory summarization], [insight retrieval], [training-free], [long-horizon tasks], [M^2]
  • 📖 TLDR: M^2 is a training-free memory augmentation method for long-horizon web agents that combines dynamic trajectory summarization with offline insight retrieval. It improves success rates on WebVoyager and OnlineMind2Web while substantially reducing token usage.

• MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

  • Yunfei Feng, Xi Zhao, Cheng Zhang, Dahu Feng, Daolin Cheng, Jianqi Yu, Yubin Xia, Erhu Feng
  • 🏛️ Institutions: SJTU
  • 📅 Date: February 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [trajectory fusion], [third-party apps], [MobiFlow]
  • 📖 TLDR: MobiFlow benchmarks mobile agents on third-party Android applications without relying on system-level APIs, using a graph-construction algorithm based on multi-trajectory fusion to compress state space and support dynamic interaction. It covers 20 widely used apps and 240 real-world tasks, with evaluation results better aligned to human assessments than AndroidWorld.

• CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoning

  • Yuxuan Liu, Weikai Xu, Kun Huang, Changyu Chen, Jiankun Zhao, Pengzhi Gao, Wei Liu, Jian Luan, Shuo Shang, Bo Du, Ji-Rong Wen, Rui Yan
  • 🏛️ Institutions: Renmin University of China, MiLM Plus, Xiaomi, NTU, Wuhan University
  • 📅 Date: February 27, 2026
  • 📑 Publisher: ICML 2026 (Poster)
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [mixture of experts], [hybrid-capabilities reasoning], [progressive training], [Info-DPO], [CoME]
  • 📖 TLDR: CoME aligns four output-oriented experts with the screen-summary, subtask-planning, action-decision, and action-function stages of mobile-agent reasoning. Progressive expert, router, and chain-of-thought training plus information-gain-driven DPO improve the balance and informativeness of intermediate reasoning, outperforming dense mobile agents and other MoE methods on AITZ and AMEX.

• GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

  • Rui Yang, Qianhui Wu, Zhaoyang Wang, Hanyang Chen, Ke Yang, Hao Cheng, Huaxiu Yao, Baoling Peng, Huan Zhang, Jianfeng Gao, Tong Zhang
  • 🏛️ Institutions: UIUC, Microsoft, UNC
  • 📅 Date: February 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [post-training], [reinforcement learning], [action-aware supervision], [partial verifiability], [GUI reasoning dataset], [GUI-Libra]
  • 📖 TLDR: GUI-Libra is a post-training recipe for native GUI agents that combines curated reasoning data, action-aware supervised fine-tuning, and partially verifiable RL. It targets the mismatch between chain-of-thought reasoning and grounding, and improves both step-level accuracy and end-to-end task completion on web and mobile benchmarks.

• OpeFlo: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding

  • Wee Joe Tan, Zi Rui Lucas Lim, Shashank Durgad, Karim Obegi, Aiden Yiliu Li
  • 🏛️ Institutions: Onflow AI
  • 📅 Date: February 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [GUI grounding], [UX evaluation], [user simulation], [OpenFlo]
  • 📖 TLDR: OpenFlo simulates user behavior on websites with multimodal GUI grounding rather than DOM parsing, producing standardized UX reports that integrate the System Usability Scale, Single Ease Questions, and Think Aloud. Built on Avenir-Web, it pairs robust web interaction with simulated user behavior profiles for continuous, scalable usability testing.

• ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory

  • Hongbin Zhong, Fazle Faisal, Luis França, Tanakorn Leesatapornwongsa, Adriana Szekeres, Kexin Rong, Suman Nath
  • 🏛️ Institutions: Georgia Tech, MSR
  • 📅 Date: February 24, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [programmatic planning], [state machine memory], [crawling agent], [execution agent], [re-grounding fallback], [ActionEngine]
  • 📖 TLDR: ActionEngine shifts GUI agents from reactive step-by-step execution to programmatic planning by building an updatable state-machine memory and synthesizing executable programs from it. Its crawler-plus-executor design reaches 95% success on WebArena Reddit tasks while greatly reducing cost and latency.

• Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization

  • Jiachen Zhu, Lingyu Yang, Rong Shan, Congmin Zheng, Zeyu Zheng, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin
  • 🏛️ Institutions: SJTU
  • 📅 Date: February 24, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dataset], [humanization], [anti-detection], [touch dynamics], [AHB]
  • 📖 TLDR: This paper formalizes mobile GUI agent humanization as a MinMax optimization between detector and agent, releases a high-fidelity dataset of mobile touch dynamics, and establishes the Agent Humanization Benchmark (AHB). Vanilla LMM agents are easily detectable due to unnatural kinematics; data-driven behavioral matching achieves high imitability without sacrificing utility.

• IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents

  • Seoyoung Lee, Seobin Yoon, Seongbeen Lee, Yoojung Chun, Dayoung Park, Doyeon Kim, Joo Yong Sim
  • 🏛️ Institutions: Sookmyung Women's University
  • 📅 Date: February 19, 2026
  • 📑 Publisher: AAMAS 2026
  • 💻 Env: [Desktop]
  • 🔑 Key: [multi-agent planning], [skill abstraction], [intent representation], [plan memory], [long-horizon automation], [IntentCUA]
  • 📖 TLDR: IntentCUA learns intent-level abstractions for desktop automation and uses them to support reusable skills and multi-agent planning. Its Planner, Plan-Optimizer, and Critic coordinate over intent-aligned memory, improving both success rate and step efficiency on long-horizon tasks.

• Modeling Distinct Human Interaction in Web Agents

  • Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo, Venu Arvind Arangarajan, Tianyue Ou, Frank Xu, Shuyan Zhou, Graham Neubig, Jeffrey P. Bigham
  • 🏛️ Institutions: CMU, Duke University
  • 📅 Date: February 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [dataset], [human-agent interaction], [intervention prediction], [interaction styles], [CowCorpus], [collaborative web agents]
  • 📖 TLDR: This paper studies collaborative web agents through human intervention patterns, introducing CowCorpus with 400 real-user trajectories that mix human and agent actions. Modeling four distinct interaction styles substantially improves intervention prediction and raises user-rated usefulness in live agents.

• Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

  • Serin Kim, Sangam Lee, Dongha Lee
  • 🏛️ Institutions: Yonsei University
  • 📅 Date: February 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [personalization], [user history], [contextual reasoning], [clarify-to-personalize], [Persona2Web]
  • 📖 TLDR: Persona2Web benchmarks personalized web agents on ambiguous tasks that require inferring user preferences from browsing history rather than explicit instructions. It highlights the difficulty of contextual reasoning with user-specific state across multiple web-agent architectures and backbone models.

• Web Verbs: Typed Abstractions for Reliable Task Composition on the Agentic Web

  • Linxi Jiang, Rui Xi, Zhijie Liu, Shuo Chen, Zhiqiang Lin, Suman Nath
  • 🏛️ Institutions: OSU, University of British Columbia, MSR
  • 📅 Date: February 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [typed action abstraction], [task composition], [semantic action layer], [typed contracts], [agentic web], [Web Verbs]
  • 📖 TLDR: Web Verbs proposes a semantic action layer for web agents built from typed, composable actions with explicit contracts, policies, and logging. It aims to replace brittle low-level clicks and keystrokes with more reliable, efficient, and auditable task composition.

• World-Model-Augmented Web Agents with Action Correction

  • Zhouzhou Shen, Xueyu Hu, Xiyun Li, Tianqing Fang, Juncheng Li, Shengyu Zhang
  • 🏛️ Institutions: ZJU, Tencent AI Lab
  • 📅 Date: February 17, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [world model], [action correction], [judge model], [consequence simulation], [WAC], [multi-agent collaboration]
  • 📖 TLDR: WAC augments web agents with a world model for strategic guidance and consequence simulation, plus a judge model for feedback-driven action correction. This combination improves performance on VisualWebArena and Online-Mind2Web over prior methods.

• EmbeWebAgent: Embedding Web Agents into Any Customized UI

  • Chenyang Ma, Clyde Fare, Matthew Wilson, Dave Braines
  • 🏛️ Institutions: IBM Research Europe, UK, Oxford
  • 📅 Date: February 16, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [embedded agents], [customized UI], [ARIA], [frontend hooks], [function registry], [EmbeWebAgent]
  • 📖 TLDR: EmbeWebAgent embeds web agents directly into customized web UIs through lightweight frontend hooks such as curated ARIA labels, URL-based observations, and a per-page function registry. It targets stack-agnostic enterprise settings where agents need mixed-granularity actions without replacing the existing UI.

• WebWorld: A Large-Scale World Model for Web Agent Training

  • Zikai Xiao, Jianhong Tu, Chuhang Zou, Yuxin Zuo, Zhi Li, Peng Wang, Bowen Yu, Fei Huang, Junyang Lin, Zuozhu Liu
  • 🏛️ Institutions: Qwen Team, Alibaba Group, ZJU
  • 📅 Date: February 16, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [world model], [data generation], [environment simulation], [trajectory synthesis], [WebArena], [WebWorld]
  • 📖 TLDR: WebWorld is a large-scale world model for web-agent training built from over one million real web interactions. It synthesizes long-horizon trajectories for training, improves WebArena performance, and is designed to transfer beyond web tasks to broader interactive environments.

• AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines

  • Yifan Wu, Yiran Peng, Yiyu Chen, Jianhao Ruan, Zijie Zhuang, Cheng Yang, Jiayi Zhang, Man Chen, Yenchi Tseng, Zhaoyang Yu, Liang Chen, Yuyao Zhai, Bang Liu, Chenglin Wu, Yuyu Luo
  • 🏛️ Institutions: HKUST(GZ), DeepWisdom, PKU, Université de Montréal, Mila
  • 📅 Date: February 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [data generation], [environment synthesis], [finite state machine], [verifiable environments], [search-and-verify], [AutoWebWorld]
  • 📖 TLDR: AutoWebWorld synthesizes controllable web environments as finite state machines, enabling automated search-and-verify trajectory generation at scale. Agents trained on the resulting synthetic data achieve strong WebVoyager performance and improve consistently with more generated data.

• GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training

  • Yuan Cao, Dezhi Ran, Mengzhou Wu, Yuzhe Guo, Xin Chen, Ang Li, Gang Cao, Gong Zhi, Hao Yu, Linyi Li, Wei Yang, Tao Xie
  • 🏛️ Institutions: PKU, Tencent, HKUST, Simon Fraser University, University of Texas at Dallas
  • 📅 Date: February 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [environment synthesis], [verifiable reward], [training efficiency], [post-training], [code-native rewards], [GUI-GENESIS]
  • 📖 TLDR: GUI-GENESIS automatically synthesizes lightweight GUI training environments with code-native verifiable rewards derived from real applications. It targets efficient post-training, cutting latency and cost while improving over both the base model and RL baselines on held-out tasks.

• Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

  • Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, Zhiyuan Chen, Jitong Liao, Qi Zheng, Jiahui Zeng, Ze Xu, Shuai Bai, Junyang Lin, Jingren Zhou, Ming Yan
  • 🏛️ Institutions: Tongyi Lab, Alibaba Group
  • 📅 Date: February 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [model], [GUI-Owl-1.5], [multi-platform], [tool use], [memory], [MRPO], [data flywheel]
  • 📖 TLDR: Mobile-Agent-v3.5 introduces GUI-Owl-1.5, a family of native GUI agents spanning desktop, mobile, and browser settings. The work combines a hybrid data flywheel, stronger memory and tool use, and multi-platform RL with MRPO to improve results across many open GUI benchmarks.

• Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision

  • A. Said Gurbuz, Sunghwan Hong, Ahmed Nassar, Marc Pollefeys, Peter Staar
  • 🏛️ Institutions: IBM Research, ETH, KAIST
  • 📅 Date: February 15, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [dataset], [screen parsing], [dense supervision], [ScreenParse], [UI understanding]
  • 📖 TLDR: This paper argues that sparse grounding supervision is insufficient for GUI understanding and introduces ScreenParse, a large-scale densely annotated screen-parsing dataset. It provides complete UI-element supervision across web screenshots to support richer grounding and UI understanding models.

• Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization

  • Yibo Wang, Guangda Huzhang, Yuwei Hu, Yu Xia, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang
  • 🏛️ Institutions: National Key Laboratory for Novel Software Technology, NJU, Ovis Team, Alibaba Group
  • 📅 Date: February 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reinforcement learning], [agentic-Q estimation], [step-wise policy optimization], [Ovis2.5-9B], [grounding]
  • 📖 TLDR: This paper trains GUI agents with an agentic-Q model that estimates each action's contribution to task completion and a step-wise policy optimization routine decoupled from online interaction. The design keeps data collection manageable while stabilizing updates and improving navigation and grounding performance.

• OpAgent: Operator Agent for Web Navigation

  • Yuyu Guo, Wenjie Yang, Siyuan Yang, Ziyang Liu, Cheng Chen, Yuan Wei, Yun Hu, Yang Huang, Guoliang Hao, Dongsheng Yuan, Jianming Wang, Xin Chen, Hang Yu, Lei Lei, Peng Di
  • 🏛️ Institutions: Ant Group
  • 📅 Date: February 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [reinforcement learning], [planner-grounder-reflector-summarizer], [hierarchical multitask fine-tuning], [WebArena], [OpAgent]
  • 📖 TLDR: OpAgent combines a modular planner-grounder-reflector-summarizer design with online reinforcement learning on unconstrained web environments. The paper reports strong WebArena performance and studies how modular coordination and RL improve web navigation.

• Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation

  • Lajanugen Logeswaran, Jaekyeom Kim, Sungryull Sohn, Creighton Glasscock, Honglak Lee
  • 🏛️ Institutions: LG AI Research
  • 📅 Date: February 13, 2026
  • 📑 Publisher: COLM 2025
  • 💻 Env: [Web]
  • 🔑 Key: [data generation], [fine-grained evaluation], [constraint-based evaluator], [BookingArena], [knowledge distillation], [partial progress]
  • 📖 TLDR: This paper builds a scalable web-agent training pipeline around a constraint-based evaluator that scores partial progress instead of only final success. It introduces BookingArena and shows that using automatically generated data plus fine-grained evaluation can train smaller web agents that match or exceed much larger systems.

• WebClipper: Efficient Evolution of Web Agents with Graph-based Trajectory Pruning

  • Junjie Wang, Zequn Xie, Dan Yang, Jie Feng, Yue Shen, Duolin Sun, Meixiu Long, Yihan Jiao, Zhehao Tan, Jian Wang, Peng Wei, Jinjie Gu
  • 🏛️ Institutions: Ant Group, ZJU
  • 📅 Date: February 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [trajectory pruning], [graph-based optimization], [minimum DAG], [accuracy-efficiency tradeoff], [F-AE Score], [WebClipper]
  • 📖 TLDR: WebClipper represents web-agent search traces as state graphs and prunes them into a minimum necessary DAG that removes loops and low-value branches. Training on these refined traces improves accuracy while reducing tool-call rounds, and the paper introduces F-AE Score to evaluate the accuracy-efficiency balance.

• Adaptive Milestone Reward for GUI Agents

  • Congmin Zheng, Xiaoyun Mo, Xinbei Ma, Qiqiang Lin, Yin Zhao, Jiachen Zhu, Xingyu Lou, Jun Wang, Zhaoxiang Wang, Weiwen Liu, Zhuosheng Zhang, Yong Yu, Weinan Zhang
  • 🏛️ Institutions: SJTU, OPPO Research Institute
  • 📅 Date: February 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reinforcement learning], [reward shaping], [credit assignment], [milestone reward], [ADMIRE], [AndroidWorld]
  • 📖 TLDR: ADMIRE is a reinforcement-learning reward design for GUI agents that distills adaptive, verifiable milestones from successful trajectories and pairs them with asymmetric credit assignment. It improves AndroidWorld performance by more than 10 absolute points and transfers to other RL algorithms and environments.

• Agentic Test-Time Scaling for WebAgents

  • Nicholas Lee, Lutfi Eren Erdogan, Chris Joseph John, Surya Krishnapillai, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
  • 🏛️ Institutions: UC Berkeley, ICSI, LBNL
  • 📅 Date: February 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [test-time scaling], [CATTS], [inference-time compute], [uncertainty estimation], [LLM arbiter], [WebArena-Lite]
  • 📖 TLDR: CATTS dynamically allocates test-time compute for multi-step web agents by using vote-based uncertainty signals to invoke an LLM arbiter only on contentious decisions. It improves performance on WebArena-Lite and GoBrowse while using fewer tokens than uniform scaling.

• AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild

  • Jiazheng Sun, Mingxuan Li, Yingying Zhang, Jiayang Niu, Yachen Wu, Ruihan Jin, Shuyu Lei, Pengrongrui Tan, Zongyu Zhang, Ruoyi Wang, Jiachen Yang, Boyu Yang, Jiacheng Liu, Xin Peng
  • 🏛️ Institutions: Fudan, Jilin University
  • 📅 Date: February 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dataset], [intent alignment], [ambiguous instructions], [MUSE], [AmbiBench]
  • 📖 TLDR: Reframes mobile-agent evaluation around intent alignment rather than perfect one-shot instructions by organizing 240 real-world tasks across 25 apps into four clarity levels from detailed to ambiguous. It also introduces MUSE, an automated judge that scores not just task completion but interaction quality, showing that current agents still struggle badly when users are incomplete or ambiguous.

• How Smart Is Your GUI Agent? A Framework for the Future of Software Interaction

  • Sidong Feng, Chunyang Chen
  • 🏛️ Institutions: CUHK-Shenzhen, TUM
  • 📅 Date: February 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [autonomy levels], [taxonomy], [trustworthy AI], [software interaction], [GUI Agent Autonomy Levels], [conceptual framework]
  • 📖 TLDR: This paper is a conceptual framing proposal rather than a new agent system. It introduces GUI Agent Autonomy Levels, a six-level taxonomy for describing capability, responsibility, and risk, with the goal of giving the field a clearer language for benchmarking and trustworthy deployment.

• WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements

  • Xiwen Teoh, Yun Lin, Duc-Minh Nguyen, Ruofei Ren, Wenjie Zhang, Jin Song Dong
  • 🏛️ Institutions: NUS, SJTU
  • 📅 Date: February 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [web testing], [bug detection], [GUI grounding], [test oracle], [symbolized GUI elements], [WebTestPilot]
  • 📖 TLDR: WebTestPilot is an agentic end-to-end web-testing system that symbolizes GUI elements and infers test oracles from natural-language specifications. It achieves high precision and recall on bug-injected web apps and outperforms existing baselines in automated web testing.

• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception

  • Lai Wei, Liangbo He, Jun Lan, Lingzhong Dong, Yutong Cai, Siyuan Li, Huijia Zhu, Weiqiang Wang, Linghe Kong, Yue Wang, Zhuosheng Zhang, Weiran Huang
  • 🏛️ Institutions: SJTU, Ant Group, Zhongguancun Academy, Shanghai Innovation Institute
  • 📅 Date: February 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [knowledge distillation], [fine-grained perception], [region-to-image distillation], [ZoomBench], [training data generation], [multimodal perception]
  • 📖 TLDR: This paper proposes Region-to-Image Distillation, which teaches a model to internalize zoom-in behavior without requiring explicit crop-and-reason inference at test time. It also introduces ZoomBench and shows stronger fine-grained perception on both perception and GUI-agent benchmarks.

• Blind Gods and Broken Screens: Architecting a Secure, Intent-Centric Mobile Agent Operating System

  • Zhenhua Zou, Sheng Guo, Qiuyang Zhan, Lepeng Zhao, Shuo Li, Qi Li, Ke Xu, Mingwei Xu, Zhuotao Liu
  • 🏛️ Institutions: Tsinghua
  • 📅 Date: February 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [agent security], [prompt injection], [access control], [intent-centric OS], [MobileSafetyBench], [Aura]
  • 📖 TLDR: This paper analyzes security failures in mobile GUI agents and proposes Aura, an intent-centric runtime architecture that replaces GUI scraping with structured interaction mediated by identity, semantic firewalls, taint-aware memory, and access control. It reports strong task performance while greatly reducing attack success on MobileSafetyBench.

• See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch

  • Xingyi Zhang, Yulei Ye, Kaifeng Huang, Wenhao Li, Xiangfeng Wang
  • 🏛️ Institutions: East China Normal University, Tongji University
  • 📅 Date: February 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [ScratchWorld], [drag-and-drop], [block-based programming], [reasoning-acting gap], [scratch]
  • 📖 TLDR: ScratchWorld evaluates multimodal GUI agents on Scratch program construction tasks that require fine-grained drag-and-drop manipulation. The benchmark exposes a large gap between high-level planning success and low-level GUI execution.

• UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics

  • Mengzhou Wu, Yuzhe Guo, Yuan Cao, Haochuan Lu, Songhe Zhu, Pingzhe Qu, Xin Chen, Kang Qin, Zhongpu Wang, Xiaode Zhang, Xinyi Wang, Wei Dai, Gang Cao, Yuetang Deng, Zhi Gong, Dezhi Ran, Linyi Li, Wei Yang, Tao Xie
  • 🏛️ Institutions: PKU, Tencent
  • 📅 Date: February 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [world model], [data generation]
  • 📖 TLDR: UI-Oceanus shifts learning from mimicking trajectories to mastering interaction physics via forward dynamics prediction. Using synthetic environmental dynamics as supervision, it improves GUI agents by 7% on offline benchmarks and 16.8% in online navigation, with performance scaling log-linearly with data volume.

• Autonomous Continual Learning of Computer-Use Agents for Environment Adaptation

  • Tianci Xue, Zeyi Liao, Tianneng Shi, Zilu Wang, Kai Zhang, Dawn Song, Yu Su, Huan Sun
  • 🏛️ Institutions: OSU, UC Berkeley
  • 📅 Date: February 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [continual learning], [reinforcement learning], [environment adaptation], [CUAJudge], [ACuRL], [curriculum learning]
  • 📖 TLDR: ACuRL studies continual adaptation for computer-use agents in changing environments without human demonstrations. It combines autonomous curriculum generation with CUAJudge-based evaluation to improve both within-environment and cross-environment adaptation while limiting forgetting.

• Code2World: A GUI World Model via Renderable Code Generation

  • Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin
  • 🏛️ Institutions: USTC, AMAP, Alibaba Group, Sun Yat-sen University, Oxford
  • 📅 Date: February 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [world model], [renderable code generation], [AndroidCode], [render-aware reinforcement learning], [next-state prediction], [Code2World]
  • 📖 TLDR: Code2World models GUI dynamics by generating renderable code for the next interface state rather than directly predicting pixels. Trained on AndroidCode with render-aware RL, it improves next-state prediction and downstream Android navigation.

• TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution

  • Deyang Jiang, Jing Huang, Xuanle Zhao, Lei Chen, Liming Zheng, Fanfan Liu, Haibo Qiu, Peng Shi, Zhixiong Zeng
  • 🏛️ Institutions: Meituan
  • 📅 Date: February 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [planning], [multi-agent], [tree-structured evolution], [trajectory generation], [TreeCUA], [TreeCUA-DPO]
  • 📖 TLDR: TreeCUA tackles the scaling bottleneck in GUI planning by organizing exploration trajectories as reusable tree structures with verification, summarization, and evaluation. The resulting data supports TreeCUA-DPO, which improves planning quality and out-of-domain generalization.

• When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents

  • Yuting Ning, Jaylen Jones, Zhehao Zhang, Chentao Ye, Weitong Ruan, Junyi Li, Rahul Gupta, Huan Sun
  • 🏛️ Institutions: OSU, Amazon AGI
  • 📅 Date: February 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [safety], [guardrail], [misaligned actions], [benchmark], [dataset], [MisActBench], [DeAction]
  • 📖 TLDR: This paper introduces MisActBench, a benchmark of 2,264 human-annotated action-level alignment labels covering malicious instruction following, harmful unintended behavior, and task-irrelevant actions. It proposes DeAction, a two-stage guardrail that detects misaligned actions before execution and iteratively corrects them, improving F1 by 15%+ over baselines and reducing attack success rate by over 90%.

• When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents

  • Jaylen Jones, Zhehao Zhang, Yuting Ning, Eric Fosler-Lussier, Pierre-Luc St-Charles, Yoshua Bengio, Dawn Song, Yu Su, Huan Sun
  • 🏛️ Institutions: OSU, LawZero, Mila, UdeM, UC Berkeley
  • 📅 Date: February 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [safety], [unintended behaviors], [AutoElicit], [benchmark], [red teaming]
  • 📖 TLDR: AutoElicit is an agentic framework that iteratively perturbs benign instructions using CUA execution feedback to surface unintended harmful behaviors while keeping inputs realistic. It elicits severe harms from frontier CUAs like Claude 4.5 and Operator in up to 72.5% of OS-domain seeds, and evaluates cross-model transferability of verified perturbations.

• Mapping the Design Space of User Experience for Computer Use Agents

  • Ruijia Cheng, Jenny T. Liang, Eldon Schoop, Jeffrey Nichols
  • 🏛️ Institutions: Apple, CMU
  • 📅 Date: February 07, 2026
  • 📑 Publisher: IUI 2026
  • 💻 Env: [Desktop]
  • 🔑 Key: [UX design], [human-computer interaction], [user study], [taxonomy], [wizard-of-oz]
  • 📖 TLDR: This paper maps the UX design space for computer-use agents through expert interviews and Wizard-of-Oz studies. It contributes a taxonomy covering prompts, explainability, control, and mental models for agent-driven software interaction.

• ANCHOR: Branch-Point Data Generation for GUI Agents

  • Jinbiao Wei, Yilun Zhao, Kangqi Ni, Arman Cohan
  • 🏛️ Institutions: Yale University, UNC
  • 📅 Date: February 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [data generation], [trajectory expansion], [branch-point sampling], [desktop automation], [verification], [ANCHOR]
  • 📖 TLDR: ANCHOR expands seed GUI demonstrations by identifying branch points and generating verified alternative task continuations. The resulting data improves desktop-agent performance on OSWorld and WindowsAgentArena over zero-shot and simpler synthesis baselines.

• POINTS-GUI-G: GUI-Grounding Journey

  • Zhongyin Zhao, Yuan Liu, Yikun Liu, Haicheng Wang, Le Tian, Xiao Zhou, Yangxiu You, Zilin Yu, Yang Yu, Jie Zhou
  • 🏛️ Institutions: WeChat AI
  • 📅 Date: February 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [data engineering], [vision encoder training], [verifiable rewards], [POINTS-GUI-G-8B], [ScreenSpot-pro]
  • 📖 TLDR: POINTS-GUI-G studies how to build strong GUI grounding from a base model without strong initial spatial grounding. It attributes gains to multi-source data engineering, improved vision-encoder training, and RL with verifiable rewards, producing competitive results across several grounding benchmarks.

• Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion

  • Longhui Ma, Di Zhao, Siwei Wang, Zhao Lv, Miao Wang
  • 🏛️ Institutions: National University of Defense Technology, Academy of Military Sciences
  • 📅 Date: February 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [multimodal fusion], [attention mechanism], [training-free], [OCR], [Trifuse]
  • 📖 TLDR: Trifuse is a training-free GUI grounding method that fuses MLLM attention, OCR text, and icon caption signals through a consensus-based fusion strategy. It improves grounding across multiple benchmarks without task-specific fine-tuning.

• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics

  • Yichen Gong, Zhuohan Cai, Sunhao Dai, Yuqi Zhou, Zhangxuan Gu, Changhua Meng, Shuheng Shen
  • 🏛️ Institutions: Ant Group, RUC
  • 📅 Date: February 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [user-centric], [capability diagnostics], [perception], [memory], [VenusBench-Mobile]
  • 📖 TLDR: VenusBench-Mobile addresses the app-centric, task-homogeneous nature of prior mobile-GUI benchmarks with a user-intent-driven task design and a capability-oriented annotation scheme for fine-grained behavior analysis. SOTA mobile GUI agents see large drops relative to existing benchmarks, with failures dominated by perception and memory deficiencies and near-zero success under environment variations, signaling persistent brittleness in realistic conditions.

• M$^2$-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Mining

  • Rui Lv, Juncheng Mo, Tianyi Chu, Chen Rao, Hongyi Jing, Jiajie Teng, Jiafu Chen, Shiqi Zhang, Liangzi Ding, Shuo Fang, Huaizhong Lin, Ziqiang Dang, Chenguang Ma, Lei Zhao
  • 🏛️ Institutions: Ant Group, ZJU
  • 📅 Date: February 05, 2026
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Mobile]
  • 🔑 Key: [data mining], [monte carlo tree search], [multi-agent system], [trajectory annotation], [intent recycling], [M$^2$-Miner]
  • 📖 TLDR: M$^2$-Miner is a mobile GUI data-mining system that combines MCTS with multiple collaborating agents to generate and verify high-quality intent-trajectory training data. It also introduces intent recycling and model-in-the-loop training, leading to stronger mobile-agent performance.

• PATHWAYS: Evaluating Investigation and Context Discovery in AI Web Agents

  • Shifat E. Arman, Syed Nazmus Sakib, Tapodhir Karmakar Taton, Nafiul Haque, Shahrear Bin Amin
  • 🏛️ Institutions: Robotics and Mechatronics Engineering, University of Dhaka
  • 📅 Date: February 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [context discovery], [investigation], [hallucination], [PATHWAYS]
  • 📖 TLDR: PATHWAYS is a benchmark of 250 multi-step web decision tasks designed to test whether agents can uncover and correctly use hidden contextual information instead of stopping at surface cues. The results show that agents often fail to retrieve decisive hidden evidence, hallucinate investigative reasoning, and struggle to incorporate discovered context into final decisions.

• SVRepair: Structured Visual Reasoning for Automated Program Repair

  • Xiaoxuan Tang, Jincheng Wang, Liwei Luo, Jingxuan Xu, Sheng Zhou, Dajun Chen, Wei Jiang, Yong Li
  • 🏛️ Institutions: Ant Group, Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, ZJU
  • 📅 Date: February 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI understanding], [program repair], [scene graph], [visual reasoning], [coding agent]
  • 📖 TLDR: SVRepair is a multimodal automated program repair framework that fine-tunes a vision-language model to convert GUI visual artifacts (screenshots, control-flow graphs) into structured semantic scene graphs, enabling a coding agent to localize faults and synthesize patches with state-of-the-art results on SWE-Bench M, MMCode, and CodeVision benchmarks.

• UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents

  • Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, Hongsheng Li
  • 🏛️ Institutions: Multimedia Laboratory (MMLab), CUHK, vivo AI Lab, Princeton, Shenzhen Loop Area Institute, Shanghai AI Laboratory
  • 📅 Date: February 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [memory], [reinforcement learning], [online RL], [hierarchical experience memory], [stratified group sampling], [UI-Mem]
  • 📖 TLDR: UI-Mem augments online RL for mobile GUI agents with a hierarchical experience memory that stores reusable workflows, subtask skills, and failure patterns as transferable templates. Its stratified group sampling lets the policy absorb memory-informed behavior and improves over standard online RL baselines.

• Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents

  • Sizhe Tang, Rongqian Chen, Tian Lan
  • 🏛️ Institutions: George Washington University
  • 📅 Date: February 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [test-time compute], [tree search], [MCTS], [alpha-UCT], [OSWorld], [Agent Alpha]
  • 📖 TLDR: Agent Alpha applies step-level MCTS with alpha-UCT exploration, comparison-based evaluation, and diversity-aware expansion to computer-use agents. It improves OSWorld performance substantially over trajectory-level sampling under similar compute.

• LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios

  • Tianyu Chen, Chujia Hu, Ge Gao, Ruofeng Yu, Yao Lu
  • 🏛️ Institutions: ShanghaiTech University, Shanghai AI Laboratory, Rice University
  • 📅 Date: February 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [safety], [MCP], [long-horizon planning], [LPS-Bench]
  • 📖 TLDR: LPS-Bench is a benchmark evaluating the planning-time safety awareness of MCP-based computer-use agents under long-horizon tasks, covering 65 scenarios across 7 task domains and 9 risk types with both benign and adversarial interactions, revealing substantial safety deficiencies in existing agents.

• MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments

  • Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Qinyi Luo, Shunye Tang, Yuxiang Chai, Weifeng Lin, Han Xiao, WenHao Wang, Siheng Chen, Zhengxi Lu, Gao Wu, Hao Wang, Liang Liu, Yong Liu
  • 🏛️ Institutions: ZJU, Nankai University, CUHK, SJTU, vivo AI Lab
  • 📅 Date: February 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dataset], [memory], [evaluation], [MemGUI-Eval], [MemGUI-Bench]
  • 📖 TLDR: MemGUI-Bench is a memory-focused benchmark for mobile GUI agents covering dynamic tasks that require cross-temporal and cross-spatial retention. Paired with MemGUI-Eval, it reveals large hidden memory deficits in current agents that standard benchmarks miss.

• WebSentinel: Detecting and Localizing Prompt Injection Attacks for Web Agents

  • Xilong Wang, Yinuo Liu, Zhun Wang, Dawn Song, Neil Gong
  • 🏛️ Institutions: Duke University, UC Berkeley
  • 📅 Date: February 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [defense], [prompt injection], [WebSentinel], [security], [attack detection]
  • 📖 TLDR: WebSentinel is a two-step defense framework that detects and localizes prompt injection attacks in webpages by first extracting segments of interest that may be contaminated and then evaluating each segment's consistency with the webpage content, substantially outperforming baseline methods on multiple datasets.

• Avenir-Web: Human-Experience-Imitating Multimodal Web Agents with Mixture of Grounding Experts

  • Aiden Yiliu Li, Xinyue Hao, Shilong Liu, Mengdi Wang
  • 🏛️ Institutions: UCL, Princeton, University of Edinburgh
  • 📅 Date: February 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [mixture of grounding experts], [experience-imitation planning], [adaptive memory], [online-Mind2Web], [Avenir-Web]
  • 📖 TLDR: Avenir-Web combines mixture-of-grounding experts, experience-imitation planning, task-tracking checklists, and adaptive memory for live web tasks. It reaches strong open-source performance on Online-Mind2Web and narrows the gap to proprietary systems.

• SafePred: A Predictive Guardrail for Computer-Using Agents via World Models

  • Yurun Chen, Zeyi Liao, Ping Yin, Taotao Xie, Keting Yin, Shengyu Zhang
  • 🏛️ Institutions: Tsinghua, OSU, CUHK-Shenzhen
  • 📅 Date: February 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [guardrail], [world model], [risk prediction], [safety], [long-horizon risk], [SafePred]
  • 📖 TLDR: SafePred is a predictive guardrail that uses a world model to simulate future states and assess delayed risk before a computer-use agent acts. It targets long-horizon hazards that reactive safety filters tend to miss.

• Agentic Reward Modeling: Verifying GUI Agent via Online Proactive Interaction

  • Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, Zhixiong Zeng
  • 🏛️ Institutions: Institute of Automation, CAS, University of Chinese Academy of Sciences, Meituan, Beijing Jiaotong University
  • 📅 Date: January 31, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reward modeling], [verification], [proactive interaction], [VAGEN], [OSWorld], [AndroidWorld]
  • 📖 TLDR: VAGEN turns GUI-agent verification into an active interaction problem, using a verifier agent to probe the environment for evidence of task completion instead of relying on passive judgment. This substantially improves verification accuracy on OSWorld and AndroidWorld.

• Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution

  • Hongze Mi, Yibo Feng, WenJie Lu, Song Cao, Jinyuan Li, Yanming Li, Xuelin Zhang, Haotian Luo, Songyang Peng, He Cui, Tengfei Tian, Jun Fang, Hua Chai, Naiqiang Tan
  • 🏛️ Institutions: Didichuxing Co. Ltd, CUHK-Shenzhen, Tianjin University, Sun Yat-sen University, Fudan
  • 📅 Date: January 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [memory system], [training-free], [self-regulating memory], [multi-app], [Darwinian Memory], [trajectory pruning]
  • 📖 TLDR: Darwinian Memory is a training-free memory system that treats agent memories as an evolving ecosystem, selecting and pruning trajectories through utility-driven competition. It improves success rate and execution stability on real-world multi-app GUI benchmarks.

• Learning with Challenges: Adaptive Difficulty-Aware Data Generation for Mobile GUI Agent Training

  • Linjia Kang, Zhimin Wang, Yongkang Zhang, Duo Wu, Jinghe Wang, Ming Ma, Haopeng Yan, Zhi Wang
  • 🏛️ Institutions: Tsinghua, Huazhong Agricultural University, Kuaishou Technology
  • 📅 Date: January 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [data generation], [curriculum learning], [difficulty-aware generation], [capability frontier], [MobileGen], [training data]
  • 📖 TLDR: MobileGen generates mobile GUI training data with difficulty-aware control over structural and semantic challenge. By matching synthesized trajectories to an agent's capability frontier, it improves downstream mobile-agent performance substantially over baseline data generation schemes.

• SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization

  • Jinyang Wu, Changpeng Yang, Yuhao Shen, Fangzhi Xu, Bolin Ni, Chonghua Liao, Yuchen Liu, Hongzhen Wang, Shuai Nie, Shuai Zhang, Haoran Luo, Jiaming Xu
  • 🏛️ Institutions: Tsinghua, Xiaomi, ZJU, NTU, Institute of Automation, CAS
  • 📅 Date: January 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reinforcement learning], [tiered rewards], [reward shaping], [GUI grounding], [planning], [SSL]
  • 📖 TLDR: SSL replaces binary verifier rewards with progressively amplified tiered rewards that distinguish higher- and lower-quality successful trajectories. Across GUI perception, short- and long-horizon planning, and reasoning benchmarks, it improves optimization stability and reaches up to 2.5x better sample efficiency than binary-reward baselines.

• ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents

  • Xiaoce Wang, Guibin Zhang, Junzhe Li, Jinzhe Tu, Chun Li, Ming Li
  • 🏛️ Institutions: Tsinghua, NUS, PKU, Shenzhen MSU-BIT University, Guangming Laboratory
  • 📅 Date: January 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [tool use], [tokenization], [curriculum learning], [visual grounding], [data efficiency], [ToolTok]
  • 📖 TLDR: ToolTok treats GUI operations as paths over learnable tool tokens with semantic anchoring and curriculum learning. This makes GUI agents more efficient and generalizable, reaching competitive performance with far less training data than other post-training methods.

• Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints for Reliable Decision-Making

  • Ruoyu Chen, Shangquan Sun, Xiaoqing Guo, Sanyi Zhang, Kangwei Liu, Shiming Liu, Zhangcheng Wang, Qunli Zhang, Hua Zhang, Xiaochun Cao
  • 🏛️ Institutions: Institute of Information Engineering, CAS, University of Chinese Academy of Sciences, NTU, Hong Kong Baptist University, Communication University of China, Huawei, USTC, Sun Yat-sen University
  • 📅 Date: January 30, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [training], [attribution], [subset-based attribution constraints], [human prior alignment], [reliability], [explainability]
  • 📖 TLDR: This paper proposes prior-aligned training with subset-based attribution constraints, penalizing models for relying on evidence that conflicts with human priors. It improves both accuracy and decision reasonability on image classification and GUI-agent click decision tasks.

• BEAP-Agent: Backtrackable Execution and Adaptive Planning for GUI Agents

  • Ziyu Lu, Tengjin Weng, Yiying Yang, Yuhang Zhao, Xinxin Huang, Wenhao Jiang
  • 🏛️ Institutions: Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen University, Guangdong University of Technology
  • 📅 Date: January 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [backtracking], [DFS planning], [task tracking], [error recovery], [OSWorld], [BEAP-Agent]
  • 📖 TLDR: BEAP-Agent formulates long-horizon GUI execution as depth-first search and adds explicit multi-level backtracking when the agent commits to a wrong exploration branch. Its Planner, Executor, and Tracker jointly support state rollback and task updates, improving recovery on OSWorld.

• DynaWeb: Model-Based Reinforcement Learning of Web Agents

  • Hang Ding, Peidong Liu, Junqiao Wang, Ziwei Ji, Meng Cao, Rongzhao Zhang, Lynn Ai, Eric Yang, Tianyu Shi, Lei Yu
  • 🏛️ Institutions: SJTU, Sichuan University, HKUST, McGill University, Shanghai AI Laboratory, Gradient, University of Toronto, Mila
  • 📅 Date: January 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [model-based reinforcement learning], [world model], [imagined rollouts], [WebArena], [WebVoyager], [DynaWeb]
  • 📖 TLDR: DynaWeb trains web agents with model-based reinforcement learning by learning a web world model that supports imagined rollouts, then interleaving those rollouts with real expert trajectories. This synthetic-environment training loop improves open-source web agents on both WebArena and WebVoyager.

• How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors

  • Kuai Yu, Naicheng Yu, Han Wang, Rui Yang, Huan Zhang
  • 🏛️ Institutions: Columbia, UC San Diego, UIUC
  • 📅 Date: January 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [evaluation pipeline], [visual attribute factors], [VAF], [visual grounding], [robustness]
  • 📖 TLDR: This paper introduces VAF, a controlled evaluation pipeline for measuring how benign UI visual attributes affect web-agent choices while preserving page semantics. Across 48 variants, 5 real websites, and 4 agents, it shows that contrast, size, position, and card clarity strongly influence agent behavior, while several text-style factors matter much less.

• WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents

  • Yao Zhang, Shijie Tang, Zeyu Li, Zhen Han, Volker Tresp
  • 🏛️ Institutions: LMU Munich, TUM, Munich Center for Machine Learning (MCML)
  • 📅 Date: January 29, 2026
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Web]
  • 🔑 Key: [process reward model], [reasoning verifier], [WebPRMBench], [reward-guided search], [WebArbiter]
  • 📖 TLDR: WebArbiter is a reasoning-first web process reward model that generates structured justifications, a preference verdict, and the better next action instead of a single scalar score. It also introduces WebPRMBench and outperforms prior web PRMs on both benchmark evaluation and reward-guided trajectory search.

• CUA-Skill: Develop Skills for Computer Using Agent

  • Tianyi Chen, Yinheng Li, Michael Solodko, Sen Wang, Nan Jiang, Tingyuan Cui, Junheng Hao, Jongwoo Ko, Sara Abdali, Leon Xu, Suzhen Zheng, Hao Fan, Pashmina Cameron, Justin Wagle, Kazuhito Koishida
  • 🏛️ Institutions: Microsoft
  • 📅 Date: January 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [skill abstraction], [skill library], [composition graphs], [desktop automation], [Windows], [CUA-Skill]
  • 📖 TLDR: CUA-Skill builds a reusable skill base for computer-use agents by encoding human computer-use knowledge as parameterized skills plus execution and composition graphs. The resulting CUA-Skill Agent improves robustness and reaches strong performance on WindowsAgentArena through dynamic skill retrieval and memory-aware recovery.

• Continual GUI Agents

  • Ziwei Liu, Borui Kang, Hangjie Yuan, Zixiang Zhao, Wei Li, Yifan Zhu, Tao Feng
  • 🏛️ Institutions: Tsinghua, ZJU, ETH, Beijing University of Posts and Telecommunications
  • 📅 Date: January 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [continual learning], [reinforcement fine-tuning], [domain shift], [resolution shift], [GUI-AiF]
  • 📖 TLDR: This paper defines continual GUI agents as agents that must keep grounding accurately while interface domains and resolutions shift over time. It introduces GUI-AiF, a reinforcement fine-tuning method with anchoring-point and anchoring-region rewards, and reports stronger continual-domain and continual-resolution performance than prior baselines.

• MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment

  • Qinzhuo Wu, Zhizhuo Yang, Hanhao Li, Pengzhi Gao, Wei Liu, Jian Luan
  • 🏛️ Institutions: MiLM Plus, Xiaomi, PKU, CUHK
  • 📅 Date: January 28, 2026
  • 📑 Publisher: Findings of ACL 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [chinese benchmark], [long-horizon tasks], [noise robustness], [auto-evaluation], [MobileBench-OL]
  • 📖 TLDR: MobileBench-OL benchmarks mobile GUI agents on 1,080 online tasks from 80 Chinese apps. It extends evaluation beyond instruction following to long-horizon execution, reasoning and exploration, and robustness to real-world noise, and pairs the benchmark with an automatic evaluation pipeline that supports environment reset.

• OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive Tasks

  • Jing Wu, Daphne Barretto, Yiye Chen, Nicholas Gydé, Yanan Jian, Yuhang He, Vibhav Vineet
  • 🏛️ Institutions: Oxford, Microsoft, Georgia Tech
  • 📅 Date: January 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [long-horizon tasks], [repetitive workflows], [condensed demonstrations], [OS-Marathon]
  • 📖 TLDR: OS-Marathon benchmarks computer-use agents on 242 long-horizon repetitive desktop workflows such as expense processing and grade entry. The paper also introduces a few-shot condensed-demonstration method for teaching the recurring sub-workflow logic behind these tasks.

• OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution

  • Le Zhang, Yixiong Xiao, Xinjiang Lu, Jingjia Cao, Yusai Zhao, Jingbo Zhou, Lang An, Zikan Feng, Wanxiang Sha, Yu Shi, Congxi Xiao, Jian Xiong, Yankai Zhang, Hua Wu, Haifeng Wang
  • 🏛️ Institutions: Baidu Frontier Research Department
  • 📅 Date: January 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [model], [synthetic data], [Mixture-of-Experts], [GRPO], [OS-Nav], [ChiM-Nav], [Ubu-Nav], [OmegaUse]
  • 📖 TLDR: OmegaUse is a general-purpose GUI agent for both phone-use and computer-use settings trained with a curated-plus-synthetic data pipeline and a two-stage SFT-then-GRPO recipe on an MoE backbone. It also introduces the OS-Nav suite and reports strong cross-terminal results on ScreenSpot-v2, AndroidControl, ChiM-Nav, and Ubu-Nav.

• MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution

  • Libo Sun, Jiwen Zhang, Siyuan Wang, Zhongyu Wei
  • 🏛️ Institutions: Fudan, USC, Shanghai Innovation Institute
  • 📅 Date: January 27, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [dataset], [procedural memory], [stationary memory], [dynamic memory evolution], [UI-40K], [MAGNET]
  • 📖 TLDR: MAGNET targets appearance drift and workflow drift in mobile GUI agents with dual-level memory: stationary memory for stable functional semantics and procedural memory for reusable task workflows. It also adds a dynamic memory evolution mechanism and the UI-40K dataset, and improves results on AndroidWorld and offline distribution-shift benchmarks.

• GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models

  • Shaokang Wang, Pei Fu, Ruoceng Zhang, Shaojie Zhang, Xiuwen Xi, Jiahui Yang, Bin Qin, Ying Huang, Zhenbo Luo, Jian Luan
  • 🏛️ Institutions: MiLM Plus, Xiaomi
  • 📅 Date: January 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [critic model], [test-time scaling], [data flywheel], [Intuitive Critic Model], [GAIA]
  • 📖 TLDR: GAIA trains an Intuitive Critic Model that judges the immediate correctness of candidate GUI actions before execution. It then uses a data flywheel that recycles agent-generated positive and negative action samples to iteratively improve the critic, yielding better test-time performance for both open-source and closed-source GUI agents.

• GUIGuard: Toward a General Framework for Privacy-Preserving GUI Agents

  • Yanxi Wang, Zhiling Zhang, Wenbo Zhou, Weiming Zhang, Jie Zhang, Qiannan Zhu, Yu Shi, Shuxin Zheng, Jiyan He
  • 🏛️ Institutions: Beijing Normal University, Zhongguancun Academy, USTC, A*STAR, Zhongguancun Institution of Artificial Intelligence
  • 📅 Date: January 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [dataset], [privacy recognition], [privacy protection], [privacy grounding], [GUIGuard], [GUIGuard-Bench]
  • 📖 TLDR: GUIGuard formulates privacy-preserving GUI automation as privacy recognition, privacy protection, and protected task execution. It also introduces GUIGuard-Bench, a cross-platform benchmark with 630 trajectories and 13,830 screenshots annotated for region-level privacy grounding, risk level, privacy category, and task necessity, and shows that current models still recognize private content very poorly.

• LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent

  • Bin Kang, Shaoguo Wen, Yifei Bi, Shunlong Wu, Xinbin Yuan, Rui Shao, Junle Wang, Zhuotao Tian
  • 🏛️ Institutions: Chengdu Institute of Computer Applications, CAS, University of Chinese Academy of Sciences, Tencent Turing Lab, Georgia Tech, Tsinghua, Nankai University, Shenzhen Loop Area Institute
  • 📅 Date: January 26, 2026
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [framework], [benchmark], [long-horizon tasks], [reflection], [rollback], [LongGUIBench], [LongHorizonUI]
  • 📖 TLDR: LongHorizonUI targets error accumulation in long-horizon GUI control by combining indexed multimodal perception, structured reflective decision-making, and rollback-based compensatory execution. It also introduces LongGUIBench for tasks longer than 15 steps across games and complex applications, and reports substantial gains on long-horizon evaluation while staying competitive on public benchmarks.

• SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis

  • Xuan Wang, Siyuan Su, Quantong Fu, Yongxiang Hu, Yangfan Zhou
  • 🏛️ Institutions: Fudan
  • 📅 Date: January 26, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dataset], [swipe synthesis], [swipe execution], [SwipeBench], [GUISwiper], [SwipeGen]
  • 📖 TLDR: SwipeGen decomposes human swipe gestures into quantifiable dimensions and uses GUI exploration to synthesize human-like swipe interactions. It also releases SwipeBench and trains GUISwiper on the synthesized data, reaching 69.07% swipe execution accuracy, a 214% improvement over existing VLM baselines.

• EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents

  • Ying Mo, Yu Bai, Dapeng Sun, Yuqian Shi, Yukai Miao, Li Chen, Dan Li
  • 🏛️ Institutions: Zhongguancun Laboratory, Tsinghua
  • 📅 Date: January 25, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [environment], [enterprise workflows], [schema-grounded task generation], [SQL verification], [EntWorld]
  • 📖 TLDR: EntWorld introduces a verifiable enterprise-agent environment and a 1,756-task benchmark spanning six business domains such as CRM, ITIL, and ERP. It synthesizes workflows from database schemas and uses SQL-based deterministic verification instead of visual matching, and current top models still trail human performance by a large margin.

• GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph

  • Mingxian Yu, Siqi Luo, Xu Chen
  • 🏛️ Institutions: Sun Yat-sen University
  • 📅 Date: January 24, 2026
  • 📑 Publisher: Journal of Intelligent Computing and Networking
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [knowledge graph], [one-query planning], [validator], [GraphPilot]
  • 📖 TLDR: GraphPilot builds app-specific knowledge graphs that encode page functions, element roles, and transition rules, then uses them to plan nearly complete action sequences in almost one LLM query. On DroidTask it improves task completion while sharply reducing latency and the number of LLM calls relative to stepwise mobile agents.

• EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience

  • Taofeng Xue, Chong Peng, Mianqiu Huang, Linsen Guo, Tiancheng Han, Haozhe Wang, Jianing Wang, Xiaocheng Zhang, Xin Yang, Dengchang Zhao, Jinrui Ding, Xiandi Ma, Yuchen Xie, Peng Pei, Xunliang Cai, Xipeng Qiu
  • 🏛️ Institutions: Meituan, Fudan, Tongji University, HKUST
  • 📅 Date: January 22, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [model], [synthetic experience], [verifiable synthesis], [OSWorld], [EvoCUA]
  • 📖 TLDR: EvoCUA replaces static imitation with an evolving training loop built on verifiable task synthesis, high-throughput sandbox rollouts, and iterative policy optimization from both successful and failed trajectories. On OSWorld it reaches 56.7% success, outperforming prior open-source computer-use agents and even some leading closed-weight systems.

• The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes

  • Simret Araya Gebreegziabher, Yukun Yang, Charles Chiang, Hojun Yoo, Chaoran Chen, Hyo Jin Do, Zahra Ashktorab, Werner Geyer, Diego Gómez-Zará, Toby Jia-Jun Li
  • 🏛️ Institutions: University of Notre Dame, IBM Research
  • 📅 Date: January 22, 2026
  • 📑 Publisher: IUI 2026
  • 💻 Env: [Web]
  • 🔑 Key: [testbed], [alignment], [human values], [preferences], [value-action gap]
  • 📖 TLDR: This paper studies how explicit user values and preferences shape the behavior of web agents on 14 interactive tasks built from replica websites. It finds that value- and preference-infused prompts do change trajectories and outcomes, but strong interface cues such as discounts and ads often override those instructions and expose a value-action gap.

• MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux

  • Zecheng Li, Zhihui Cao, Wenke Huang, Yudong Zhang, Keying Qi, Rui Wang, Zeyu Zheng, Jian Zhao, Hao Zhu, Hengxin Wu, Yuran Wang, Guitao Fan, Guokun Wu, Yicong Liu, Zhilin Gao, Haikun Xu, He Yang, Minqi Xiang, Xingyu Liu, Zuojian Wang
  • 🏛️ Institutions: Honor Device Co., Ltd.
  • 📅 Date: January 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reward model], [multi-agent system], [data construction], [data reflux], [MagicGUI-RMS]
  • 📖 TLDR: MagicGUI-RMS combines a domain-specific reward model with a general-purpose reward model to score GUI trajectories, propose corrections, and feed improved data back into later training rounds. It also introduces an automated reward-data construction pipeline and reports gains in task accuracy and behavioral robustness.

• MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction

  • Wenqi Zhang, Yulin Shen, Changyue Jiang, Jiarun Dai, Geng Hong, Xudong Pan
  • 🏛️ Institutions: Fudan, Shanghai Innovation Institute
  • 📅 Date: January 19, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [simulation-to-real], [reasoning correction], [MirrorWorld], [MirrorGuard]
  • 📖 TLDR: MirrorGuard is a plug-and-play defense that trains on high-risk trajectories synthesized in a neural-symbolic text simulator called MirrorWorld, then corrects insecure reasoning before real computer-use agents act. Across multiple benchmarks and architectures, it cuts unsafe behavior sharply while preserving utility better than prior defenses.

• Zero-Permission Manipulation: Can We Trust Large Multimodal Model Powered GUI Agents?

  • Yi Qian, Kunwei Qian, Xingbang He, Ligeng Chen, Jikang Zhang, Tiantai Zhang, Haiyang Wei, Linzhang Wang, Hao Wu, Bing Mao
  • 🏛️ Institutions: National Key Laboratory for Novel Software Technology, NJU, Honor Device Co., Ltd., Institute of Dataspace, Hefei Comprehensive National Science Center
  • 📅 Date: January 18, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [security], [attack], [Android], [Action Rebinding], [verification gates]
  • 📖 TLDR: This paper introduces Action Rebinding, a zero-permission Android attack that exploits the observation-to-action gap in multimodal GUI agents by changing foreground UI state before the planned action executes. Across six agents and 15 tasks it achieves 100% atomic rebinding success, and with intent alignment can also bypass confirmation-style verification gates.

• CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents

  • Hanna Foerster, Tom Blanchard, Kristina Nikolić, Ilia Shumailov, Cheng Zhang, Robert Mullins, Nicolas Papernot, Florian Tramèr, Yiren Zhao
  • 🏛️ Institutions: University of Cambridge, University of Toronto, Vector Institute, ETH, AI Sequrity Company
  • 📅 Date: January 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [safety], [prompt injection], [control flow integrity], [Single-Shot Planning], [Branch Steering], [OSWorld], [CaMeLs]
  • 📖 TLDR: This paper adapts the Dual-LLM security paradigm to computer-use agents through Single-Shot Planning, where a trusted planner writes a full branching execution graph before seeing untrusted UI content. That gives control-flow integrity against injected instructions, but the paper also identifies Branch Steering as a remaining data-flow threat and studies its tradeoff with utility on OSWorld.

• Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agents

  • Yurun Song, Jiong Yin, Rongjunchen Zhang, Ian G. Harris
  • 🏛️ Institutions: Hithink Research, UC Irvine, Hangzhou Dianzi University
  • 📅 Date: January 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [policy optimization], [coordinate compression], [Coordinate-Aware Spatial Compression], [distance-based advantage], [CCPO]
  • 📖 TLDR: CCPO tackles context inflation in multi-turn GUI agents by compressing historical screenshots around task-relevant coordinates collected across rollouts. Its Coordinate-Aware Spatial Compression and distance-based advantage improve both compression quality and grounding, reaching state-of-the-art results with up to 55% token compression and 3.8x training speedup.

• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents

  • Chen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu, Xiangcheng Liu, Hantao Yao, Wu Liu
  • 🏛️ Institutions: USTC, Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai Innovation Institute, SJTU
  • 📅 Date: January 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reinforcement learning], [active perception], [tool-augmented perception], [ScreenSpot-pro], [GUI-Eyes]
  • 📖 TLDR: GUI-Eyes frames GUI grounding as active perception, letting the agent learn when and how to call tools such as cropping and zooming inside a two-stage reasoning process. It pairs that policy with a spatially continuous reward for tool use and reaches 44.8% grounding accuracy on ScreenSpot-Pro using only 3k labeled samples.

• PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records

  • Yibo Lyu, Gongwei Chen, Rui Shao, Weili Guan, Liqiang Nie
  • 🏛️ Institutions: HIT-Shenzhen
  • 📅 Date: January 14, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [personalization], [implicit intent], [proactive assistance], [AndroidIntent], [HIM-Agent]
  • 📖 TLDR: PersonalAlign studies personalized GUI agents that must infer omitted preferences and latent routines from long-term user records rather than explicit instructions alone. It introduces the AndroidIntent benchmark with 775 annotated user preferences and 215 routines from 20k records, and proposes HIM-Agent to hierarchically organize those signals for better execution and proactive assistance.

• ExpSeek: Self-Triggered Experience Seeking for Web Agents

  • Wenyuan Zhang, Xinghua Zhang, Haiyang Yu, Shuaiyi Nie, Bingli Wu, Juwei Yue, Tingwen Liu, Yongbin Li
  • 🏛️ Institutions: Institute of Information Engineering, CAS, University of Chinese Academy of Sciences, Tongyi Lab, Alibaba Group
  • 📅 Date: January 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [experience seeking], [entropy-based triggering], [experience intervention], [ExpSeek]
  • 📖 TLDR: ExpSeek turns experience intervention from a passive pre-task context into a step-level mechanism that is triggered only when the agent’s own entropy signal indicates uncertainty. It also generates tailored experience content per step, improving Qwen3-8B and 32B web agents by 9.3 and 7.5 absolute points across four benchmarks.

• WebTrap Park: An Automated Platform for Systematic Security Evaluation of Web Agents

  • Xinyi Wu, Jiagui Chen, Geng Hong, Jiayi Dong, Xudong Pan, Jiarun Dai, Min Yang
  • 🏛️ Institutions: Fudan, Shanghai Innovation Institute
  • 📅 Date: January 13, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [safety], [security evaluation], [behavioral evaluation], [WebTrap Park]
  • 📖 TLDR: WebTrap Park is an automated platform for evaluating web-agent security by directly observing actions on live web pages rather than relying on internal logs. It instantiates malicious user prompts, prompt injections, and deceptive website designs into 1,226 executable tasks and shows clear security differences across agent frameworks.

• ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolution

  • Jihong Wang, Jiamu Zhou, Weiming Zhang, Weiwen Liu, Zhuosheng Zhang, Xingyu Lou, Weinan Zhang, Huarong Deng, Jun Wang
  • 🏛️ Institutions: OPPO Research Institute, SJTU
  • 📅 Date: January 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [human-in-the-loop], [knowledge adaptation], [progress summarization], [WebArena], [ColorBrowserAgent]
  • 📖 TLDR: ColorBrowserAgent targets site heterogeneity and long-horizon instability in web automation with two mechanisms: human-in-the-loop knowledge adaptation and progressive progress summarization. On WebArena it reaches 71.2% success, and the paper also reports transfer to WebChoreArena and gains in industrial deployment.

• ShowUI-Aloha: Human-Taught GUI Agent

  • Yichun Zhang, Xiangwu Guo, Yauhong Goh, Jessica Hu, Zhiheng Chen, Xin Wang, Difei Gao, Mike Zheng Shou
  • 🏛️ Institutions: Show Lab, NUS
  • 📅 Date: January 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [framework], [learning from demonstration], [screen recording], [human teaching], [ShowUI-Aloha]
  • 📖 TLDR: ShowUI-Aloha converts in-the-wild desktop screen recordings into structured teaching trajectories through a recorder, learner, planner, and executor pipeline. The goal is to let GUI agents learn complex desktop tasks from ordinary human demonstrations rather than curated annotations or synthetic traces.

• V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peaking

  • Jikai Chen, Long Chen, Dong Wang, Qinglin Su, Zhixuan Chu, Bingguang Hao, Leilei Gan, Chenyi Zhuang, Jinjie Gu
  • 🏛️ Institutions: ZJU, Inclusion AI, Ant Group
  • 📅 Date: January 11, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [visual attention], [background suppression], [Fitts' law], [V2P]
  • 📖 TLDR: V2P improves attention-based GUI grounding by suppressing irrelevant background regions and using a Fitts’ Law-inspired Gaussian peak to emphasize an element’s actionable center over its edges. The paper reports 92.4% on ScreenSpot-v2 and 52.5% on ScreenSpot-Pro, with ablations confirming both components matter.

• From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation

  • Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang, Zhuzhong Qian, Yan Lu
  • 🏛️ Institutions: NJU, PKU, MSR Asia
  • 📅 Date: January 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reinforcement learning], [RLVR], [off-policy assimilation], [BEPA], [OSWorld]
  • 📖 TLDR: BEPA improves end-to-end GUI-agent training with verifiable rewards by turning scarce off-policy expert traces into policy-aligned guidance through self-rolled reachable trajectories and a dynamically updated per-task cache. On OSWorld-Verified it raises UI-TARS-1.5-7B from 22.87% to 32.13%, with additional gains on MMBench-GUI and Online-Mind2Web.

• GUITester: Enabling GUI Agents for Exploratory Defect Discovery

  • Yifei Gao, Jiang Wu, Xiaoyi Chen, Yifan Yang, Zhe Cui, Tianyi Ma, Jiaming Zhang, Jitao Sang
  • 🏛️ Institutions: Beijing Jiaotong University, Hithink Research, NTU
  • 📅 Date: January 08, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [benchmark], [GUI testing], [defect discovery], [GUITestBench], [GUITester]
  • 📖 TLDR: GUITester targets exploratory defect discovery in mobile apps, where agents must both navigate and recognize that anomalous behavior is a product defect rather than their own mistake. It introduces GUITestBench with 143 tasks across 26 defects and a multi-agent framework that separates planning-execution from hierarchical reflection, reaching 48.90% F1 (Pass@3).

• Agent-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learning

  • Zheng Wu, Xingyu Lou, Xinbei Ma, Yansi Li, Weiwen Liu, Weinan Zhang, Jun Wang, Zhuosheng Zhang
  • 🏛️ Institutions: SJTU, OPPO Research Institute
  • 📅 Date: January 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [continual learning], [parameter fusion], [catastrophic forgetting], [geometric consensus], [Agent-Dice]
  • 📖 TLDR: Agent-Dice addresses the stability-plasticity dilemma in agent continual learning by separating shared knowledge from task-specific interference during parameter fusion. Its two-stage method combines geometric consensus filtering with curvature-based importance weighting, and the paper reports strong continual-learning results for both GUI and tool-use agents.

• InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training

  • Ziyun Zhang, Zezhou Wang, Xiaoyi Zhang, Zongyu Guo, Jiahao Li, Bin Li, Yan Lu
  • 🏛️ Institutions: PKU, NJU, MSR Asia
  • 📅 Date: January 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [environment synthesis], [data generation], [verifiable rewards], [InfiniteWeb]
  • 📖 TLDR: InfiniteWeb automatically builds functional multi-page web environments for GUI-agent training rather than just generating isolated webpages. It uses unified specifications, task-centric test-driven development, and reference design images, and the resulting environments improve agent training on Online-Mind2Web and OSWorld.

• MobileDreamer: Generative Sketch World Model for GUI Agent

  • Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Wenji Mao, Wan Guanglu
  • 🏛️ Institutions: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, University of Chinese Academy of Sciences, Meituan
  • 📅 Date: January 07, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [world model], [lookahead], [rollout imagination], [MobileDreamer]
  • 📖 TLDR: MobileDreamer equips mobile GUI agents with a lightweight world model that predicts task-relevant textual sketches of future interface states instead of full screenshots. It then uses rollout imagination over those predicted futures for action selection, improving AndroidWorld performance by 5.25% and reaching state of the art.

• WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks

  • Hao Bai, Alexey Taymanov, Tong Zhang, Aviral Kumar, Spencer Whitehead
  • 🏛️ Institutions: Microsoft, UIUC, CMU
  • 📅 Date: January 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [training environment], [reinforcement learning], [asynchronous rollouts], [WebGym]
  • 📖 TLDR: WebGym provides a large-scale open training environment for visual web agents with nearly 300,000 rubric-evaluated tasks on realistic websites. It also includes a high-throughput asynchronous rollout system, and agents fine-tuned on WebGym improve from 26.2% to 42.9% on out-of-distribution websites, outperforming GPT-4o and GPT-5-Thinking.

• ShowUI-π: Flow-based Generative Models as GUI Dexterous Hands

  • Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou
  • 🏛️ Institutions: Show Lab, NUS
  • 📅 Date: December 31, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [dataset], [benchmark], [model], [drag interaction], [flow-based model], [continuous action], [ScreenDrag], [ShowUI-π]
  • 📖 TLDR: ShowUI-π treats GUI dragging as a continuous dexterous-control problem rather than only discrete point prediction, while still supporting ordinary click actions in the same model. It also introduces ScreenDrag with 20K trajectories across five domains, and the 450M-parameter model outperforms much larger proprietary GUI agents on this benchmark.

• It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents

  • Karolina Korgul, Yushi Yang, Arkadiusz Drohomirecki, Piotr Błaszczyk, Will Howard, Lukas Aichberger, Chris Russell, Philip H.S. Torr, Adam Mahdi, Adel Bibi
  • 🏛️ Institutions: Oxford, SoftServe, Independent, Johannes Kepler University Linz
  • 📅 Date: December 29, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [safety], [prompt injection], [social engineering], [TRAP]
  • 📖 TLDR: TRAP studies persuasion-style prompt injection on realistic cloned websites, varying factors such as injection interface, persuasion principle, placement, and tailoring. Across six frontier models, it finds web agents are redirected in 25% of tasks on average, and small interface or contextual changes often double attack success.

• DECEPTICON: How Dark Patterns Manipulate Web Agents

  • Phil Cuvin, Hao Zhu, Diyi Yang
  • 🏛️ Institutions: Stanford
  • 📅 Date: December 28, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [safety], [dark patterns], [human benchmark], [DECEPTICON]
  • 📖 TLDR: DECEPTICON isolates individual dark patterns in 700 web-navigation tasks, including 600 generated tasks and 100 real-world ones, to measure both task success and manipulation effectiveness. It finds dark patterns steer state-of-the-art web agents toward malicious outcomes in over 70% of tested tasks, exceed human susceptibility, and remain hard to mitigate with current defenses.

• MAI-UI Technical Report: Real-World Centric Foundation GUI Agents

  • Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, Steven Hoi
  • 🏛️ Institutions: Tongyi Lab, Alibaba Group
  • 📅 Date: December 26, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [model], [agent-user interaction], [MCP], [device-cloud collaboration], [online reinforcement learning], [MAI-UI]
  • 📖 TLDR: MAI-UI is a foundation GUI-agent family aimed at realistic deployment rather than benchmark-only optimization. It extends pure UI control with agent-user interaction, MCP tool calls, native device-cloud collaboration, and long-horizon online reinforcement learning, and sets strong results on grounding, AndroidWorld, and MobileWorld.

• SmartSnap: Proactive Evidence Seeking for Self-Verifying Agents

  • Shaofei Cai, Yulei Qin, Haojia Lin, Zihan Xu, Gang Li, Yuchen Shi, Zongyi Li, Yong Mao, Siqi Cai, Xiaoyu Tan, Yitao Liang, Ke Li, Xing Sun
  • 🏛️ Institutions: Tencent Youtu Lab, Institute for Artificial Intelligence, PKU
  • 📅 Date: December 26, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reinforcement learning], [self-verification], [evidence seeking], [LLM-as-a-judge], [AndroidLab]
  • 📖 TLDR: SmartSnap turns task verification from a passive post-hoc check into proactive evidence seeking, training mobile GUI agents to collect a minimal set of decisive snapshots under the 3C principles so an LLM judge can verify success more reliably, yielding large gains on AndroidLab across 8B and 30B agents.

• iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception

  • Sarthak Mehrotra, Sairam V C Rebbapragada, Mani Hemanth Reddy Bonthu, Vineeth N Balasubramanian
  • 🏛️ Institutions: Indian Institute of Technology, Bombay, Indian Institute of Technology, Hyderabad
  • 📅 Date: December 26, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [slow-fast reasoning], [visual grounding], [latent thinking], [adaptive perception], [iSHIFT]
  • 📖 TLDR: iSHIFT is a 2.5B GUI agent that combines latent thinking with perception-control tokens so it can switch between a fast global mode and a slower grounding-heavy mode. The paper positions this as a way to allocate reasoning depth and visual focus adaptively while still matching state-of-the-art results on multiple GUI benchmarks.

• AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents

  • Yue Cao, Yingyao Wang, Pi Bu, Jingxuan Xing, Wei Jiang, Zekun Zhu, Junpeng Ma, Sashuai Zhou, Tong Lu, Jun Song, Yu Cheng, Yuning Jiang, Bo Zheng
  • 🏛️ Institutions: NJU, Alibaba Group, Fudan, ZJU
  • 📅 Date: December 24, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [long-latency tasks], [nested sub-targets], [Average Task Progress], [AndroidLens]
  • 📖 TLDR: AndroidLens evaluates Android GUI agents on 571 long-latency tasks from 38 domains in both Chinese and English settings, with each task decomposed into nested sub-targets. It combines anomaly-preserving static evaluation with milestone-based Average Task Progress, and the paper reports that even the best models remain far from robust on these tasks.

• EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration

  • Runze Li, Yuwen Zhai, Bo Xu, LiWu Xu, Nian Shi, Wei Zhang, Ran Lin, Liang Wang
  • 🏛️ Institutions: East China Normal University, Alibaba Group, Shanghai Innovation Institute
  • 📅 Date: December 22, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [actionable memory], [trajectory retrieval], [critic-guided exploration], [in-context learning], [EchoTrail-GUI]
  • 📖 TLDR: EchoTrail-GUI tackles GUI-agent "digital amnesia" by first using critic-guided self-exploration to build a human-free repository of successful trajectories, then retrieving relevant ones as actionable memories for new tasks. The paper shows this memory loop improves both success rate and efficiency on Android World and AndroidLab.

• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments

  • Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, Yue Wang
  • 🏛️ Institutions: Tongyi Lab, Alibaba Group, HKUST(GZ), University of Florida
  • 📅 Date: December 22, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [agent-user interaction], [MCP], [cross-app workflows], [long-horizon tasks], [MobileWorld]
  • 📖 TLDR: MobileWorld is a harder mobile-agent benchmark built to move beyond AndroidWorld by adding longer cross-app workflows, explicit user interaction, and MCP-augmented tool use. Across 201 tasks over 20 apps, it shows that current agents remain weak at clarification, memory, tool integration, and long-horizon coordination.

• DAVE: A VLM Vision Encoder for Document Understanding and Web Agents

  • Brandon Huang, Hang Hua, Zhuoran Yu, Trevor Darrell, Rogerio Feris, Roei Herzig
  • 🏛️ Institutions: MIT-IBM Watson AI Lab, UC Berkeley, University of Wisconsin-Madison
  • 📅 Date: December 19, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [vision encoder], [document understanding], [web localization], [self-supervised pretraining], [DAVE]
  • 📖 TLDR: DAVE is a vision encoder tailored to document understanding and web-agent workloads, where structural and spatial features matter more than generic image semantics. Its pipeline combines self-supervised pretraining on unlabeled document and web images with supervised pretraining, model merging across decoder setups, and ensemble training with generalist encoders.

• OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models

  • Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Qiushi Sun, Zhaoyang Liu, Zhoumianze Liu, Yu Qiao, Xiangyu Yue, Zun Wang, Zichen Ding
  • 🏛️ Institutions: SJTU, Shanghai AI Laboratory, CUHK MMLab, HKU, HKUST
  • 📅 Date: December 18, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [critic model], [benchmark], [step-level evaluation], [CP-GRPO], [OS-Critic Bench], [OS-Oracle]
  • 📖 TLDR: OS-Oracle targets step-level action criticism for computer-use agents with a 310k-sample cross-platform training pipeline, a two-stage SFT plus CP-GRPO recipe, and the OS-Critic Bench benchmark. The resulting 7B critic reaches state of the art among open-source VLM critics and improves downstream GUI agents when used as a pre-critic.

• VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks

  • Beitong Zhou, Zhexiao Huang, Yuan Guo, Zhangxuan Gu, Tianyu Xia, Zichen Luo, Fei Tang, Dehan Kong, Yanyi Shang, Suling Ou, Zhenlin Guo, Changhua Meng, Shuheng Shen
  • 🏛️ Institutions: Venus Team, Ant Group, iMean AI
  • 📅 Date: December 18, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [benchmark], [GUI grounding], [bilingual benchmark], [hierarchical evaluation], [VenusBench-GD]
  • 📖 TLDR: VenusBench-GD is a bilingual GUI grounding benchmark spanning mobile, desktop, and web platforms, and organizes evaluation into basic and advanced grounding tasks. The paper uses this hierarchy to show that general-purpose multimodal models have mostly caught up on basic grounding, while advanced tasks still expose substantial reasoning and robustness gaps.

• Step-GUI Technical Report

  • Haolong Yan, Jia Wang, Xin Huang, Yeqing Shen, Ziyang Meng, Zhimin Fan, Kaijun Tan, Jin Gao, Lieyu Shi, Mi Yang, Shiliang Yang, Zhirui Wang, Brian Li, Kang An, Chenyang Li, Lei Lei, Mengmeng Duan, Danxun Liang, Guodong Liu, Hang Cheng, Hao Wu, Jie Dong, Junhao Huang, Mei Chen, Renjie Yu, Shunshan Li, Xu Zhou, Yiting Dai, Yineng Deng, Yingdan Liang, Zelin Chen, Wen Sun, Chengxu Yan, Chunqin Xu, Dong Li, Fengqiong Xiao, Guanghao Fan, Guopeng Li, Guozhen Peng, Hongbing Li, Hang Li, Hongming Chen, Jingjing Xie, Jianyong Li, Jingyang Zhang, Jiaju Ren, Jiayu Yuan, Jianpeng Yin, Kai Cao, Liang Zhao, Liguo Tan, Liying Shi, Mengqiang Ren, Min Xu, Manjiao Liu, Mao Luo, Mingxin Wan, Na Wang, Nan Wu, Ning Wang, Peiyao Ma, Qingzhou Zhang, Qiao Wang, Qinlin Zeng, Qiong Gao, Qiongyao Li, Shangwu Zhong, Shuli Gao, Shaofan Liu, Shisi Gao, Shuang Luo, Xingbin Liu, Xiaojia Liu, Xiaojie Hou, Xin Liu, Xuanti Feng, Xuedan Cai, Xuan Wen, Xianwei Zhu, Xin Liang, Xin Zhou, Yifan Sui, Yingxiu Zhao, Yukang Shi, Yunfang Xu, Yuqing Zeng, Yixun Zhang, Zejia Weng, Zhonghao Yan, Zhiguo Huang, Zhuoyu Wang, Zihan Yan, Zheng Ge, Jing Li, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Daxin Jiang
  • 🏛️ Institutions: StepFun
  • 📅 Date: December 17, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [model], [Calibrated Step Reward System], [self-evolving training], [GUI-MCP], [AndroidDaily], [Step-GUI]
  • 📖 TLDR: Step-GUI centers on a self-evolving training pipeline built around the Calibrated Step Reward System, which calibrates model-generated GUI trajectories against trajectory-level signals to produce high-quality supervision at much lower annotation cost. On top of that pipeline, the paper introduces 4B and 8B GUI specialist models, the GUI-MCP protocol, and the AndroidDaily benchmark.

• MobileWorldBench: Towards Semantic World Modeling For Mobile Agents

  • Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Aditya Grover
  • 🏛️ Institutions: UCLA, Panasonic AI Research, Salesforce AI Research
  • 📅 Date: December 16, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [dataset], [benchmark], [semantic world model], [world model], [planning], [MobileWorldBench]
  • 📖 TLDR: MobileWorldBench studies mobile world modeling through language-described state transitions instead of pixel prediction. It benchmarks vision-language models as mobile world models, releases a 1.4M-sample MobileWorld training set, and shows that these semantic world models can directly improve downstream mobile-agent planning.

• Modular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents

  • Youngmin Im, Byeongung Jo, Jaeyoung Wi, Seungwoo Baek, Tae Hoon Min, Joo Hyung Lee, Sangeun Oh, Insik Shin, Sunjae Lee
  • 🏛️ Institutions: KAIST, Sungkyunkwan University, Korea University, Fluiz
  • 📅 Date: December 14, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [offline evaluation], [modular analysis], [multi-path evaluation], [MobiBench]
  • 📖 TLDR: MobiBench is an offline mobile-agent benchmark that explicitly supports multiple valid action paths and evaluates agent modules separately rather than treating the system as a black box. The paper reports 94.72% agreement with human evaluators while preserving the scalability and reproducibility advantages of offline evaluation.

• WebOperator: Action-Aware Tree Search for Autonomous Agents in Web Environment

  • Mahir Labib Dihan, Tanzima Hashem, Mohammed Eunus Ali, Md Rizwan Parvez
  • 🏛️ Institutions: Bangladesh University of Engineering and Technology (BUET), Monash University, Qatar Computing Research Institute (QCRI)
  • 📅 Date: December 14, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [tree search], [backtracking], [action safety], [WebArena], [WebOperator]
  • 📖 TLDR: WebOperator adds action-aware tree search to web agents, combining best-first exploration with safety-aware action ranking and verified backtracking before replaying prior paths. It also diversifies and filters candidate actions before execution, and reaches a reported 54.6% success rate on WebArena with GPT-4o.

• Using GUI Agent for Electronic Design Automation

  • Chunyi Li, Longfei Li, Zicheng Zhang, Xiaohong Liu, Min Tang, Weisi Lin, Guangtao Zhai
  • 🏛️ Institutions: NTU, Shanghai AI Laboratory, SJTU
  • 📅 Date: December 12, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [EDA], [GUI-EDA], [EDAgent], [industrial CAD]
  • 📖 TLDR: This paper presents GUI-EDA, the first large-scale benchmark for deploying GUI agents in Electronic Design Automation workflows across 5 CAD tools and 5 physical domains, and proposes EDAgent, an EDA-specialized GUI agent with a reflection mechanism that outperforms PhD students in Electrical Engineering on industrial CAD software tasks.

• AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Management

  • Shizuo Tian, Hao Wen, Yuxuan Chen, Jiacheng Liu, Shanhui Zhao, Guohong Liu, Ju Ren, Yunxin Liu, Yuanchun Li
  • 🏛️ Institutions: Tsinghua, PKU
  • 📅 Date: December 11, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [context management], [belief state], [Semantic Task Program], [AW-Extend], [AgentProg]
  • 📖 TLDR: AgentProg targets the context-overload problem in long-horizon mobile GUI tasks, where flat interaction histories become expensive and lose task-critical state. It reframes execution history as a Semantic Task Program with variables and control flow, adds a global belief state for partial observability, and reports stronger long-horizon performance on AndroidWorld and AW-Extend than context-compression baselines.

• GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection

  • Zishu Wei, Qixiang Ma, Xavier Hu, Yuhang Liu, Hui Zang, Yudong Zhao, Tao Wang, Shengyu Zhang, Fei Wu
  • 🏛️ Institutions: ZJU, Huawei Technologies Ltd.
  • 📅 Date: December 10, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [information-joint reasoning], [group reflection], [multi-model collaboration], [decision making], [GAIR]
  • 📖 TLDR: GAIR builds a GUI automation system by combining several GUI-specific MLLMs with a general-purpose decision model that jointly reasons over their observations. When evidence is insufficient, its group-reflection stage sends targeted follow-up instructions back to the specialist models, and the framework reports gains on web, desktop, and mobile GUI benchmarks such as UI-I2E-Bench and ScreenSpot.

• MVP: Multiple View Prediction Improves GUI Grounding

  • Yunzhu Zhang, Zeyu Pan, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Linchao Zhu
  • 🏛️ Institutions: ZJU, Hangzhou Dianzi University, Ant Group
  • 📅 Date: December 09, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [training-free], [multi-view inference], [prediction instability], [coordinate clustering], [MVP]
  • 📖 TLDR: MVP studies prediction instability in GUI grounding, where tiny visual perturbations can flip coordinate predictions between correct and incorrect. It improves training-free grounding by proposing attention-guided cropped views and clustering the resulting coordinate predictions, yielding consistent gains on ScreenSpot-Pro, UI-Vision, and OS-World-G.

• Privacy Practices of Browser Agents

  • Alisha Ukani, Hamed Haddadi, Ali Shahin Shamsabadi, Peter Snyder
  • 🏛️ Institutions: UC San Diego, Brave Software, Imperial College London
  • 📅 Date: December 08, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [privacy], [privacy measurement], [cross-site tracking], [personal information leakage], [security evaluation]
  • 📖 TLDR: This paper systematically audits the privacy behavior of eight browser agents across 15 measurements covering browser configuration, website protections, cross-site tracking, privacy-relevant prompts, and personal-data leakage. It identifies 30 vulnerabilities, including disabled browser privacy features and unintended disclosure of sensitive user information in web forms.

• Permission Manifests for Web Agents

  • Samuele Marro, Alan Chan, Xinxing Ren, Lewis Hammond, Jesse Wright, Gurjyot Wanga, Tiziano Piccardi, Nuno Campos, Tobin South, Jialin Yu, Sunando Sengupta, Eric Sommerlade, Alex Pentland, Philip Torr, Jiaxin Pei
  • 🏛️ Institutions: Oxford, Institute for Decentralized AI, Centre for the Governance of AI, Coral Protocol, Cooperative AI Foundation, Webair, JHU, Witan Labs, Anthropic, Stanford, UT Austin
  • 📅 Date: December 07, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [safety], [agent-permissions.json], [permission manifest], [robots.txt-style governance], [web governance]
  • 📖 TLDR: This paper proposes agent-permissions.json, a lightweight robots.txt-style manifest for web agents that lets site owners specify which resources and interactions are allowed, optionally pointing agents to preferred APIs. The goal is to replace blanket blocking with a lower-friction compliance mechanism for browser-based agent traffic.

• Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding

  • Zhiyuan Jiang, Shenghao Xie, Wenyi Li, Wenqiang Zu, Peihang Li, Jiahao Qiu, Siqi Pei, Lei Ma, Tiejun Huang, Mengdi Wang, Shilong Liu
  • 🏛️ Institutions: Xi’an Jiaotong University, Princeton, PKU, University of Chinese Academy of Sciences, HKU, Michigan State University
  • 📅 Date: December 05, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [training-free], [zoom], [test-time scaling], [ZoomClick], [GUIZoom-Bench]
  • 📖 TLDR: This paper studies zooming as a test-time prior for GUI grounding and proposes ZoomClick, which decides when to zoom, how far to zoom, and when to return to the original view during localization. It also introduces GUIZoom-Bench and reports stronger grounding results across several mainstream benchmarks.

• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning

  • Haolong Yan, Yeqing Shen, Xin Huang, Jia Wang, Kaijun Tan, Zhixuan Liang, Hongxin Li, Zheng Ge, Osamu Yoshie, Si Li, Xiangyu Zhang, Daxin Jiang
  • 🏛️ Institutions: Beijing University of Posts and Telecommunications, StepFun, Waseda University, Institute of Automation, CAS
  • 📅 Date: December 02, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [screen navigation], [simulation environment], [reinforcement learning], [exploration strategy], [GUI Exploration Lab]
  • 📖 TLDR: GUI Exploration Lab is a simulation engine for studying screen navigation, exposing full screen and navigation-graph structure so agents can be trained and evaluated without proprietary GUI environments. The paper compares supervised fine-tuning, single-turn RL, and multi-turn RL, and finds that multi-turn RL is what most clearly induces exploratory navigation behavior.

• DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks

  • Hyunjun Kim, Sooyoung Ryu
  • 🏛️ Institutions: Independent
  • 📅 Date: December 01, 2025
  • 📑 Publisher: AAAI 2026 TrustAgent Workshop
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [spatial reasoning], [mouse-based drawing], [verifiable evaluation], [multi-turn feedback], [DrawingBench]
  • 📖 TLDR: DrawingBench evaluates agentic models through mouse-based drawing tasks that require issuing low-level GUI actions on a canvas UI rather than answering static spatial questions. It provides 250 prompts, deterministic rule-based scoring, and multi-turn external feedback, showing both strong baseline performance and clear failure modes in tool-state management and long-horizon control.

• HiconAgent: History Context-aware Policy Optimization for GUI Agents

  • Xurui Zhou, Gongwei Chen, Yuquan Xie, Zaijing Li, Kaiwen Zhou, Shuai Wang, Shuo Yang, Zhuotao Tian, Rui Shao
  • 🏛️ Institutions: HIT-Shenzhen, Huawei Noah's Ark Lab
  • 📅 Date: December 01, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [reinforcement learning], [history usage], [Dynamic Context Sampling], [Anchor-guided History Compression], [HiconAgent]
  • 📖 TLDR: HiconAgent studies how GUI agents should use historical context during sequential navigation instead of always keeping a fixed full history. Its HCPO training recipe combines Dynamic Context Sampling with Anchor-guided History Compression, and the resulting 3B model improves GUI-Odyssey while matching or approaching larger baselines on AndroidControl and AITW at lower compute cost.

• AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent

  • Neeraj Anand, Rishabh Jain, Sohan Patnaik, Balaji Krishnamurthy, Mausoom Sarkar
  • 🏛️ Institutions: Adobe
  • 📅 Date: November 30, 2025
  • 📑 Publisher: WACV 2026
  • 💻 Env: [Mobile]
  • 🔑 Key: [model], [GUI grounding], [smartphone automation], [adaptive feature renormalization], [instruct-BLIP], [AFRAgent]
  • 📖 TLDR: AFRAgent targets the loss of spatial detail that hurts mobile GUI automation models built on low-resolution vision features. It adds an adaptive feature renormalization module to enrich instruct-BLIP image embeddings with high-resolution information, and reports state-of-the-art results on Meta-GUI and AITW with a much smaller model than prior baselines.

• MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents

  • Ruihan Chen, Qiming Li, Xiaocheng Feng, Xiaoliang Yang, Weihong Zhong, Yuxuan Gu, Zekun Zhou, Bing Qin
  • 🏛️ Institutions: Harbin Institute of Technology, Pengcheng Laboratory
  • 📅 Date: November 30, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [multilingual], [cross-lingual], [perception and reasoning], [MPR-GUI-Bench], [GUI-XLI]
  • 📖 TLDR: MPR-GUI studies the gap between English and non-English GUI understanding by introducing a fine-grained multilingual benchmark for GUI perception and reasoning. It also proposes GUI-XLI, a hidden-state intervention method for cross-lingual transfer, and reports average multilingual gains of 6.5%.

• LegalWebAgent: Empowering Access to Justice via LLM-Based Web Agents

  • Jinzhe Tan, Karim Benyekhlef
  • 🏛️ Institutions: Université de Montréal
  • 📅 Date: November 28, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [legal domain], [form filling], [schedule booking], [LegalWebAgent]
  • 📖 TLDR: LegalWebAgent applies browser-based web automation to access-to-justice workflows such as legal information search, form completion, and appointment booking on Quebec civil-law sites. The paper introduces a 15-task real-world benchmark around those processes and reports average success rates above 84% across tested models.

• Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation

  • Zehao Deng, Tianjie Ju, Zheng Wu, Zhuosheng Zhang, Gongshen Liu
  • 🏛️ Institutions: Soochow University, SJTU
  • 📅 Date: November 27, 2025
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [reinforcement learning], [long-horizon tasks], [state tracking], [task decomposition], [CES]
  • 📖 TLDR: This paper targets long-horizon GUI automation by training high-level scheduling modules instead of a single end-to-end executor. Its CES framework separates coordination, execution, and state tracking, and uses execution-feedback reinforcement learning to improve planning and task-state management across different low-level executors.

• Prune4Web: DOM Tree Pruning Programming for Web Agent

  • Jiayuan Zhang, Kaiquan Chen, Zhihao Lu, Enshen Zhou, Qian Yu, Jing Zhang
  • 🏛️ Institutions: Beihang University
  • 📅 Date: November 26, 2025
  • 📑 Publisher: AAAI 2026
  • 💻 Env: [Web]
  • 🔑 Key: [DOM pruning], [code generation], [observation simplification], [programmatic filtering], [Prune4Web]
  • 📖 TLDR: Prune4Web tackles oversized web DOMs by having the model generate executable Python scoring programs that prune irrelevant elements before grounding and action selection. This shifts work from raw DOM reading to programmatic filtering and substantially improves grounding accuracy while shrinking candidate sets by 25x to 50x.

• STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization

  • Yuhan Chen, Yuxuan Liu, Long Zhang, Pengzhi Gao, Jian Luan, Wei Liu
  • 🏛️ Institutions: MiLM Plus, Xiaomi, Renmin University of China, Wuhan University
  • 📅 Date: November 17, 2025
  • 📑 Publisher: Findings of ACL 2026
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [reinforcement learning], [sample efficiency], [trajectory optimization], [step-level optimization], [OSWorld], [AndroidWorld], [STEP]
  • 📖 TLDR: STEP improves the efficiency of online GUI-agent reinforcement learning by using per-task success rates to resample difficult tasks and decomposing trajectories into step-level training samples. Its success-rate-weighted advantages and step-level GRPO augmentation improve sample efficiency and training stability over trajectory-level GRPO on OSWorld and AndroidWorld.

• Adapting Web Agents with Synthetic Supervision

  • Zhaoyang Wang, Yiming Liang, Xuchao Zhang, Qianhui Wu, Siwei Han, Anson Bastos, Rujia Wang, Chetan Bansal, Baolin Peng, Jianfeng Gao, Saravan Rajmohan, Huaxiu Yao
  • 🏛️ Institutions: UNC, Purdue University, Microsoft
  • 📅 Date: November 08, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [synthetic data], [training data], [trajectory refinement], [task synthesis], [SynthAgent]
  • 📖 TLDR: SynthAgent adapts web agents to new sites by synthesizing site-specific tasks and demonstrations, then refining both the tasks and collected trajectories to reduce hallucinations and execution noise. The paper argues that this dual refinement is what makes synthetic supervision effective for website adaptation.

• Beyond Clicking: A Step Towards Generalist GUI Grounding via Text Dragging

  • Zeyi Liao, Yadong Lu, Boyu Gou, Huan Sun, Ahmed Awadallah
  • 🏛️ Institutions: OSU, MSR, Redmond
  • 📅 Date: November 07, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [dataset], [benchmark], [text dragging], [GUI-Drag], [ScreenDrag]
  • 📖 TLDR: This paper expands GUI grounding beyond click actions by focusing on text dragging, a common but previously underexplored mouse interaction. It introduces the GUI-Drag training set and the ScreenDrag benchmark, and shows that continual training for dragging can improve drag performance without sacrificing click grounding.

• GUI-360: A Comprehensive Dataset and Benchmark for Computer-Using Agents

  • Jian Mu, Chaoyun Zhang, Chiming Ni, Lu Wang, Bo Qiao, Kartik Mathur, Qianhui Wu, Yuhang Xie, Xiaojun Ma, Mengyu Zhou, Si Qin, Liqun Li, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
  • 🏛️ Institutions: Microsoft, NJU, ZJU-UIUC, PKU
  • 📅 Date: November 06, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [dataset], [benchmark], [Windows], [accessibility metadata], [reasoning supervision], [GUI-360]
  • 📖 TLDR: GUI-360 addresses the lack of large real-world CUA data and unified evaluation by releasing 1.2M+ executed action steps across thousands of trajectories in popular Windows office applications, including full-resolution screenshots, accessibility metadata, intermediate reasoning, and both successful and failed trajectories. It is the first corpus to jointly cover GUI grounding, screen parsing, action prediction, and API-level actions, exposing cascading failures of off-the-shelf VLMs on heterogeneous layouts.

• GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

  • Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil, Wanrong Zhu, Changyou Chen, Ruiyi Zhang
  • 🏛️ Institutions: State University of New York at Buffalo, Adobe Research
  • 📅 Date: November 02, 2025
  • 📑 Publisher: ECCV 2026
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI-AIMA], [model], [GUI grounding], [attention], [coordinate-free], [ScreenSpot-Pro], [OSWorld-G]
  • 📖 TLDR: GUI-AIMA is an attention-based, coordinate-free supervised fine-tuning framework for GUI grounding that aligns the intrinsic multimodal attention of an MLLM with patch-wise grounding signals and supports a plug-and-play zoom-in stage. GUI-AIMA-3B, trained on about 101k screenshots, reaches 61.5% on ScreenSpot-Pro and 68.1% on OSWorld-G.

• WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation

  • Jiali Cheng, Anjishnu Kumar, Roshan Lal, Rishi Rajasekaran, Hani Ramezani, Omar Zia Khan, Oleg Rokhlenko, Sunny Chiu-Webster, Gang Hua, Hadi Amiri
  • 🏛️ Institutions: University of Massachusetts Lowell, Amazon Alexa AI
  • 📅 Date: October 26, 2025
  • 📑 Publisher: NeurIPS 2025 Workshop on Language Agents and World Models
  • 💻 Env: [Web]
  • 🔑 Key: [training-free], [framework], [memory], [planning], [action simulation], [WebArena-Lite], [WebATLAS]
  • 📖 TLDR: WebATLAS is a training-free web agent that reuses past interaction outcomes as persistent experience memory and simulates candidate actions before executing them. Its planner-simulator-critic loop is designed to improve long-horizon behavior on unseen websites, and it reports 63% success on WebArena-Lite without site-specific fine-tuning.

• Surfer 2: The Next Generation of Cross-Platform Computer Use Agents

  • Mathieu Andreux, Märt Bakler, Yanael Barbier, Hamza Benchekroun, Emilien Biré, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Aleix Cambray, Pierre-Louis Cedoz, Antoine Chassang, Gautier Cloix, Ethan Connelly, Alexandra Constantinou, Ramzi De Coster, Hubert de la Jonquiere, Aurélien Delfosse, Maxime Delpit, Alexis Deprez, Augustin Derupti, Mathieu Diaz, Shannon D'Souza, Julie Dujardin, Abai Edmund, Michael Eickenberg, Armand Fatalot, Wissem Felissi, Isaac Herring, Xavier Koegler, Erwan Le Jumeau de Kergaradec, Aurélien Lac, Maxime Langevin, Corentin Lauverjat, Antonio Loison, Avshalom Manevich, Axel Moyal, Axel Nguyen Kerbel, Marinela Parovic, Julien Revelle, Guillaume Richard, Mats Richter, Ronan Riochet, María Santos, Romain Savidan, Laurent Sifre, Maxime Theillard, Marc Thibault, Ivan Valentini, Tony Wu, Laura Yie, Kai Yuan, Jevgenij Zubovskij
  • 🏛️ Institutions: H Company
  • 📅 Date: October 22, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [framework], [cross-platform], [hierarchical context management], [self-verification], [adaptive recovery], [Surfer 2]
  • 📖 TLDR: Surfer 2 is a visual-only cross-platform computer-use agent designed to work across web, desktop, and mobile without task-specific fine-tuning. It combines hierarchical context management, decoupled planning and execution, and self-verification with adaptive recovery, and reports state-of-the-art results on WebVoyager, WebArena, OSWorld, and AndroidWorld.

• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos

  • Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, Yuchen Mao, Jingren Zhou, Junyang Lin, Binyuan Hui, Tao Yu
  • 🏛️ Institutions: Google Cloud AI Research, OSU
  • 📅 Date: October 22, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [dataset], [pretraining], [video mining], [inverse dynamics], [Video2Action], [VideoAgentTrek]
  • 📖 TLDR: VideoAgentTrek studies how to pretrain computer-use agents from passive screen recordings instead of manually labeled trajectories. Its Video2Action pipeline recovers action boundaries and structured parameters from 39,000 tutorial videos, yielding 1.52 million steps that improve both OSWorld-Verified and AgentNetBench after continued pretraining.

• WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation

  • Yaoyao Qian, Yuanli Wang, Jinda Zhang, Yun Zong, Meixu Chen, Hanhan Zhou, Jindan Huang, Yifan Zeng, Xinyu Hu, Chan Hee Song, Danqing Zhang
  • 🏛️ Institutions: Northeastern University, Boston University, University of Victoria, University of Minnesota, George Washington University, Tufts University, Oregon State University, University of Texas at San Antonio, OSU, PathOnAI.org
  • 📅 Date: October 22, 2025
  • 📑 Publisher: NeurIPS 2025 Workshop on Multi-Turn Interactions in Large Language Models
  • 💻 Env: [Web]
  • 🔑 Key: [evaluation], [trajectory analysis], [weighted action graph], [multi-path evaluation], [WebGraphEval]
  • 📖 TLDR: WebGraphEval evaluates web agents by converting many interaction trajectories into a unified weighted action graph instead of scoring only final success or conformity to one reference path. This graph view highlights redundancy, inefficiency, and critical decision points across agents and benchmark runs.

• CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent

  • Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, Ke Li, Xing Sun
  • 🏛️ Institutions: Tencent Youtu Lab, PKU, NJU
  • 📅 Date: October 21, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [reward model], [outcome reward model], [process reward model], [expert annotation], [CUARewardBench], [UPE]
  • 📖 TLDR: CUARewardBench benchmarks both outcome and process reward models for desktop computer-use evaluation using expert-annotated trajectories from 10 software categories and 7 agent architectures. It shows that current reward models are still unreliable and introduces Unanimous Prompt Ensemble (UPE) to improve reward-model precision.

• Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming

  • Zheng Zhang, Jiarui He, Yuchen Cai, Deheng Ye, Peilin Zhao, Ruili Feng, Hao Wang
  • 🏛️ Institutions: HKUST(GZ), Tencent, SJTU, Alibaba Group
  • 📅 Date: October 21, 2025
  • 📑 Publisher: ICME 2026
  • 💻 Env: [Web]
  • 🔑 Key: [safety], [attack], [prompt injection], [genetic algorithm], [red-teaming], [Genesis]
  • 📖 TLDR: Genesis studies automated red-teaming for web agents by evolving attack strategies over repeated interactions instead of relying on fixed prompts or manually designed attacks. Its attacker-scorer-strategist loop builds and reuses a growing strategy library, yielding stronger adversarial injections across web tasks.

• Investigating the Impact of Dark Patterns on LLM-Based Web Agents

  • Devin Ersoy, Brandon Lee, Ananth Shreekumar, Arjun Arunasalam, Muhammad Ibrahim, Antonio Bianchi, Z. Berkay Celik
  • 🏛️ Institutions: Purdue University, Florida International University, Georgia Tech
  • 📅 Date: October 20, 2025
  • 📑 Publisher: IEEE S&P 2026
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [safety], [dark patterns], [TrickyArena], [LiteAgent]
  • 📖 TLDR: This paper studies how deceptive interface designs affect web agents, introducing LiteAgent for controlled execution and TrickyArena as a dark-pattern test environment. Across six agent frameworks and three underlying models, it finds that agents are frequently steered by dark patterns and that both visual and HTML-level manipulations can change susceptibility.

• UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action

  • Yuhao Yang, Zhen Yang, Zi-Yi Dou, Anh Nguyen, Keen You, Omar Attia, Andrew Szot, Michael Feng, Ram Ramrakhya, Alexander Toshev, Chao Huang, Yinfei Yang, Zhe Gan
  • 🏛️ Institutions: Apple, HKU
  • 📅 Date: October 20, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [model], [hybrid action], [tool calls], [synthetic tasks], [online reinforcement learning], [UltraCUA]
  • 📖 TLDR: UltraCUA bridges low-level GUI actions and higher-level tool use in one computer-use model instead of forcing every task through clicks, typing, and scrolling alone. Its pipeline combines automated tool extraction, synthetic verifiable tasks, supervised fine-tuning, and online RL, and the resulting hybrid-action models improve both OSWorld performance and transfer to WindowsAgentArena.

• CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs

  • Gucongcong Fan, Chaoyue Niu, Chengfei Lyu, Fan Wu, Guihai Chen
  • 🏛️ Institutions: SJTU, Alibaba Group
  • 📅 Date: October 17, 2025
  • 📑 Publisher: NeurIPS 2025 (Poster)
  • 💻 Env: [Mobile]
  • 🔑 Key: [framework], [privacy], [cloud-local collaboration], [UI exposure reduction], [CORE]
  • 📖 TLDR: CORE studies how to reduce unnecessary screen exposure when mobile agents depend on cloud LLMs for planning and action selection. It partitions the UI into layout-aware blocks and lets local and cloud models collaborate on planning and decision-making so only task-relevant UI subsets are sent to the cloud, substantially reducing exposure while keeping accuracy close to cloud-only systems.

• PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction

  • Simon Yu, Gang Li, Weiyan Shi, Peng Qi
  • 🏛️ Institutions: Northeastern University, Uniphore
  • 📅 Date: October 17, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [framework], [skill learning], [compositional skills], [transfer generalization], [polymorphism], [PolySkill], [continual learning]
  • 📖 TLDR: PolySkill targets the tendency of web-agent skills to overfit one site by separating each skill's abstract goal from its concrete site-specific implementation. This polymorphic abstraction improves skill reuse, cross-site transfer, and continual learning behavior on Mind2Web-style settings.

• WebServ: A Browser-Server Environment for Efficient Training of Reinforcement Learning-based Web Agents at Scale

  • Yuxuan Lu, Jing Huang, Hui Liu, Jiri Gesi, Yan Han, Shihan Fu, Tianqi Zheng, Dakuo Wang
  • 🏛️ Institutions: Northeastern University, Amazon
  • 📅 Date: October 17, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [reinforcement learning], [training environment], [browser-server architecture], [scalability], [WebServ]
  • 📖 TLDR: WebServ is a full-stack environment for large-scale RL training of web agents that combines a compact browser-side interface with efficient isolated server-side state management. It is designed to make parallel rollouts practical, and the paper reports faster launches, lower storage cost, and strong WebArena performance.

• In-Browser LLM-Guided Fuzzing for Real-Time Prompt Injection Testing in Agentic AI Browsers

  • Avihay Cohen
  • 🏛️ Institutions: BrowserTotal
  • 📅 Date: October 15, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [prompt injection], [in-browser fuzzing], [feedback-guided fuzzing], [progressive evasion], [page summarization], [question answering]
  • 📖 TLDR: This paper studies prompt-injection testing for agentic AI browsers with an LLM-guided fuzzing loop that runs inside a real browser and mutates malicious pages using immediate attack feedback. It reports that simple attacks are usually blocked, but adaptive mutations drive failure rates to 58-74% by the tenth iteration, with page summarization and question-answering features showing the highest risk.

• HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities

  • Xiaoxue Ren, Penghao Jiang, Kaixin Li, Zhiyong Huang, Xiaoning Du, Jiaojiao Jiang, Zhenchang Xing, Jiamou Sun, Terry Yue Zhuo
  • 🏛️ Institutions: ZJU, University of New South Wales, NUS, Monash University, CSIRO’s Data61, Australian National University
  • 📅 Date: October 14, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [security], [web vulnerabilities], [CTF], [penetration testing], [HackWorld]
  • 📖 TLDR: HackWorld uses a CTF-style setup over 36 vulnerable web applications spanning 11 frameworks and 7 languages to test whether computer-use agents can discover and exploit realistic web flaws through GUI interaction. Current agents achieve exploitation rates below 12% and often fail at multi-step attack planning and security-tool use.

• R-WoM: Retrieval-augmented World Model For Computer-use Agents

  • Kai Mei, Jiang Guo, Shuaichen Chang, Mingwen Dong, Dongkyu Lee, Xing Niu, Jiarong Jiang
  • 🏛️ Institutions: Rutgers University, AWS Agentic AI
  • 📅 Date: October 13, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [world model], [tutorial retrieval], [future state prediction], [OSWorld], [WebArena], [R-WoM]
  • 📖 TLDR: This paper tests whether LLMs can act as world models for computer-use agents and finds that simulation quality degrades sharply on full-procedure planning even when short-range prediction remains reasonable. R-WoM addresses this by grounding simulated rollouts with retrieved up-to-date tutorials, improving performance on OSWorld and WebArena, especially on longer-horizon tasks.

• SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agents

  • Longjie Guo, Chenjie Yuan, Mingyuan Zhong, Robert Wolfe, Ruican Zhong, Yue Xu, Bingbing Wen, Hua Shen, Lucy Lu Wang, Alexis Hiniker
  • 🏛️ Institutions: University of Washington, Rutgers University, CMU, New York University Shanghai
  • 📅 Date: October 13, 2025
  • 📑 Publisher: IUI 2026
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [dark patterns], [human-agent comparison], [code injections], [SusBench]
  • 📖 TLDR: SusBench injects nine dark pattern types into 55 real consumer websites and evaluates five computer-use agents alongside 29 human participants across 313 tasks. It finds both agents and humans especially susceptible to preselection, trick wording, and hidden information, while other overt dark patterns are easier to resist.

• WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agent

  • Tao Li, Jinlong Hu, Yang Wang, Junfeng Liu, Xuejun Liu
  • 🏛️ Institutions: Nanjing University of Aeronautics and Astronautics, Hong Kong Baptist University, Beihang University, Pengcheng Laboratory
  • 📅 Date: October 13, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [query routing], [variational information bottleneck], [cost efficiency], [WebVoyager], [WebRouter]
  • 📖 TLDR: WebRouter trains a query-specific router for web agents with a cost-aware Variational Information Bottleneck objective that compresses verbose agent prompts before model selection. On five real-world WebVoyager websites, it reduces operational cost by 87.8% versus a GPT-4o baseline with only a 3.8% accuracy drop.

• BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions

  • Tao Yu, Zhengbo Zhang, Zhiheng Lyu, Junhao Gong, Hongzhu Yi, Xinming Wang, Yuxuan Zhou, Jiabing Yang, Ping Nie, Yan Huang, Wenhu Chen
  • 🏛️ Institutions: CAS, University of Waterloo, PKU, Tsinghua, Independent Researcher
  • 📅 Date: October 12, 2025
  • 📑 Publisher: TMLR
  • 💻 Env: [Web]
  • 🔑 Key: [human-inspired actions], [Playwright], [rejection fine-tuning], [multi-hop QA], [BrowserAgent]
  • 📖 TLDR: BrowserAgent is a browser-native web agent that uses Playwright actions such as scrolling, clicking, typing, and tab management instead of converting pages into static text summaries. With a two-stage SFT plus rejection fine-tuning pipeline and an explicit memory mechanism, BrowserAgent-7B improves multi-hop QA performance by about 20% over Search-R1.

• SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents

  • Zonghao Ying, Yangguang Shao, Jianle Gan, Gan Xu, Junjie Shen, Wenxin Zhang, Quanchen Zou, Junzheng Shi, Zhenfei Yin, Mingchuan Zhang, Aishan Liu, Xianglong Liu
  • 🏛️ Institutions: Beihang University, Institute of Information Engineering, CAS, China University of Petroleum (East China), Zhejiang University of Technology, University of Chinese Academy of Sciences, 360 AI Security Lab, University of Sydney, Henan University of Science and Technology, Zhongguancun Laboratory, Institute of Dataspace
  • 📅 Date: October 11, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [security evaluation], [attack taxonomy], [failure analysis], [adversarial manipulation], [SecureWebArena]
  • 📖 TLDR: SecureWebArena evaluates the security of LVLM-based web agents with six realistic simulated environments, 2,970 trajectories, and six attack vectors spanning both user-level and environment-level manipulations. Its multi-layered protocol separates failures in reasoning, behavior, and task outcome, and shows that all tested agents remain vulnerable to subtle adversarial attacks.

• WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions

  • Sanjari Srivastava, Gang Li, Cheng Chang, Rishu Garg, Manpreet Kaur, Charlene Y. Lee, Yuezhang Li, Yining Mao, Ignacio Cases, Yanan Xie, Peng Qi
  • 🏛️ Institutions: Uniphore
  • 📅 Date: October 10, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [GUI subtasks], [web archive], [RLVR], [WARC-Bench]
  • 📖 TLDR: WARC-Bench uses Web ARChive files to create sandboxed interactive webpages for evaluating short-horizon GUI subtasks such as date picking and container scrolling. Across 438 benchmark tasks, leading computer-use models still struggle, and RL with verifiable rewards improves open models beyond supervised fine-tuning alone.

• Code Agent can be an End-to-end System Hacker: Benchmarking Real-world Threats of Computer-use Agent

  • Weidi Luo, Qiming Zhang, Tianyu Lu, Xiaogeng Liu, Bin Hu, Hung-Chun Chiu, Siyuan Ma, Yizhe Zhang, Xusheng Xiao, Yinzhi Cao, Zhen Xiang, Chaowei Xiao
  • 🏛️ Institutions: University of Georgia, University of Wisconsin-Madison, JHU, UMD, HKUST, CUHK, Apple, Arizona State University
  • 📅 Date: October 08, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [security benchmark], [MITRE ATT&CK], [kill chain], [multi-host sandbox], [hard-coded evaluation], [AdvCUA]
  • 📖 TLDR: This paper introduces AdvCUA, a security benchmark for desktop computer-use agents aligned with MITRE ATT&CK Enterprise tactics, techniques, and procedures. It evaluates 140 malicious tasks in a realistic multi-host sandbox with encrypted credentials and hard-coded judgment, showing that current CUAs still fail to cover many OS security threats even as they lower the skill needed for complex intrusions.

• From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents

  • Yuan Wang, Mingyu Li, Haibo Chen
  • 🏛️ Institutions: Key Laboratory of System Software, Institute of Software, CAS, University of Chinese Academy of Sciences, SJTU
  • 📅 Date: October 06, 2025
  • 📑 Publisher: EuroSys 2026
  • 💻 Env: [Desktop]
  • 🔑 Key: [declarative interface], [policy-mechanism separation], [accessibility APIs], [Microsoft Office], [DMI]
  • 📖 TLDR: DMI transforms human-oriented GUIs into three declarative primitives, letting LLMs focus on semantic planning while the system handles low-level navigation and interaction through accessibility interfaces. On Microsoft Office tasks in Windows, it improves success rate by 67%, reduces interaction steps by 43.5%, and completes over 61% of successful tasks with a single LLM call.

• Watch and Learn: Learning to Use Computers from Online Videos

  • Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, Tomas Pfister
  • 🏛️ Institutions: Google Cloud AI Research, OSU
  • 📅 Date: October 06, 2025
  • 📑 Publisher: CVPR 2026
  • 💻 Env: [Desktop]
  • 🔑 Key: [dataset], [video demonstrations], [inverse dynamics], [trajectory annotation], [OSWorld], [WindowsAgentArena], [Watch & Learn]
  • 📖 TLDR: Watch & Learn converts Internet videos of human computer use into more than 53K executable UI trajectories by framing annotation as an inverse dynamics problem over consecutive screen states. The resulting data improves both general-purpose and specialized CUAs on OSWorld and yields state-of-the-art 7B-scale performance on WindowsAgentArena under the 15-step limit.

• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding

  • Bin Lei, Nuo Xu, Ali Payani, Mingyi Hong, Chunhua Liao, Yu Cao, Caiwen Ding
  • 🏛️ Institutions: University of Minnesota, Cisco Research, Lawrence Livermore National Labs
  • 📅 Date: October 05, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [image-grounded reasoning], [iterative focus refinement], [ScreenSpot-pro], [GUI-Spotlight]
  • 📖 TLDR: GUI-Spotlight is a GUI grounding model that performs image-grounded reasoning by iteratively invoking specialized tools to narrow attention to the relevant screen region. Trained with only 18.5K examples, it reaches 52.8% accuracy on ScreenSpot-Pro, outperforming prior 7B grounding models trained on much larger datasets.

• JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation

  • Hadi Nekoei, Aman Jaiswal, Patrice Bechard, Oleh Shliazhko, Orlando Marquez Ayala, Mathieu Reymond, Massimo Caccia, Alexandre Drouin, Sarath Chandar, Alexandre Lacoste
  • 🏛️ Institutions: ServiceNow Research, Mila, Université de Montréal, Dalhousie University, Polytechnique Montréal, Canada CIFAR AI Chair
  • 📅 Date: October 05, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [offline trajectories], [hint retrieval], [failed trajectories], [WebArena-Lite], [JEF-Hinter]
  • 📖 TLDR: JEF-Hinter distills offline web-agent traces into compact, context-aware hints that can be retrieved at inference time instead of relying on online exploration or large demonstration sets. It uses both successful and failed trajectories, highlights decisive steps with a zooming mechanism, and outperforms strong human- and document-hint baselines on MiniWoB++, WorkArena-L1, and WebArena-Lite.

• Cross-Modal Content Optimization for Steering Web Agent Preferences

  • Tanqiu Jiang, Min Bai, Nikolaos Pappas, Yanjun Qi, Sandesh Swamy
  • 🏛️ Institutions: Stony Brook University, AWS AI Labs
  • 📅 Date: October 04, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [preference steering], [black-box attack], [multimodal attack], [stealth], [CPS]
  • 📖 TLDR: This paper introduces Cross-Modal Preference Steering (CPS), a black-box attack that jointly perturbs an item's image and text to bias web-agent ranking and selection decisions. Under a realistic threat model where the attacker controls only their own listing metadata, CPS outperforms prior baselines across GPT-4.1, Qwen-2.5VL, and Pixtral-Large while keeping detection rates much lower.

• FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents

  • Imene Kerboua, Sahar Omidi Shayegan, Megh Thakkar, Xing Han Lù, Léo Boisvert, Massimo Caccia, Jérémy Espinas, Alexandre Aussem, Véronique Eglin, Alexandre Lacoste
  • 🏛️ Institutions: LIRIS - CNRS, INSA Lyon, Universite Claude Bernard Lyon 1, Esker, ServiceNow Research, Mila, McGill University, Polytechnique Montréal
  • 📅 Date: October 03, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [context pruning], [accessibility tree], [LLM retriever], [prompt injection defense], [FocusAgent]
  • 📖 TLDR: FocusAgent trims long web-agent observations by using a lightweight LLM retriever to keep only task-relevant lines from the accessibility tree. It cuts observation size by more than 50% while matching strong baselines on WorkArena and WebArena, and its defense variant reduces banner and pop-up prompt-injection success without hurting clean-task performance.

• BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks

  • Sagnik Anupam, Davis Brown, Shuo Li, Eric Wong, Hamed Hassani, Osbert Bastani
  • 🏛️ Institutions: University of Pennsylvania
  • 📅 Date: October 02, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [live web evaluation], [head-to-head ranking], [step-level human feedback], [failure modes], [BrowserArena]
  • 📖 TLDR: BrowserArena is a live open-web evaluation platform that compares web agents on user-submitted tasks with Arena-style head-to-head judgments and step-level human annotations. It surfaces recurring real-world failure modes such as captcha resolution, pop-up removal, and direct URL navigation, and uses targeted datasets to study how different models handle them.

• Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness

  • Erfan Shayegani, Keegan Hines, Yue Dong, Nael Abu-Ghazaleh, Roman Lutz, Spencer Whitehead, Vidhisha Balachandran, Besmira Nushi, Vibhav Vineet
  • 🏛️ Institutions: Microsoft Research AI Frontiers, Microsoft AI Red Team, University of California, Riverside, NVIDIA
  • 📅 Date: October 02, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [blind goal-directedness], [safety benchmark], [OSWorld], [thought-action disconnect], [BLIND-ACT]
  • 📖 TLDR: This paper identifies Blind Goal-Directedness (BGD), where computer-use agents continue pursuing goals despite feasibility, safety, reliability, or context concerns. It introduces BLIND-ACT, a 90-task benchmark on OSWorld, and finds high average BGD rates across frontier models even after prompting-based mitigations.

• Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents

  • Lingzhong Dong, Ziqi Zhou, Shuaibo Yang, Haiyue Sheng, Pengzhou Cheng, Zongru Wu, Zheng Wu, Gongshen Liu, Zhuosheng Zhang
  • 🏛️ Institutions: SJTU, Beijing Institute of Technology
  • 📅 Date: October 02, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reasoning-execution gap], [ground-truth alignment], [execution gap], [reasoning gap], [mobile evaluation], [GTA]
  • 📖 TLDR: This paper introduces Ground-Truth Alignment (GTA), a metric for checking whether a mobile agent's chain-of-thought implies the ground-truth action rather than only whether the final action is correct. Combined with Exact Match, the framework separates execution gaps from reasoning gaps and shows that execution gaps are common across mobile benchmarks even for larger VLM agents.

• Scaling Agents for Computer Use

  • Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, Xin Eric Wang
  • 🏛️ Institutions: Simular Research
  • 📅 Date: October 02, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [test-time scaling], [multiple rollouts], [behavior narratives], [behavior judge], [OSWorld], [BJudge]
  • 📖 TLDR: This paper argues that computer-use agents scale more effectively across multiple rollouts than within a single rollout, and introduces Behavior Judge (BJudge) to compare candidate trajectories via compact behavior narratives. BJudge reaches 72.6% on OSWorld, slightly surpassing reported human performance, and also generalizes to WindowsAgentArena and AndroidWorld.

• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness

  • Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai, Caiming Xiong, Chien-Sheng Wu
  • 🏛️ Institutions: Salesforce AI Research, UCLA
  • 📅 Date: October 01, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [training-free], [KV cache compression], [spatio-temporal redundancy], [spatial saliency], [AgentNetBench], [GUI-KV]
  • 📖 TLDR: GUI-KV is a training-free KV-cache compression method for GUI agents that exploits two GUI-specific signals: spatial saliency within a frame and temporal redundancy across frames. It closely matches or beats full-cache performance on standard benchmarks, and in a 5-screenshot AgentNetBench setting cuts decoding FLOPs by 38.9% while improving step accuracy.

• PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents

  • Zikang Liu, Junyi Li, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-rong Wen
  • 🏛️ Institutions: Renmin University of China, NUS, Alibaba Group
  • 📅 Date: October 01, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile], [Web]
  • 🔑 Key: [active look-back], [memory retrieval], [screenshot retrieval], [mobile navigation], [PAL-UI]
  • 📖 TLDR: PAL-UI equips vision-based GUI agents with active look-back instead of relying only on truncated history or coarse textual summaries. It combines dual-level summaries with a retrieval tool for recalling specific past screenshots during planning, improving long-horizon mobile navigation and transferring to web navigation without additional training.

• WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents

  • Yinuo Liu, Ruohan Xu, Xilong Wang, Yuqi Jia, Neil Zhenqiang Gong
  • 🏛️ Institutions: Duke University
  • 📅 Date: October 01, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [prompt injection detection], [threat model taxonomy], [text detectors], [image detectors], [WAInjectBench]
  • 📖 TLDR: WAInjectBench is a benchmark study of prompt-injection detection for web agents that organizes attacks by threat model and evaluates both text-based and image-based detectors on malicious and benign samples. It finds that detectors can handle explicit textual attacks or visible image perturbations reasonably well, but largely fail on attacks without explicit instructions or with imperceptible perturbations.

• WALT: Web Agents that Learn Tools

  • Viraj Prabhu, Yutong Dai, Matthew Fernandez, Jing Gu, Krithika Ramakrishnan, Yanqi Luo, Silvio Savarese, Caiming Xiong, Junnan Li, Zeyuan Chen, Ran Xu
  • 🏛️ Institutions: Salesforce AI Research
  • 📅 Date: October 01, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Web]
  • 🔑 Key: [tool discovery], [latent website functionality], [browser automation], [VisualWebArena], [WebArena], [WALT]
  • 📖 TLDR: WALT reframes web automation around reusable tools already implicit in websites, such as search, filter, sort, posting, and content management, instead of relying on brittle low-level UI actions. By reverse-engineering these latent tools, it improves success on WebArena and VisualWebArena while using fewer steps and less LLM-heavy reasoning.

• Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

  • Zhen Yang, Zi-Yi Dou, Di Feng, Forrest Huang, Anh Nguyen, Keen You, Omar Attia, Yuhao Yang, Michael Feng, Haotian Zhang, Ram Ramrakhya, Chao Jia, Jeffrey Nichols, Alexander Toshev, Yinfei Yang, Zhe Gan
  • 🏛️ Institutions: Apple
  • 📅 Date: September 30, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [on-device agent], [3B model], [visual tool-use], [ScreenSpot-V2], [OSWorld-G], [Ferret-UI Lite]
  • 📖 TLDR: Ferret-UI Lite is a compact 3B end-to-end GUI agent for on-device use across mobile, web, and desktop environments. Built from a mixed real-and-synthetic data recipe plus chain-of-thought, visual tool-use, and reward-designed RL, it reaches competitive grounding scores and navigation success despite the constraints of small on-device models.

• SCUBA: Salesforce Computer Use Benchmark

  • Yutong Dai, Krithika Ramakrishnan, Jing Gu, Matthew Fernandez, Yanqi Luo, Viraj Prabhu, Zhenyu Hu, Silvio Savarese, Caiming Xiong, Zeyuan Chen, Ran Xu
  • 🏛️ Institutions: Salesforce AI Research
  • 📅 Date: September 30, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [benchmark], [CRM workflows], [Salesforce sandbox], [milestone evaluation], [SCUBA]
  • 📖 TLDR: SCUBA is a benchmark for computer-use agents on Salesforce customer-relationship-management workflows, with 300 task instances derived from real user interviews across administrator, sales, and service personas. It runs in Salesforce sandbox environments with interpretable milestone evaluation and shows that enterprise tasks remain much harder than standard CUA benchmarks, especially for open models in zero-shot settings.

• Scaling Synthetic Task Generation for Agents via Exploration

  • Ram Ramrakhya, Andrew Szot, Omar Attia, Yuhao Yang, Anh Nguyen, Bogdan Mazoure, Zhe Gan, Harsh Agrawal, Alexander Toshev
  • 🏛️ Institutions: Apple
  • 📅 Date: September 29, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [dataset], [task generation], [environment exploration], [synthetic tasks], [AutoPlay], [Android apps], [Ubuntu apps]
  • 📖 TLDR: AutoPlay is a scalable task-generation pipeline that first explores interactive environments to uncover functionalities and then synthesizes diverse, executable, verifiable tasks grounded in those states. It generates 20k Android tasks and 10k Ubuntu tasks, enabling large-scale post-training and additional RL gains for UI agents without human annotation.

• Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation

  • Pengxiang Li, Zechen Hu, Zirui Shang, Jingrong Wu, Yang Liu, Hui Liu, Zhi Gao, Chenrui Shi, Bofei Zhang, Zihao Zhang, Xiaochuan Shi, Zedong YU, Yuwei Wu, Xinxiao Wu, Yunde Jia, Liuyu Xiang, Zhaofeng He, Qing Li
  • 🏛️ Institutions: Beijing Institute of Technology, State Key Laboratory of General Artificial Intelligence, BIGAI, DataCanvas, Beijing University of Posts and Telecommunications, Shenzhen MSU-BIT University
  • 📅 Date: September 28, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [reinforcement learning], [decoupled training], [adaptive data curation], [asynchronous modules], [OSWorld], [DART]
  • 📖 TLDR: DART is a decoupled RL training framework for GUI agents that separates environment execution, rollout service, data management, and training into asynchronous modules to improve multi-turn learning efficiency. It pairs that system design with adaptive data curation, including difficulty-aware rollout control and high-entropy step selection, and substantially improves OSWorld performance over the base model.

• GUI-PRA: Process Reward Agent for GUI Tasks

  • Tao Xiong, Xavier Hu, Yurun Chen, Yuhang Liu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, Shengyu Zhang
  • 🏛️ Institutions: ZJU, MiLM Plus, Xiaomi
  • 📅 Date: September 27, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reward model], [process reward], [visual verification], [long-horizon tasks], [GUI-PRA]
  • 📖 TLDR: GUI-PRA turns GUI process-reward evaluation from passive scoring into active investigation. It synthesizes state-specific verification criteria from experience and uses them to navigate visual tools and gather grounded evidence, improving success over standard process reward models on AndroidWorld and Mobile-MiniWoB++.

• ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration

  • Gaole Dai, Shiqi Jiang, Ting Cao, Yuqing Yang, Yuanchun Li, Rui Tan, Mo Li, Lili Qiu
  • 🏛️ Institutions: NTU, MSR, Institute for AI Industry Research (AIR), Tsinghua, HKUST
  • 📅 Date: September 26, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [proactive reward], [state probing], [reasoner-actor collaboration], [chain-of-claims], [reward accuracy], [ProRe]
  • 📖 TLDR: ProRe turns GUI-agent reward assignment into an active probing process: a general-purpose reasoner schedules targeted checks and evaluator agents interact with the environment to gather extra evidence before scoring a trajectory. On more than 3,000 trajectories it improves reward accuracy and F1 over static judges, and those rewards also improve downstream policy-agent success rates.

• Secure and Efficient Access Control for Computer-Use Agents via Context Space

  • Haochen Gong, Chenxiao Li, Rui Chang, Wenbo Shen
  • 🏛️ Institutions: ZJU
  • 📅 Date: September 26, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [access control], [static policies], [intent-aware policies], [context-aware policies], [CSAgent]
  • 📖 TLDR: CSAgent is a system-level access-control framework for computer-use agents that combines static policies with intent- and context-aware constraints to limit what actions an agent may execute. It supports API, CLI, and GUI control paths, and the paper reports complete defense coverage on the benchmark with low performance overhead and modest utility loss.

• BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent

  • Shaojie Zhang, Ruoceng Zhang, Pei Fu, Shaokang Wang, Jiahui Yang, Xin Du, Shiqi Cui, Bin Qin, Ying Huang, Zhenbo Luo, Jian Luan
  • 🏛️ Institutions: MiLM Plus, Xiaomi
  • 📅 Date: September 19, 2025
  • 📑 Publisher: NeurIPS 2025 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [blink-think-link], [blink data generation], [BTL Reward], [process-and-outcome reward], [human-GUI interaction], [BTL-UI]
  • 📖 TLDR: BTL-UI models GUI interaction as Blink, Think, and Link phases inspired by human visual attention, planning, and action. It adds a blink-oriented data generation pipeline and a rule-based reward that supervises both process and outcome, and reports competitive results on both static GUI understanding and dynamic interaction benchmarks.

• Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

  • Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, Yu Su
  • 🏛️ Institutions: OSU, Amazon AGI
  • 📅 Date: September 18, 2025
  • 📑 Publisher: NeurIPS 2025 Datasets & Benchmarks Track (Poster)
  • 💻 Env: [Web]
  • 🔑 Key: [agentic search], [agent-as-a-judge], [tree-structured rubric], [source attribution], [human evaluation], [Mind2Web 2]
  • 📖 TLDR: Mind2Web 2 benchmarks long-horizon agentic search with 130 human-crafted tasks that require real-time browsing and citation-backed synthesis. It evaluates systems with task-specific judge agents built from tree-structured rubrics that score both answer correctness and source attribution, and compares ten frontier systems against human performance.

• ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data

  • Zhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, Shenglong Ye, Qingyun Li, Xuan Dong, Yue Yu, Chenyu Lu, YunXiang Mo, Yao Yan, Zeyue Tian, Xiao Zhang, Yuan Huang, Yiqian Liu, Weijie Su, Gen Luo, Xiangyu Yue, Biqing Qi, Kai Chen, Bowen Zhou, Yu Qiao, Qifeng Chen, Wenhai Wang
  • 🏛️ Institutions: Shanghai AI Laboratory
  • 📅 Date: September 18, 2025
  • 📑 Publisher: ICLR 2026 (Oral)
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [model], [cross-platform data], [closed-loop data pipeline], [six operating systems], [grounding mode], [reasoned action], [ScaleCUA]
  • 📖 TLDR: ScaleCUA builds an open computer-use dataset across six operating systems and three GUI task families through a closed-loop pipeline that combines automated agents with human experts. Models trained on this corpus support grounding, direct-action, and reasoned-action inference modes and reach strong cross-platform results on WebArena-Lite-v2, OSWorld-G, and MMBench-GUI.

• Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environments

  • Yitong Zhang, Ximo Li, Liyi Cai, Jia Li
  • 🏛️ Institutions: Tsinghua, Beihang University, PKU
  • 📅 Date: September 14, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [security], [environmental injection attack], [dynamic environments], [LLM-driven environment simulation], [attention black hole], [trigger optimization], [Chameleon]
  • 📖 TLDR: This paper studies environmental injection attacks on GUI agents under dynamic web conditions where trigger position and surrounding context vary across pages and sessions. It shows prior attacks degrade sharply under this more realistic threat model, then proposes Chameleon with LLM-driven environment simulation and Attention Black Hole supervision to recover attack effectiveness across six websites and four LVLM agents.

• MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents

  • Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, Yuxiao Dong
  • 🏛️ Institutions: Tsinghua, Zhipu
  • 📅 Date: September 10, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Mobile]
  • 🔑 Key: [online reinforcement learning], [difficulty-adaptive GRPO], [positive replay], [failure curriculum filtering], [AndroidWorld], [AndroidLab], [MobileRL]
  • 📖 TLDR: MobileRL trains mobile GUI agents with online agentic reinforcement learning built around AdaGRPO, which combines shortest-path reward adjustment, difficulty-adaptive positive replay, and failure curriculum filtering. Applied to open vision-language models, it improves sample efficiency and reaches strong success rates on AndroidWorld and AndroidLab.

• AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents

  • Haitao Hu, Peng Chen, Yanpeng Zhao, Yuqi Chen
  • 🏛️ Institutions: ShanghaiTech University, Independent Researcher
  • 📅 Date: September 09, 2025
  • 📑 Publisher: CCS 2025
  • 💻 Env: [Desktop]
  • 🔑 Key: [security], [real-time defense], [security audit], [system traces], [sensitive operations], [BadComputerUse], [AgentSentinel]
  • 📖 TLDR: AgentSentinel is a real-time defense layer for computer-use agents that intercepts sensitive operations and pauses execution until they are audited against both task context and system traces. The companion BadComputerUse benchmark contains 60 attacks across six categories, and the paper reports that AgentSentinel substantially improves defense success over baseline protections.

• MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents

  • Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, Yuehao Huang, Yaxuan Guo, Kexin Zhang, Hao Wang, Liang Liu, Yong Liu
  • 🏛️ Institutions: ZJU, vivo AI Lab, Huzhou Institute of Zhejiang University
  • 📅 Date: September 08, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [shortcut-augmented agents], [APIs], [deep links], [RPA scripts], [shortcut generation], [MAS-Bench]
  • 📖 TLDR: MAS-Bench benchmarks hybrid mobile agents that can combine ordinary GUI interaction with shortcuts such as APIs, deep links, and RPA scripts. It covers 139 tasks across 11 real-world apps, includes 88 predefined shortcuts and 7 evaluation metrics, and explicitly tests whether agents can discover and construct reusable low-cost workflows on their own.

• Are LLM Agents the New RPA? A Comparative Study with RPA Across Enterprise Workflows

  • Petr Průcha, Michaela Matoušková, Jan Strnad
  • 🏛️ Institutions: Technical University of Liberec, Pointee Inc.
  • 📅 Date: September 04, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [agentic automation with computer use], [robotic process automation], [UiPath], [Anthropic Computer Use], [development effort], [enterprise workflows]
  • 📖 TLDR: This paper compares traditional RPA with agentic automation based on computer-use agents across enterprise workflows for data entry, monitoring, and document extraction. It finds that UiPath remains faster and more reliable in repetitive stable settings, while Anthropic's Computer Use agent is more flexible and much quicker to develop for changing interfaces.

• OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds

  • Longrong Yang, Zhixiong Zeng, Yufeng Zhong, Huang Jing, Liming Zheng, Lei Chen, Haibo Qiu, Zequn Qin, Lin Ma, Xi Li
  • 🏛️ Institutions: ZJU, Meituan Technology
  • 📅 Date: September 02, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [generalist agent], [layer-heterogeneous MoE], [unified action space], [GUI-embodied transfer], [2D and 3D worlds], [OmniActor]
  • 📖 TLDR: OmniActor studies a single agent that can act in both GUI-based 2D environments and embodied 3D settings. It uses a layer-heterogeneous MoE design to share shallow GUI/embodied representations while separating deeper action-specific parameters, and combines that structure with unified action spaces and mixed training data to improve both GUI and embodied performance.

• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning

  • Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Aoyan Li, Bo Li, Chen Dun, Chong Liu, Daoguang Zan, Fuxing Leng, Hanbin Wang, Hao Yu, Haobin Chen, Hongyi Guo, Jing Su, Jingjia Huang, Kai Shen, Kaiyu Shi, Lin Yan, Peiyao Zhao, Pengfei Liu, Qinghao Ye, Renjie Zheng, Shulin Xin, Wayne Xin Zhao, Wen Heng, Wenhao Huang, Wenqian Wang, Xiaobo Qin, Yi Lin, Youbin Wu, Zehui Chen, Zihao Wang, Baoquan Zhong, Xinchun Zhang, Xujing Li, Yuanfan Li, Zhongkai Zhao, Chengquan Jiang, Faming Wu, Haotian Zhou, Jinlin Pang, Li Han, Qi Liu, Qianli Ma, Siyao Liu, Songhua Cai, Wenqi Fu, Xin Liu, Yaohui Wang, Zhi Zhang, Bo Zhou, Guoliang Li, Jiajun Shi, Jiale Yang, Jie Tang, Li Li, Qihua Han, Taoran Lu, Woyu Lin, Xiaokang Tong, Xinyao Li, Yichi Zhang, Yu Miao, Zhengxuan Jiang, Zili Li, Ziyuan Zhao, Chenxin Li, Dehua Ma, Feng Lin, Ge Zhang, Haihua Yang, Hangyu Guo, Hongda Zhu, Jiaheng Liu, Junda Du, Kai Cai, Kuanye Li, Lichen Yuan, Meilan Han, Minchao Wang, Shuyue Guo, Tianhao Cheng, Xiaobo Ma, Xiaojun Xiao, Xiaolong Huang, Xinjie Chen, Yidi Du, Yilin Chen, Yiwen Wang, Zhaojian Li, Zhenzhu Yang, Zhiyuan Zeng, Chaolin Jin, Chen Li, Hao Chen, Haoli Chen, Jian Chen, Qinghao Zhao, Guang Shi
  • 🏛️ Institutions: ByteDance Seed
  • 📅 Date: September 02, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [data flywheel], [multi-turn reinforcement learning], [hybrid GUI-terminal environment], [unified sandbox], [online-Mind2Web], [UI-TARS-2]
  • 📖 TLDR: UI-TARS-2 studies how to scale native GUI agents with a data flywheel, stabilized multi-turn reinforcement learning, hybrid GUI-plus-terminal environments, and a unified sandbox. It reports strong results across GUI benchmarks, games, information-seeking tasks, and software-engineering settings, and also analyzes training dynamics and parameter interpolation during RL.

• Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control

  • Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao
  • 🏛️ Institutions: Huawei Noah's Ark Lab, UCL
  • 📅 Date: September 01, 2025
  • 📑 Publisher: NeurIPS 2025 (Poster)
  • 💻 Env: [Mobile]
  • 🔑 Key: [off-policy reinforcement learning], [positive-sample updates], [negative-sample regularization], [successful transition replay], [AndroidWorld], [SoLS], [STR]
  • 📖 TLDR: SoLS is an off-policy RL algorithm for mobile app control that updates directly on successful samples but applies conservative regularized updates on negative ones to avoid policy degradation in sparse-reward settings. With Successful Transition Replay, it improves AndroidWorld performance substantially while using far less compute than GPT-4o-based baselines.

• Throttling Web Agents Using Reasoning Gates

  • Abhinav Kumar, Jaechul Roh, Ali Naseh, Amir Houmansadr, Eugene Bagdasarian
  • 🏛️ Institutions: UMass Amherst
  • 📅 Date: September 01, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [web agent throttling], [reasoning gates], [rebus puzzles], [computational asymmetry], [MCP servers], [resource access control]
  • 📖 TLDR: This paper proposes web-agent throttling by forcing agents to solve reasoning gates before they can access protected resources. It introduces rebus-based puzzles plus a scalable generation and verification pipeline, shows about 9.2x computational asymmetry on strong models, and evaluates the mechanism on both websites and MCP servers.

• A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants

  • Hans G.W. van Dam
  • 🏛️ Institutions: uxx.ai
  • 📅 Date: August 31, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [MCP], [MVVM], [GUI tree router], [speech-enabled assistants], [voice accessibility]
  • 📖 TLDR: This paper proposes an MCP-driven GUI architecture that lets existing applications expose navigation structure and action semantics to speech-enabled assistants through ViewModels and a GUI tree router. The design targets multimodal interaction with aligned spoken and visual feedback, and the paper also reports a small evaluation of locally deployable open-weight models for this setting.

• Are CAPTCHAs Still Bot-hard? Generalized Visual CAPTCHA Solving with Agentic Vision Language Model

  • Xiwen Teoh, Yun Lin, Siqi Li, Ruofan Liu, Avi Sollomoni, Yaniv Harel, Jin Song Dong
  • 🏛️ Institutions: SJTU, NUS, Tel Aviv University
  • 📅 Date: August 2025
  • 📑 Publisher: USENIX Security 2025
  • 💻 Env: [Web]
  • 🔑 Key: [Halligan], [CAPTCHA], [benchmark], [visual reasoning]
  • 📖 TLDR: Halligan reduces visual CAPTCHA solving to a search problem over a standardized metamodel of frames, elements, and keypoints, letting one agentic vision-language model generalize across CAPTCHA types without per-challenge adaptation. It solves 60.7% of 2,600 challenges spanning 26 visual CAPTCHAs, and 70.6% of previously unseen CAPTCHAs in a 30-day field study, arguing that current visual CAPTCHA defenses are no longer reliably bot-hard.

• CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning

  • Zeyi Sun, Yuhang Cao, Jianze Liang, Qiushi Sun, Ziyu Liu, Zhixiong Zhang, Yuhang Zang, Xiaoyi Dong, Kai Chen, Dahua Lin, Jiaqi Wang
  • 🏛️ Institutions: SJTU, Shanghai AI Laboratory, CUHK, HKU
  • 📅 Date: August 27, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [dual-brain architecture], [decoupled GRPO], [planner-executor coordination], [ScienceBoard], [specialization-to-generalization], [CODA]
  • 📖 TLDR: CODA is a trainable planner-executor composition for specialized computer-use tasks, where a generalist planner is paired with a specialist executor and improved through a two-stage specialization-then-generalization pipeline. On ScienceBoard's scientific software tasks, it uses decoupled GRPO to train application-specific planners and then consolidates successful trajectories into a stronger cross-domain planner.

• Mobile-Agent-v3: Fundamental Agents for GUI Automation

  • Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, Ming Yan
  • 🏛️ Institutions: Tongyi Lab, Alibaba Group
  • 📅 Date: August 21, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile]
  • 🔑 Key: [model], [GUI-Owl], [self-evolving trajectory production], [trajectory correctness judgment], [TRPO], [multi-agent framework], [Mobile-Agent-v3]
  • 📖 TLDR: This paper introduces GUI-Owl as a foundation model for GUI automation and builds Mobile-Agent-v3 as a multi-agent framework on top of it. The work combines cross-OS trajectory production, diverse GUI data synthesis, reasoning enhancement, and trajectory-aware RL, and reports stronger open-source results on both AndroidWorld and OSWorld.

• ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents

  • Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, Jie Tang
  • 🏛️ Institutions: Tsinghua, Zhipu, University of Chinese Academy of Sciences
  • 📅 Date: August 19, 2025
  • 📑 Publisher: ICLR 2026 (Poster)
  • 💻 Env: [Desktop]
  • 🔑 Key: [model], [reinforcement learning], [API-GUI paradigm], [distributed RL infrastructure], [parallel virtual desktops], [Entropulse], [OSWorld], [ComputerRL]
  • 📖 TLDR: ComputerRL is a desktop-agent training framework that combines direct GUI interaction with programmatic APIs and scales online RL through a distributed infrastructure over thousands of parallel virtual desktops. Its Entropulse schedule alternates RL and supervised fine-tuning to stabilize long training runs, and the resulting GLM-ComputerRL-9B reaches 48.9% on OSWorld.

• WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents [Technical Report]

  • Ralph Peeters, Aaron Steiner, Luca Schwarz, Julian Yuya Caspary, Christian Bizer
  • 🏛️ Institutions: Data and Web Science Group, University of Mannheim
  • 📅 Date: August 18, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [comparison shopping], [multi-shop benchmark], [offline web benchmark], [common crawl product offers], [WebMall]
  • 📖 TLDR: WebMall is an offline benchmark for comparison-shopping web agents built from four simulated shops populated with product offers extracted from the Common Crawl. It includes 91 cross-shop tasks spanning exact search, price comparison, vague search, substitute and complementary products, and checkout, and the paper shows that even the best agents remain weak on the harder shopping categories.

• You Don’t Know Until You Click: Automated GUI Testing for Production-Ready Software Evaluation

  • Yutong Bian, Xianhao Lin, Yupeng Xie, Tianyang Liu, Mingchen Zhuge, Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang, Jiaqi Chen, Xiangru Tang, Yongxin Ni, Sirui Hong, Chenglin Wu
  • 🏛️ Institutions: DeepWisdom, Fudan, HKUST(GZ), UC San Diego, KAUST, Westlake University, Stanford, Yale University, NUS
  • 📅 Date: August 17, 2025
  • 📑 Publisher: SEA @ NeurIPS 2025 (Poster)
  • 💻 Env: [General GUI]
  • 🔑 Key: [RealDevWorld], [RealDevBench], [AppEvalPilot], [agent-as-a-judge], [interactive GUI testing], [production-ready software]
  • 📖 TLDR: RealDevWorld is an evaluation framework for repository-scale software generation that judges whether produced applications actually work when interacted with through their GUIs. It pairs a 194-task benchmark, RealDevBench, with AppEvalPilot, an agent-as-a-judge system for functional, visual, and runtime evaluation, and reports strong alignment with expert human assessments.

• MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

  • Shilong Li, Xingyuan Bu, Wenjie Wang, Jiaheng Liu, Jun Dong, Haoyang He, Hao Lu, Haozhe Zhang, Chenchen Jing, Zhen Li, Chuanhao Li, Jiayi Tian, Chenchen Zhang, Tianhao Peng, Yancheng He, Jihao Gu, Yuanxing Zhang, Jian Yang, Ge Zhang, Wenhao Huang, Wangchunshu Zhou, Zhaoxiang Zhang, Ruizhe Ding, Shilei Wen
  • 🏛️ Institutions: ByteDance, NJU, M-A-P, CASIA, ZJU
  • 📅 Date: August 14, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [multimodal browsing], [multimodal retrieval], [image-and-video evidence], [verified checklist], [MM-BrowseComp]
  • 📖 TLDR: MM-BrowseComp is a 224-question benchmark for browsing agents that must retrieve and reason over multimodal web evidence rather than text alone. It pairs each question with a verified checklist for fine-grained analysis and shows that even strong tool-using models remain weak on multimodal browsing.

• UI-Venus Technical Report: Building High-performance UI Agents with RFT

  • Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, Weiqiang Wang
  • 🏛️ Institutions: Ant Group
  • 📅 Date: August 14, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [model], [reinforcement fine-tuning], [trajectory history alignment], [sparse action enhancement], [Qwen2.5-VL], [UI-Venus]
  • 📖 TLDR: UI-Venus is a screenshot-only UI agent built on Qwen2.5-VL and trained with reinforcement fine-tuning plus data-cleaning pipelines for both grounding and navigation. The report attributes its gains to reward design and a self-evolving history-alignment and sparse-action mechanism, and reports strong results on ScreenSpot benchmarks and AndroidWorld.

• OpenCUA: Open Foundations for Computer-Use Agents

  • Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y.Charles, Zhilin Yang, Tao Yu
  • 🏛️ Institutions: XLANG Lab, HKU, Moonshot AI, Stanford, University of Waterloo, CMU
  • 📅 Date: August 12, 2025
  • 📑 Publisher: NeurIPS 2025 (Spotlight)
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [model], [dataset], [AgentNet], [AgentNet Tool], [reflective long CoT], [OSWorld-Verified], [OpenCUA]
  • 📖 TLDR: OpenCUA is an open-source computer-use stack centered on AgentNet Tool for demonstration capture, AgentNet for large-scale cross-platform trajectories, and a training pipeline that adds reflective long chain-of-thought supervision. The paper reports strong open-model results on OSWorld-Verified and argues that cross-platform data and test-time reasoning both materially improve agent performance.

• Test‑Time Reinforcement Learning for GUI Grounding via Region Consistency

  • Yong Du, Yuchen Yan, Fei Tang, Zhengxi Lu, Chang Zong, Weiming Lu, Shengpei Jiang, Yongliang Shen
  • 🏛️ Institutions: ZJU, Central South University, Zhejiang University of Science and Technology, SF Technology
  • 📅 Date: August 07, 2025
  • 📑 Publisher: AAAI 2026
  • 💻 Env: [Desktop], [Mobile], [Web]
  • 🔑 Key: [GUI-RC], [GUI-RCPO], [region consistency], [test-time scaling], [test-time reinforcement learning]
  • 📖 TLDR: This paper uses consistency across multiple grounding predictions as a test-time signal for GUI grounding. GUI-RC aggregates sampled outputs into consensus regions without extra training, while GUI-RCPO turns the same signal into rewards for test-time policy optimization on unlabeled data, improving ScreenSpot results across several model families.

• Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agent

  • Yuhao Cheng, Liang Tang, Shuxian Li, Yukang Huo, Tiaonan Duan, Kaer Huang, Yanzhe Jing, Yiqiang Yan
  • 🏛️ Institutions: Lenovo, China Agricultural University
  • 📅 Date: August 06, 2025
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop], [Web]
  • 🔑 Key: [Self-Evolution Agent], [step-wise reinforcement learning], [grounding-based generalization enhancement], [temporal compressed sensing], [OSWorld]
  • 📖 TLDR: This paper proposes the Self-Evolution Agent (SEA) for computer use, combining automatic verifiable trajectory generation, efficient step-wise reinforcement learning, and a model-enhancement path that merges grounding and planning ability. It evaluates the resulting agent on grounding benchmarks and OSWorld and frames the method as a way to improve computer-use performance without relying purely on manually curated data.