UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

The first benchmark for interaction inference — generating runnable web apps from UI screenshots alone.

Important

🚧 Code release coming soon. This repository is a placeholder for the official UI2App benchmark and evaluation pipeline. We are cleaning up the code for public release. ⭐ Star / Watch this repo to be notified when it lands.

Overview

Large language models are increasingly capable of generating web pages, but existing image-to-web benchmarks measure only visual fidelity — whether the render looks like the reference. A generated interface can be pixel-perfect yet be a behaviorally inert façade where every button is a no-op.

UI2App is the first benchmark targeting interaction inference: recovering an application's behavior from screenshots alone, with no textual prompt, action trace, or interaction description. Given a set of state-coherent UI screenshots, a model must generate the source code of a runnable, multi-route web application that reproduces both the look and the interactive behavior implied by those screens.

Highlights

• 🖼️ Image-only input. 327 screenshots grouped into 45 state-coherent sets → runnable multi-route apps.
• 📐 Four-dimensional protocol. EXEC@k (executability), NRS (navigation reachability), VFS (visual fidelity), and IIS (interaction inference — the core metric).
• 🧩 IIS taxonomy. 7 interaction categories (toggle, expand/collapse, list ops, data CRUD, form validation, notification, cross-route state) × 3 scope tiers (UI-state → data-state → cross-route persistence). Implementation-agnostic: any valid realization gets credit.
• 📊 6 frontier VLMs benchmarked. Visual fidelity ≠ interaction ability: the VFS leader ranks 4th on IIS (5.2× behind the leader), and cross-route state persistence is a frontier-wide bottleneck — half the models score exactly zero.

Leaderboard (main results)

Model	EXEC@1	EXEC@3	NRS	VFS	IIS
Claude Sonnet 4.6	95.6	100.0	87.3	75.7	39.3
Gemini 3.1 Pro Preview	88.9	100.0	78.2	78.1	7.5
Kimi K2.5 Thinking	86.7	86.7	64.6	65.3	20.7
Qwen3.5-397B-A17B	66.7	77.8	55.7	56.0	13.2
GPT-5.4	64.4	82.2	66.6	65.0	6.7
GLM-4.6V	33.3	35.6	22.3	22.6	4.5

Citation

If you find UI2App useful, please consider citing:

@article{chen2026ui2app,
  title         = {UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation},
  author        = {Chen, Grace Man and Guo, Litao and Wu, Yifan and Chen, Yiyu and Tseng, Yenchi and Liu, Sicheng and Luo, Yuyu and Chen, Ying-Cong},
  journal       = {arXiv preprint arXiv:2607.06306},
  year          = {2026},
  eprint        = {2607.06306},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE}
}

License

Code is released under the MIT License. The dataset is released under CC-BY-4.0 (see the dataset card).