Skip to content

Commit c1ee6d1

Browse files
authored
Revise README to clarify vision model insights
1 parent 76d452d commit c1ee6d1

1 file changed

Lines changed: 10 additions & 4 deletions

File tree

README.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,15 +40,21 @@ uitag benchmark <image> Measure per-stage pipeline timing
4040

4141
## Why This Exists
4242

43-
We needed a vision model that could find every button, label, and icon on a macOS screenshot — so an agent could click on them. We surveyed 14 detection models. The best ones (Screen2AX, OmniParser) were AGPL — unusable for MIT distribution. The MIT-licensed options under 10B parameters — Florence-2, PTA-1, and others — all produced the same failure: a single bounding box covering the entire screen.
43+
### The problem
44+
We needed a vision model that could find every button, label, and icon on a macOS screenshot — to enable agent interactivity.
4445

45-
We tried 7 configurations of frequency and repetition penalties. Prompt engineering. Resolution reduction. Nothing fixed it. This isn't a tuning problem — it's a model capacity limitation.
46+
### No feasible solutions existed
47+
We surveyed 14 detection models. The best ones (Screen2AX, OmniParser) were AGPL — unusable for MIT distribution. The MIT-licensed options under 10B parameters — Florence-2, PTA-1, and others — all produced the same failure: a single bounding box covering the entire screen.
4648

49+
We tried 7 configurations of frequency and repetition penalties. Prompt engineering. Resolution reduction. Nothing fixed it. So it wasn't a tuning problem. Was it a model capacity limitation?
50+
51+
### The research breakthrough
4752
Then we noticed something: the same models detect reliably on cropped regions.
4853

49-
That's the core insight. uitag doesn't force a small model to see a complex desktop. It tiles the screenshot into quadrants first — with cut lines placed to avoid bisecting UI elements — and runs detection on each tile separately. Apple Vision handles text and rectangles natively on the ANE (fast, free, no model download). Florence-2 catches everything else — icons, buttons, images — at 159MB on Metal.
54+
💡 **That's the core insight.** uitag doesn't force a small model to see a complex desktop. It tiles the screenshot into quadrants first — with cut lines placed to avoid bisecting UI elements — and runs detection on each tile separately. Apple Vision handles text and rectangles natively on the ANE (fast, free, no model download). Florence-2 catches everything else — icons, buttons, images — at 159MB on Metal.
5055

51-
The result: 151 elements detected on a VS Code screenshot in ~1.7 seconds. A numbered element map and a JSON manifest that any downstream agent can consume directly. [Full research methodology →](docs/research.md)
56+
### The result
57+
151 elements detected on a VS Code screenshot in ~1.7 seconds. A numbered element map and a JSON manifest that any downstream agent can consume directly. [Full research methodology →](docs/research.md)
5258

5359
## Pipeline Architecture
5460

0 commit comments

Comments
 (0)