You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+18-20Lines changed: 18 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,18 @@ A Set-of-Mark (SoM) detection pipeline for macOS that transforms screenshots int
8
8
9
9
*229 elements detected in 2.4s — text labels (Apple Vision), rectangles, icons, and buttons (Florence-2). [Full manifest JSON →](docs/examples/vscode-229-manifest.json)*
10
10
11
+
## Why This Exists
12
+
13
+
We needed a vision model that could find every button, label, and icon on a macOS screenshot — to make screenshots machine-readable. We surveyed 14 detection models. The best ones (Screen2AX, OmniParser) were AGPL — unusable for MIT distribution. The MIT-licensed options under 10B parameters — Florence-2, PTA-1, and others — all produced the same failure: a single bounding box covering the entire screen.
14
+
15
+
We tried 7 configurations of frequency and repetition penalties. Prompt engineering. Resolution reduction. Nothing fixed it. This isn't a tuning problem — it's a model capacity limitation.
16
+
17
+
Then we noticed something: the same models detect reliably on cropped regions.
18
+
19
+
That's the core insight. uitag doesn't force a small model to see a complex desktop. It tiles the screenshot into quadrants first — with cut lines placed to avoid bisecting UI elements — and runs detection on each tile separately. Apple Vision handles text and rectangles natively on the ANE (fast, free, no model download). Florence-2 catches everything else — icons, buttons, images — at 159MB on Metal.
20
+
21
+
⚡ **Every model we tested returned 1 bounding box. uitag returns 151 — in 1.7 seconds, fully open-source under MIT.**[Full research methodology →](docs/research.md)
22
+
11
23
## Quick Start
12
24
13
25
```bash
@@ -76,26 +88,6 @@ Patch file format:
76
88
}
77
89
```
78
90
79
-
## Why This Exists
80
-
81
-
We needed a vision model that could find every button, label, and icon on a macOS screenshot — to make screenshots machine-readable. We surveyed 14 detection models. The best ones (Screen2AX, OmniParser) were AGPL — unusable for MIT distribution. The MIT-licensed options under 10B parameters — Florence-2, PTA-1, and others — all produced the same failure: a single bounding box covering the entire screen.
82
-
83
-
We tried 7 configurations of frequency and repetition penalties. Prompt engineering. Resolution reduction. Nothing fixed it. This isn't a tuning problem — it's a model capacity limitation.
84
-
85
-
Then we noticed something: the same models detect reliably on cropped regions.
86
-
87
-
That's the core insight. uitag doesn't force a small model to see a complex desktop. It tiles the screenshot into quadrants first — with cut lines placed to avoid bisecting UI elements — and runs detection on each tile separately. Apple Vision handles text and rectangles natively on the ANE (fast, free, no model download). Florence-2 catches everything else — icons, buttons, images — at 159MB on Metal.
88
-
89
-
⚡ **Every model we tested returned 1 bounding box. uitag returns 151 — in 1.7 seconds, fully open-source under MIT.**[Full research methodology →](docs/research.md)
90
-
91
-
92
-
## Pipeline Architecture
93
-
94
-
| Pipeline Stage | Process Flow Description |
95
-
| :--- | :--- |
96
-
||*[1] VNRecognizeTextRequest + VNDetectRectanglesRequest<br> * [3]\<OD\> detection on each tile<br> * [4] IoU-based overlap removal, source priority ranking, numbered markers + colored bounding boxes<br> * [5] manifest includes element list with coordinates, labels, sources, timing |
97
-
98
-
99
91
## Output Format
100
92
### JSON
101
93
@@ -127,6 +119,12 @@ That's the core insight. uitag doesn't force a small model to see a complex desk
127
119

128
120
*151 elements detected in ~1.7s — text labels (Apple Vision), rectangles, icons, and buttons (Florence-2). [Full manifest JSON →](docs/examples/vscode-manifest.json)*
129
121
122
+
## Pipeline Architecture
123
+
124
+
| Pipeline Stage | Process Flow Description |
125
+
| :--- | :--- |
126
+
||*[1] VNRecognizeTextRequest + VNDetectRectanglesRequest<br> * [3]\<OD\> detection on each tile<br> * [4] IoU-based overlap removal, source priority ranking, numbered markers + colored bounding boxes<br> * [5] manifest includes element list with coordinates, labels, sources, timing |
127
+
130
128
## Tips
131
129
132
130
-**Use light mode for best OCR accuracy.** Apple Vision produces measurably better results on light mode screenshots, especially for special characters in code, regex patterns, and variable names. In testing, a backslash character (`\`) that was unrecoverable in dark mode across all techniques was correctly read in light mode. [Full research findings →](docs/research/ocr-rescan-experiments.md)
0 commit comments