Skip to content

Commit 4123d3b

Browse files
committed
docs: reorder README — Why This Exists above Quick Start, Output after commands
1 parent 423a8dc commit 4123d3b

1 file changed

Lines changed: 18 additions & 20 deletions

File tree

README.md

Lines changed: 18 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,18 @@ A Set-of-Mark (SoM) detection pipeline for macOS that transforms screenshots int
88

99
*229 elements detected in 2.4s — text labels (Apple Vision), rectangles, icons, and buttons (Florence-2). [Full manifest JSON →](docs/examples/vscode-229-manifest.json)*
1010

11+
## Why This Exists
12+
13+
We needed a vision model that could find every button, label, and icon on a macOS screenshot — to make screenshots machine-readable. We surveyed 14 detection models. The best ones (Screen2AX, OmniParser) were AGPL — unusable for MIT distribution. The MIT-licensed options under 10B parameters — Florence-2, PTA-1, and others — all produced the same failure: a single bounding box covering the entire screen.
14+
15+
We tried 7 configurations of frequency and repetition penalties. Prompt engineering. Resolution reduction. Nothing fixed it. This isn't a tuning problem — it's a model capacity limitation.
16+
17+
Then we noticed something: the same models detect reliably on cropped regions.
18+
19+
That's the core insight. uitag doesn't force a small model to see a complex desktop. It tiles the screenshot into quadrants first — with cut lines placed to avoid bisecting UI elements — and runs detection on each tile separately. Apple Vision handles text and rectangles natively on the ANE (fast, free, no model download). Florence-2 catches everything else — icons, buttons, images — at 159MB on Metal.
20+
21+
**Every model we tested returned 1 bounding box. uitag returns 151 — in 1.7 seconds, fully open-source under MIT.** [Full research methodology →](docs/research.md)
22+
1123
## Quick Start
1224

1325
```bash
@@ -76,26 +88,6 @@ Patch file format:
7688
}
7789
```
7890

79-
## Why This Exists
80-
81-
We needed a vision model that could find every button, label, and icon on a macOS screenshot — to make screenshots machine-readable. We surveyed 14 detection models. The best ones (Screen2AX, OmniParser) were AGPL — unusable for MIT distribution. The MIT-licensed options under 10B parameters — Florence-2, PTA-1, and others — all produced the same failure: a single bounding box covering the entire screen.
82-
83-
We tried 7 configurations of frequency and repetition penalties. Prompt engineering. Resolution reduction. Nothing fixed it. This isn't a tuning problem — it's a model capacity limitation.
84-
85-
Then we noticed something: the same models detect reliably on cropped regions.
86-
87-
That's the core insight. uitag doesn't force a small model to see a complex desktop. It tiles the screenshot into quadrants first — with cut lines placed to avoid bisecting UI elements — and runs detection on each tile separately. Apple Vision handles text and rectangles natively on the ANE (fast, free, no model download). Florence-2 catches everything else — icons, buttons, images — at 159MB on Metal.
88-
89-
**Every model we tested returned 1 bounding box. uitag returns 151 — in 1.7 seconds, fully open-source under MIT.** [Full research methodology →](docs/research.md)
90-
91-
92-
## Pipeline Architecture
93-
94-
| Pipeline Stage | Process Flow Description |
95-
| :--- | :--- |
96-
| ![system architecture uitag](docs/assets/uitag-architecture-diagram.jpg)|* [1] VNRecognizeTextRequest + VNDetectRectanglesRequest<br> * [3] \<OD\> detection on each tile<br> * [4] IoU-based overlap removal, source priority ranking, numbered markers + colored bounding boxes<br> * [5] manifest includes element list with coordinates, labels, sources, timing |
97-
98-
9991
## Output Format
10092
### JSON
10193

@@ -127,6 +119,12 @@ That's the core insight. uitag doesn't force a small model to see a complex desk
127119
![uitag output — 151 tagged UI elements on a VS Code screenshot](docs/examples/hero-after.png)
128120
*151 elements detected in ~1.7s — text labels (Apple Vision), rectangles, icons, and buttons (Florence-2). [Full manifest JSON →](docs/examples/vscode-manifest.json)*
129121

122+
## Pipeline Architecture
123+
124+
| Pipeline Stage | Process Flow Description |
125+
| :--- | :--- |
126+
| ![system architecture uitag](docs/assets/uitag-architecture-diagram.jpg)|* [1] VNRecognizeTextRequest + VNDetectRectanglesRequest<br> * [3] \<OD\> detection on each tile<br> * [4] IoU-based overlap removal, source priority ranking, numbered markers + colored bounding boxes<br> * [5] manifest includes element list with coordinates, labels, sources, timing |
127+
130128
## Tips
131129

132130
- **Use light mode for best OCR accuracy.** Apple Vision produces measurably better results on light mode screenshots, especially for special characters in code, regex patterns, and variable names. In testing, a backslash character (`\`) that was unrecoverable in dark mode across all techniques was correctly read in light mode. [Full research findings →](docs/research/ocr-rescan-experiments.md)

0 commit comments

Comments
 (0)