Skip to content

Commit bea9010

Browse files
committed
docs: add API reference, performance benchmarks, and troubleshooting guide
- docs/api.md: full API reference with parameter tables, examples, custom backends, manifest schema - docs/performance.md: fresh benchmarks (M2 Max, 3-run avg), stage breakdown, backend comparison - docs/troubleshooting.md: common errors, model FAQ, detection quality, platform limitations - docs/roadmap.md: cleaned for public consumption - README: updated perf numbers to match benchmarks, added Documentation section - docs/research.md: reconciled perf numbers with fresh benchmark data - .gitignore: exclude internal docs from public repo
1 parent 6da4d2c commit bea9010

7 files changed

Lines changed: 920 additions & 22 deletions

File tree

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ _docs/
2323
CLAUDE.md
2424
docs/plans/
2525
docs/interagent-somflow/
26+
docs/interagent-uitag/
27+
docs/current-state.md
28+
docs/uitag-launch-steps.md
29+
docs/f1.3-coreml-acceleration.md
2630
.sediment/
2731

2832
# Worktrees

README.md

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ A Set-of-Mark (SoM) detection pipeline for macOS that transforms screenshots int
66

77
![uitag output — 151 UI elements detected on a 1920x1080 screenshot](https://raw.githubusercontent.com/swaylenhayes/uitag/main/docs/examples/vscode-som.png)
88

9-
*151 numbered elements detected in ~0.8s — text labels (Apple Vision), rectangles, icons, and buttons (Florence-2). [Full manifest JSON →](docs/examples/vscode-manifest.json)*
9+
*151 numbered elements detected in ~1.7s — text labels (Apple Vision), rectangles, icons, and buttons (Florence-2). [Full manifest JSON →](docs/examples/vscode-manifest.json)*
1010

1111
## Why This Exists
1212

@@ -27,13 +27,13 @@ Screenshot (1920x1080)
2727
v
2828
[1] Apple Vision (Swift binary)
2929
| VNRecognizeTextRequest + VNDetectRectanglesRequest
30-
| ~189ms (fast) / ~980ms (accurate)
30+
| ~213ms (fast) / ~977ms (accurate)
3131
v
3232
[2] Object-Aware Tiling
3333
| Split into 4 quadrants, cut lines avoid bounding boxes
3434
v
3535
[3] Florence-2 (mlx_vlm, per quadrant)
36-
| <OD> detection on each tile, ~160ms/quadrant
36+
| <OD> detection on each tile, ~220ms/quadrant
3737
v
3838
[4] Merge + Deduplicate
3939
| IoU-based overlap removal, source priority ranking
@@ -48,8 +48,8 @@ Output: annotated.png + manifest.json
4848
```
4949

5050
End-to-end on a 1920x1080 VS Code screenshot (~151 UI elements detected):
51-
- **~0.8s** with fast OCR (Florence-2 ~650ms + Vision ~189ms)
52-
- **~1.6s** with accurate OCR (Florence-2 ~650ms + Vision ~980ms)
51+
- **~1.7s** with fast OCR (Florence-2 ~1.5s + Vision ~213ms)
52+
- **~2.6s** with accurate OCR (Florence-2 ~1.5s + Vision ~977ms)
5353

5454
## Quick Start
5555

@@ -112,6 +112,14 @@ Options:
112112
--backend BACKEND Detection backend: auto (default), coreml, mlx
113113
```
114114

115+
## Documentation
116+
117+
- [API Reference](docs/api.md) — Functions, types, and manifest schema
118+
- [Performance](docs/performance.md) — Benchmarks and optimization tips
119+
- [Troubleshooting](docs/troubleshooting.md) — Common issues and FAQ
120+
- [Research Background](docs/research.md) — Model selection and benchmark methodology
121+
- [Contributing](CONTRIBUTING.md) — Setup and PR guidelines
122+
115123
## Requirements
116124

117125
- **macOS** (Apple Vision Framework is macOS-only)
@@ -123,7 +131,7 @@ Options:
123131

124132
uitag supports pluggable detection backends via the `DetectionBackend` protocol:
125133

126-
- **MLX** (default) — Florence-2 inference on GPU via Metal. ~160ms per quadrant on M2 Max.
134+
- **MLX** (default) — Florence-2 inference on GPU via Metal. ~220ms per quadrant on M2 Max.
127135
- **CoreML** — DaViT vision encoder on Apple Neural Engine, decoder on GPU. Useful when GPU is contended by other workloads. Requires a converted model (`python tools/convert_davit_coreml.py`).
128136

129137
```bash

0 commit comments

Comments
 (0)