|
1 | | -# Body Composition Estimation from Photos |
| 1 | +# Body Composition from Two Photos |
2 | 2 |
|
3 | | -Estimates **body fat %** and **appendicular lean mass** from body measurements, |
4 | | -trained against **real DXA scans** from NHANES. |
| 3 | +Estimates **body fat %** and **lean mass** from a front and side photo, validated |
| 4 | +against real **DXA scans**. Everything runs on the device — no upload, no API. |
5 | 5 |
|
6 | | -This is Stage 2 of a two-stage pipeline: |
| 6 | +**2.82% mean absolute error against clinical DXA**, beating the standard |
| 7 | +tape-measure formula (3.17%) without needing a tape. |
7 | 8 |
|
8 | 9 | ``` |
9 | | -front + side photo ──▶ [Stage 1: segmentation + measurement regression] ──▶ circumferences |
10 | | - │ |
11 | | - height, weight, age, sex ───────────────────────────────────────────────┤ |
12 | | - ▼ |
13 | | - [Stage 2: this repo] ──▶ body fat %, lean mass |
14 | | - + calibrated intervals |
| 10 | +photo pair ──▶ segmentation ──▶ Stage 1 ──▶ circumferences ──▶ Stage 2 ──▶ body fat %, lean mass |
| 11 | + (BodyM) (NHANES DXA) + 80% intervals |
15 | 12 | ``` |
16 | 13 |
|
17 | | -Stage 2 was built first deliberately: if measurements → body fat cannot be done |
18 | | -accurately, no amount of computer vision saves the product. Both stages are now |
19 | | -trained, and the pipeline runs end to end from a silhouette pair. |
| 14 | +| | | |
| 15 | +|---|---| |
| 16 | +| **Body fat, end to end** | 2.82% MAE vs DXA | |
| 17 | +| **Waist from silhouette** | 1.97 cm MAE | |
| 18 | +| **Interval coverage** | 80.5% (claimed 80%) | |
| 19 | +| **Model size / latency** | 1.4 MB, ~16 ms in-browser | |
| 20 | +| **Training data** | 11,701 DXA scans, 2,018 scanned bodies | |
| 21 | + |
| 22 | + |
| 23 | + |
| 24 | +### What makes it worth reading |
| 25 | + |
| 26 | +- **Deployment splits, never random.** Headline numbers hold out the entire |
| 27 | + 2017–2018 NHANES cycle. |
| 28 | +- **The intervals were lying, and I caught it.** Quantile regression claimed 80% |
| 29 | + coverage and delivered 65.6%. Conformal calibration fixed it to 80.5%. |
| 30 | +- **IoU is the wrong metric for segmentation here.** A 0.90-IoU soft matte was |
| 31 | + harmless; a 0.92-IoU dilated mask was the worst case measured. Direction of |
| 32 | + error beats magnitude. |
| 33 | +- **Python and JavaScript agree exactly.** 440 predictions bit-identical, 906 |
| 34 | + feature values to 4.6e-13 — asserted in CI, not assumed. |
| 35 | +- **Limitations are measured, not hidden.** A generalisation gap on the harder |
| 36 | + test set, a ~1.4 point bias for people of Asian descent, and clothing |
| 37 | + (a normal t-shirt adds ~5 cm to the waist). |
| 38 | + |
| 39 | +### Run it |
20 | 40 |
|
21 | | -**Headline: 2.84% mean absolute error on body fat vs DXA, end to end from photos** — |
22 | | -better than the best published tape-measure formula (3.16%), without the user |
23 | | -owning a tape measure. |
24 | | - |
25 | | -## Results |
26 | | - |
27 | | -Held-out **2017–2018 NHANES cycle** (n = 2,425) — trained only on 2011–2016, so |
28 | | -the test cohort was collected years later by different examiners. |
29 | | - |
30 | | -| Model | MAE | RMSE | R² | 95% limits of agreement | within ±3 pts | |
31 | | -|---|---|---|---|---|---| |
32 | | -| Deurenberg (BMI formula) | 4.58 | 5.89 | 0.533 | −12.0 to +11.0 | 42.3% | |
33 | | -| RFM (waist formula) | 3.16 | 4.00 | 0.784 | −5.9 to +8.8 | 54.8% | |
34 | | -| Linear regression | 2.86 | 3.65 | 0.821 | −7.3 to +7.0 | 60.4% | |
35 | | -| **LightGBM** | **2.70** | **3.46** | **0.839** | **−7.0 to +6.5** | **63.7%** | |
36 | | - |
37 | | -Appendicular lean mass: **MAE 1.27 kg**, R² 0.933. |
38 | | - |
39 | | -**14% better than the best published formula baseline, 41% better than BMI.** |
40 | | -Bias is −0.23 percentage points — essentially unbiased. For scale, two different |
41 | | -DXA machines disagree with each other by about 1–2 points. |
42 | | - |
43 | | - |
44 | | - |
45 | | -### Calibrated uncertainty |
46 | | - |
47 | | -Raw LightGBM quantile heads were badly over-confident. Split-conformal |
48 | | -calibration fixes it: |
49 | | - |
50 | | -| target | nominal | coverage before | coverage after | width | |
51 | | -|---|---|---|---|---| |
52 | | -| body fat % | 80% | 65.6% | **80.5%** | 6.4 → 8.8 | |
53 | | -| lean mass kg | 80% | 68.6% | **81.4%** | 3.2 → 4.1 | |
54 | | - |
55 | | -The app reports "18.2% ± 2.4%", not a bare number that implies false precision. |
56 | | - |
57 | | -### Where it fails |
58 | | - |
59 | | -| slice | worst group | MAE | bias | |
60 | | -|---|---|---|---| |
61 | | -| BMI | under 18.5 (n=61) | 3.41 | −0.23 | |
62 | | -| sex | female | 2.85 | −0.12 | |
63 | | -| ethnicity | Asian (n=437) | 2.90 | **−1.40** | |
64 | | - |
65 | | -Two honest caveats: accuracy degrades at the lean tail where training data is |
66 | | -thin, and the model **under-estimates body fat in Asian subjects by 1.4 points** — |
67 | | -a known limitation of anthropometric methods, which is why race/ethnicity is |
68 | | -excluded from the feature set and used only to audit fairness. Encoding it as a |
69 | | -feature would improve the metric while baking demographic assumptions into a |
70 | | -consumer health product. |
71 | | - |
72 | | - |
73 | | -## Stage 1 — silhouette → tape measurements |
74 | | - |
75 | | -Trained on **BodyM** (2,018 subjects, 6,134 photo pairs), evaluated on BodyM's own |
76 | | -held-out `testA` and `testB` splits — different subjects, so no leakage by construction. |
77 | | - |
78 | | -Each silhouette is sampled at 48 normalised heights; at every level the front and |
79 | | -side chord widths define an ellipse, and Ramanujan's formula turns that into a |
80 | | -circumference estimate. The model only has to learn the correction from ellipse |
81 | | -to true tape measurement, not the whole mapping from pixels. |
82 | | - |
83 | | -| measurement | h+w+sex only | **+ silhouette** | gain | R² | |
84 | | -|---|---|---|---|---| |
85 | | -| waist | 3.37 | **1.97** | 41% | 0.937 | |
86 | | -| hip | 2.29 | **1.51** | 34% | 0.949 | |
87 | | -| chest | 3.07 | **2.37** | 23% | 0.919 | |
88 | | -| thigh | 1.67 | **1.36** | 19% | 0.884 | |
89 | | -| bicep | 1.29 | **1.15** | 11% | 0.821 | |
90 | | -| shoulder breadth | 0.78 | **0.60** | 23% | 0.903 | |
91 | | - |
92 | | -MAE in cm on `testA`. The first column is the ablation that matters: a model given |
93 | | -only height, weight and sex. **The silhouette cuts waist error by 41%**, which is |
94 | | -what justifies the camera existing at all. |
95 | | - |
96 | | -On the harder `testB` split waist MAE rises to 3.18 cm and the gain shrinks to 3%. |
97 | | -That gap is the real engineering finding: performance is dominated by capture |
98 | | -conditions, not model capacity, so the app must control pose, distance and |
99 | | -lighting rather than chase a bigger network. |
100 | | - |
101 | | -## End-to-end |
102 | | - |
103 | | -No public dataset pairs photos with DXA, so the pipeline cannot be scored |
104 | | -directly. Instead the Stage 1 error distribution is propagated through Stage 2 by |
105 | | -Monte Carlo (25 draws) over NHANES test subjects: |
106 | | - |
107 | | -| input to Stage 2 | body fat MAE | vs tape | |
108 | | -|---|---|---| |
109 | | -| true tape measurements | 2.70 | — | |
110 | | -| Stage 1 estimates, good conditions | **2.84** | +0.13 | |
111 | | -| Stage 1 estimates, harder conditions | **2.99** | +0.28 | |
112 | | -| *published RFM formula, tape* | *3.16* | — | |
113 | | - |
114 | | -Vision costs only **0.13–0.28 points** of body fat accuracy, and the photo pipeline |
115 | | -still beats the best tape-measure formula. |
116 | | - |
117 | | -One integration risk was found and closed. Stage 2's strongest single feature was |
118 | | -NHANES upper-leg length, which BodyM does not define the same way (its |
119 | | -`leg-length` is the full leg). Training a *photo-available* Stage 2 variant that |
120 | | -drops limb lengths entirely costs **0.01 points** (2.70 → 2.71) — the tree model had |
121 | | -collinear substitutes. The shipped model uses only quantities Stage 1 can |
122 | | -genuinely supply, so no measurement definition is silently mismatched. |
123 | | - |
124 | | - |
125 | | -## Segmentation and capture quality |
126 | | - |
127 | | -The last gap is closed: `src/segment.py` turns an ordinary camera photo into the |
128 | | -binary silhouette Stage 1 expects, using MediaPipe selfie segmentation plus pose |
129 | | -landmarks. Two design decisions come straight out of the robustness study below. |
130 | | - |
131 | | -**The alpha matte is thresholded conservatively** (0.62, not 0.5). Over-segmentation |
132 | | -costs real accuracy; under-segmentation does not. When in doubt, cut in. |
133 | | - |
134 | | -**Quality control gates on pose and framing, not mask quality.** The photo path |
135 | | -refuses to return a number when a check fails — a silent failure hands the user a |
136 | | -confidently wrong reading. Checks: whole body in frame, feet and head visible, |
137 | | -arms clear of the torso, shoulders level and square. |
138 | | - |
139 | | -### How good does the segmenter have to be? |
140 | | - |
141 | | -Ground-truth BodyM masks were degraded in ways real segmenters fail, then fed |
142 | | -through the trained Stage 1 model (600 photo pairs, unseen subjects): |
143 | | - |
144 | | -| Failure mode | Mask IoU | Waist MAE | Penalty | |
145 | | -|---|---|---|---| |
146 | | -| clean ground truth | 1.000 | 2.01 | — | |
147 | | -| boundary noise | 0.99 | 2.03 | +0.02 | |
148 | | -| erosion (under-segment) | 0.91–0.97 | 1.89–1.95 | **−0.09** | |
149 | | -| soft/blurred matte | 0.76–0.97 | 1.93–2.60 | +0.16 | |
150 | | -| **dilation (over-segment)** | 0.92–0.97 | 2.14–2.70 | **+0.41** | |
151 | | - |
152 | | -**IoU is the wrong metric to tune the segmenter on.** A 0.90-IoU soft matte was |
153 | | -harmless, while a 0.92-IoU dilated mask was the worst case in the study. What |
154 | | -matters is the *direction* of the error: eating a few boundary pixels is free, |
155 | | -adding them is not — loose clothing and hair are the real enemy, not jagged edges. |
156 | | - |
157 | | -### Validated how far |
158 | | - |
159 | | -`segment.py` was run on a real public-domain full-length photograph. Segmentation |
160 | | -recovered the head and torso but **lost the legs entirely**, where dark trousers met a |
161 | | -dark background — and quality control correctly rejected the photo rather than |
162 | | -returning a number. Forced past the gate, it produced a 122.8 cm hip for a slim |
163 | | -subject, which is exactly the silent failure the gate prevents. |
| 41 | +```bash |
| 42 | +cd app && npm install && npm start # http://localhost:8742 |
| 43 | +``` |
164 | 44 |
|
165 | | -That is a stress test on a low-contrast 19th-century plate, not a representative |
166 | | -one. **Segmentation accuracy on ordinary phone photos is not yet measured** — that |
167 | | -needs a handful of real A-pose captures against a plain background. |
| 45 | +```bash |
| 46 | +pip install -r requirements.txt # Python side: training and evaluation |
| 47 | +``` |
168 | 48 |
|
| 49 | +No camera? The app has a **Run a sample body** button — a real BodyM subject with |
| 50 | +known tape measurements, so the output is shown beside the right answer. |
169 | 51 |
|
170 | 52 | ## The app |
171 | 53 |
|
@@ -202,7 +84,7 @@ is how wrong numbers ship: |
202 | 84 | | `selftest.html` | full pipeline + multi-frame median | exact on all 11 values | |
203 | 85 | | `scan_score_test.mjs` | frame scoring, alignment gate | 15 checks pass | |
204 | 86 | | `scan_steps_test.mjs` | stepped capture state machine | 18 checks pass | |
205 | | -| `overlay_test.mjs` | contour and point-mesh geometry | 20 checks pass | |
| 87 | +| `overlay_test.mjs` | contour, mesh and chord geometry | 26 checks pass | |
206 | 88 | | `plan_test.mjs` | plan maths and safety floors | 30 checks pass | |
207 | 89 |
|
208 | 90 | ### A bug worth keeping |
|
0 commit comments