Skip to content

Commit a5ba354

Browse files
borys3kkmsluszniak
andauthored
feat(cv): add MI-GAN image inpainting (#1573)
## Description Adds image inpainting on MI-GAN: paint a mask over a photo and the model fills the region with plausible content (object removal, cleanup). - **Native `cv` op**: `blendMasked` (masked composite, with optional erosion and inward feathering of the mask; the presets use neither, since the upsampled fill is already soft), with host C++ tests. - **Task + hook**: `createInpainter` / `useInpainter`. The task resizes the whole frame to the model, runs `forward`, resizes the output back and blends it into the full-resolution frame through the mask. Cropping around the hole is left to the caller. The public mask marks the hole with any non-zero pixel; the inversion to the graph's `1 = known` convention happens inside the task. An empty mask returns the input unchanged. - **Registry**: `models.inpainting.MIGAN` with `SIZE_512`, `SIZE_256` (Places2) and `FACES_256` (FFHQ), fp16 only, on XNNPACK, Core ML and Vulkan. New `inpainting` feature in the native libs map. - **Docs**: new Image Inpainting page; camera integration renumbered to 12. - **Example app**: Image Inpainting screen with a brush-painted mask (`MaskCanvas`), result/original toggle. `app.json` gains `NSLocalNetworkUsageDescription` so a dev build on a physical iPhone can reach Metro and local files. The exports come from export-scripts MR !31 and are published to [`software-mansion/react-native-executorch-migan`](https://huggingface.co/software-mansion/react-native-executorch-migan) at `v0.11.0`. Native libs are pinned to `0.10.7`, which the XNNPACK exports need. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [x] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions 1. `apps/computer-vision` → Image Inpainting. 2. Pick a photo, paint over an object with the brush (cover it fully, with a margin), Run Async / Run Sync. 3. Toggle Show Original to compare. Library: `yarn workspace react-native-executorch test` (12 new tests in `inpainter.test.ts`), `scripts/run-native-tests.sh -R CvOpsTest` (needs OpenCV 4; Homebrew's `opencv` is 5.x and does not link against the vendored 4.x headers, use `opencv@4` with `OpenCV_DIR`). Measured with the published `v0.11.0` files and native libs 0.10.7, `forward` only, medians after warm-up: | Device | Model | Backend | `forward` | | :-- | :-- | :-- | :-- | | iPhone 17 | MI-GAN 512 | Core ML | 29 ms | | iPhone 17 | MI-GAN 512 | XNNPACK | 81 ms | | iPhone 17 | MI-GAN 256 | Core ML | 11 ms | | iPhone 17 | MI-GAN 256 | XNNPACK | 47 ms | | Galaxy S23 | MI-GAN 512 | Vulkan | 175 ms | | Galaxy S23 | MI-GAN 512 | XNNPACK | 194 ms | | Galaxy S23 | MI-GAN 256 | Vulkan | 101 ms | | Galaxy S23 | MI-GAN 256 | XNNPACK | 116 ms | Faces 256 runs at the same speed as 256. Whole `inpaint()` with MI-GAN 512 on a 708x424 photo: ~35 ms Core ML / ~89 ms XNNPACK on the iPhone 17, ~200-260 ms Vulkan / ~260-300 ms XNNPACK on the S23. At 12 MP the full-resolution resize and blend dominate: 100-250 ms on the iPhone, 0.4-1.6 s on the S23 with a large run-to-run spread. End-to-end on both devices (all six variants per platform, eight real photos incl. two at 12 MP, synthetic edge cases down to 1x1 and a single-pixel hole, RGB vs RGBA input, sync vs async, two calls in flight, dispose): pixels outside the mask come back unchanged on every case, outputs are deterministic, and inside the mask they are within 1-9 levels (mean) of upstream's reference results on the `places2_512` examples. Vulkan and XNNPACK outputs on the S23 differ by about one level on average (fp16 on GPU vs CPU). With upstream's `sheep_mask.png` example the output matches upstream's `sheep_out.png`. ### Screenshots _(to be added)_ ### Related issues #1373 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes - A mask that cuts through an object makes MI-GAN extend that object into the hole rather than remove it; the docs carry a tip about covering the whole object. - `@shopify/react-native-skia` deprecates `SkPath.moveTo/lineTo`; the mask component uses `PathBuilder`. - Pre-commit hooks locally need `cspell` on `PATH` (`node_modules/.bin`) and all workspaces built (`yarn prepare`), otherwise `spell-check-other` and `types` fail for reasons unrelated to the change. --------- Co-authored-by: Mateusz Słuszniak <mateusz.sluszniak@swmansion.com>
1 parent 5c521a5 commit a5ba354

24 files changed

Lines changed: 1687 additions & 8 deletions

File tree

‎.cspell-wordlist.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -350,6 +350,7 @@ dontMock
350350
BIES
351351
binarize
352352
binarized
353+
binarizing
353354
unclip
354355
phonemizations
355356
phonemizes
@@ -428,7 +429,17 @@ zipdepth
428429
xiaobei
429430
xiaoxiao
430431
xiaoyi
432+
xiaoni
431433
yunjian
432434
yunxi
433435
yunxia
434436
yunyang
437+
migan
438+
inpaint
439+
inpainted
440+
inpainter
441+
inpainting
442+
Picsart
443+
FFHQ
444+
grayscale
445+
Premul

‎apps/computer-vision/app.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@
1717
"supportsTablet": true,
1818
"bundleIdentifier": "com.anonymous.computervision",
1919
"infoPlist": {
20-
"NSCameraUsageDescription": "Process photo from camera"
20+
"NSCameraUsageDescription": "Process photo from camera",
21+
"NSLocalNetworkUsageDescription": "Connect to the Metro dev server and local model files on your computer",
22+
"NSBonjourServices": ["_http._tcp"]
2123
}
2224
},
2325
"android": {

‎apps/computer-vision/app/_layout.tsx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,13 @@ export default function Layout() {
4848
title: 'Style Transfer',
4949
}}
5050
/>
51+
<Drawer.Screen
52+
name="inpainting/index"
53+
options={{
54+
drawerLabel: 'Image Inpainting',
55+
title: 'Image Inpainting',
56+
}}
57+
/>
5158
<Drawer.Screen
5259
name="textToImage/index"
5360
options={{

‎apps/computer-vision/app/index.tsx‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ export default function Home() {
3030
<TouchableOpacity style={styles.button} onPress={() => router.navigate('styleTransfer/')}>
3131
<Text style={styles.buttonText}>Style Transfer</Text>
3232
</TouchableOpacity>
33+
<TouchableOpacity style={styles.button} onPress={() => router.navigate('inpainting/')}>
34+
<Text style={styles.buttonText}>Image Inpainting</Text>
35+
</TouchableOpacity>
3336
<TouchableOpacity style={styles.button} onPress={() => router.navigate('textToImage/')}>
3437
<Text style={styles.buttonText}>Text to Image</Text>
3538
</TouchableOpacity>
Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
import React, { useState } from 'react';
2+
import { View, Text, Platform, TouchableOpacity, StyleSheet } from 'react-native';
3+
// The gesture-handler ScrollView yields to the brush pan over the photo, so
4+
// painting never scrolls the page and scrolling still works everywhere else.
5+
import { ScrollView } from 'react-native-gesture-handler';
6+
import { useSafeAreaInsets } from 'react-native-safe-area-context';
7+
import { commonStyles, theme } from '../../theme';
8+
import {
9+
useImage,
10+
Skia,
11+
ColorType,
12+
AlphaType,
13+
type SkImage as SkiaImageType,
14+
} from '@shopify/react-native-skia';
15+
import { useInpainter, models } from 'react-native-executorch';
16+
import ScreenWrapper from '../../components/ScreenWrapper';
17+
import { getImage, skImageToBuffer } from '../../utils';
18+
import { ModelPicker, type ModelOption } from '../../components/ModelPicker';
19+
import { ImageViewport } from '../../components/ImageViewport';
20+
import { ModelStatus } from '../../components/ModelStatus';
21+
import { LatencyIndicator } from '../../components/LatencyIndicator';
22+
import { Button } from '../../components/Button';
23+
import { MaskCanvas, rasterizeMask, type MaskStroke } from '../../components/MaskCanvas';
24+
25+
const MODEL_OPTIONS: ModelOption[] = [
26+
{ label: 'MI-GAN 512 (XNNPACK FP16)', value: models.inpainting.MIGAN.SIZE_512.XNNPACK_FP16 },
27+
{
28+
label: 'MI-GAN 512 (CoreML FP16)',
29+
value: models.inpainting.MIGAN.SIZE_512.COREML_FP16,
30+
disabled: Platform.OS !== 'ios',
31+
},
32+
{
33+
label: 'MI-GAN 512 (Vulkan FP16)',
34+
value: models.inpainting.MIGAN.SIZE_512.VULKAN_FP16,
35+
disabled: Platform.OS !== 'android',
36+
},
37+
{ label: 'MI-GAN 256 (XNNPACK FP16)', value: models.inpainting.MIGAN.SIZE_256.XNNPACK_FP16 },
38+
{
39+
label: 'MI-GAN 256 (CoreML FP16)',
40+
value: models.inpainting.MIGAN.SIZE_256.COREML_FP16,
41+
disabled: Platform.OS !== 'ios',
42+
},
43+
{
44+
label: 'MI-GAN 256 (Vulkan FP16)',
45+
value: models.inpainting.MIGAN.SIZE_256.VULKAN_FP16,
46+
disabled: Platform.OS !== 'android',
47+
},
48+
{
49+
label: 'MI-GAN Faces 256 (XNNPACK FP16)',
50+
value: models.inpainting.MIGAN.FACES_256.XNNPACK_FP16,
51+
},
52+
{
53+
label: 'MI-GAN Faces 256 (CoreML FP16)',
54+
value: models.inpainting.MIGAN.FACES_256.COREML_FP16,
55+
disabled: Platform.OS !== 'ios',
56+
},
57+
{
58+
label: 'MI-GAN Faces 256 (Vulkan FP16)',
59+
value: models.inpainting.MIGAN.FACES_256.VULKAN_FP16,
60+
disabled: Platform.OS !== 'android',
61+
},
62+
];
63+
64+
/** Brush diameters in image pixels. */
65+
const BRUSH_SIZES = [16, 32, 64] as const;
66+
67+
function InpaintingContent() {
68+
const insets = useSafeAreaInsets();
69+
const [selectedModel, setSelectedModel] = useState<any>(MODEL_OPTIONS[0].value);
70+
const [imageUri, setImageUri] = useState<string | null>(null);
71+
const [strokes, setStrokes] = useState<readonly MaskStroke[]>([]);
72+
const [brushWidth, setBrushWidth] = useState<number>(BRUSH_SIZES[1]);
73+
const [isProcessing, setIsProcessing] = useState(false);
74+
const [resultImage, setResultImage] = useState<SkiaImageType | null>(null);
75+
const [showOriginal, setShowOriginal] = useState(false);
76+
const [latency, setLatency] = useState<number | null>(null);
77+
const [error, setError] = useState<string | null>(null);
78+
79+
const skiaImage = useImage(imageUri, (err) => setError(err.message || String(err)));
80+
81+
const {
82+
isReady,
83+
downloadProgress,
84+
error: loadError,
85+
inpaint,
86+
inpaintWorklet,
87+
} = useInpainter(selectedModel);
88+
89+
const resetResult = () => {
90+
setResultImage(null);
91+
setShowOriginal(false);
92+
setLatency(null);
93+
};
94+
95+
const handlePickImage = async (useCamera: boolean) => {
96+
setError(null);
97+
try {
98+
const uri = await getImage(useCamera);
99+
if (uri) {
100+
setImageUri(uri);
101+
setStrokes([]);
102+
resetResult();
103+
}
104+
} catch (e: any) {
105+
setError(e.message || String(e));
106+
}
107+
};
108+
109+
const runInpainting = async (sync: boolean) => {
110+
if (!skiaImage || !inpaint || !inpaintWorklet) return;
111+
if (!sync) setIsProcessing(true);
112+
setError(null);
113+
try {
114+
const image = skImageToBuffer(skiaImage);
115+
const mask = rasterizeMask(strokes, image.width, image.height);
116+
if (!mask) throw new Error('Could not rasterize the mask');
117+
118+
const start = Date.now();
119+
const output = sync ? inpaintWorklet(image, mask) : await inpaint(image, mask);
120+
setLatency(Date.now() - start);
121+
122+
const info = {
123+
width: output.width,
124+
height: output.height,
125+
colorType: ColorType.RGBA_8888,
126+
alphaType: AlphaType.Premul,
127+
};
128+
const result = Skia.Image.MakeImage(info, Skia.Data.fromBytes(output.data), output.width * 4);
129+
if (!result) throw new Error('Failed to decode the inpainted image in Skia');
130+
setResultImage(result);
131+
setShowOriginal(false);
132+
} catch (e: any) {
133+
setError(e.message || String(e));
134+
} finally {
135+
if (!sync) setIsProcessing(false);
136+
}
137+
};
138+
139+
const activeError = loadError ? String(loadError) : error;
140+
const showingResult = resultImage !== null && !showOriginal;
141+
const activeImage = showingResult ? resultImage : skiaImage;
142+
const canRun = !!skiaImage && isReady && !isProcessing && strokes.length > 0;
143+
144+
return (
145+
<ScrollView
146+
style={commonStyles.container}
147+
contentContainerStyle={[
148+
commonStyles.contentContainer,
149+
{ paddingBottom: insets.bottom + theme.spacing.large },
150+
]}
151+
>
152+
<Text style={commonStyles.description}>
153+
Pick a photo, paint over what should disappear, and run the model to fill it in.
154+
</Text>
155+
156+
<ModelPicker
157+
label="Inpainting Model"
158+
options={MODEL_OPTIONS}
159+
selectedValue={selectedModel}
160+
onValueChange={(model) => {
161+
setSelectedModel(model);
162+
resetResult();
163+
setError(null);
164+
}}
165+
/>
166+
167+
<ModelStatus
168+
isReady={isReady}
169+
downloadProgress={downloadProgress}
170+
error={activeError}
171+
modelTypeLabel="inpainting model"
172+
/>
173+
174+
<ImageViewport skiaImage={activeImage} onPressPlaceholder={() => handlePickImage(false)}>
175+
{skiaImage && !showingResult && (
176+
<MaskCanvas
177+
image={skiaImage}
178+
strokes={strokes}
179+
onStrokesChange={setStrokes}
180+
brushWidth={brushWidth}
181+
enabled={!isProcessing}
182+
/>
183+
)}
184+
</ImageViewport>
185+
186+
{skiaImage && !showingResult && (
187+
<View style={styles.brushRow}>
188+
<Text style={styles.brushLabel}>Brush</Text>
189+
{BRUSH_SIZES.map((size) => (
190+
<TouchableOpacity
191+
key={size}
192+
style={[styles.brushChip, brushWidth === size && styles.brushChipActive]}
193+
onPress={() => setBrushWidth(size)}
194+
>
195+
<Text style={[styles.brushText, brushWidth === size && styles.brushTextActive]}>
196+
{size}px
197+
</Text>
198+
</TouchableOpacity>
199+
))}
200+
<TouchableOpacity
201+
style={styles.brushChip}
202+
onPress={() => {
203+
setStrokes([]);
204+
resetResult();
205+
}}
206+
disabled={strokes.length === 0}
207+
>
208+
<Text style={styles.brushText}>Clear</Text>
209+
</TouchableOpacity>
210+
</View>
211+
)}
212+
213+
<View style={commonStyles.buttonRow}>
214+
<Button title="Gallery" onPress={() => handlePickImage(false)} variant="secondary" />
215+
<Button title="Camera" onPress={() => handlePickImage(true)} variant="secondary" />
216+
</View>
217+
218+
<View style={commonStyles.buttonRow}>
219+
<Button
220+
title="Run Async"
221+
onPress={() => runInpainting(false)}
222+
disabled={!canRun}
223+
loading={isProcessing}
224+
/>
225+
<Button
226+
title="Run Sync"
227+
onPress={() => runInpainting(true)}
228+
disabled={!canRun}
229+
variant="accent"
230+
/>
231+
</View>
232+
233+
{resultImage && (
234+
<View style={commonStyles.buttonRow}>
235+
<Button
236+
title={showOriginal ? 'Show Result' : 'Show Original'}
237+
onPress={() => setShowOriginal(!showOriginal)}
238+
variant="secondary"
239+
/>
240+
</View>
241+
)}
242+
243+
<LatencyIndicator latency={latency} />
244+
</ScrollView>
245+
);
246+
}
247+
248+
export default function InpaintingScreen() {
249+
return (
250+
<ScreenWrapper>
251+
<InpaintingContent />
252+
</ScreenWrapper>
253+
);
254+
}
255+
256+
const styles = StyleSheet.create({
257+
brushRow: {
258+
flexDirection: 'row',
259+
alignItems: 'center',
260+
gap: 8,
261+
marginBottom: 20,
262+
},
263+
brushLabel: {
264+
color: theme.colors.textMuted,
265+
marginRight: 4,
266+
},
267+
brushChip: {
268+
borderWidth: 1,
269+
borderColor: theme.colors.border,
270+
borderRadius: 16,
271+
paddingVertical: 6,
272+
paddingHorizontal: 12,
273+
},
274+
brushChipActive: {
275+
backgroundColor: theme.colors.primary,
276+
borderColor: theme.colors.primary,
277+
},
278+
brushText: {
279+
color: theme.colors.textMuted,
280+
fontSize: 13,
281+
},
282+
brushTextActive: {
283+
color: '#fff',
284+
},
285+
});

0 commit comments

Comments
 (0)