[borys3kk]

Description

Adds image inpainting on MI-GAN: paint a mask over a photo and the model fills the region with plausible content (object removal, cleanup).

• Native cv op: blendMasked (masked composite, with optional erosion and inward feathering of the mask; the presets use neither, since the upsampled fill is already soft), with host C++ tests.
• Task + hook: createInpainter / useInpainter. The task resizes the whole frame to the model, runs forward, resizes the output back and blends it into the full-resolution frame through the mask. Cropping around the hole is left to the caller. The public mask marks the hole with any non-zero pixel; the inversion to the graph's 1 = known convention happens inside the task. An empty mask returns the input unchanged.
• Registry: models.inpainting.MIGAN with SIZE_512, SIZE_256 (Places2) and FACES_256 (FFHQ), fp16 only, on XNNPACK, Core ML and Vulkan. New inpainting feature in the native libs map.
• Docs: new Image Inpainting page; camera integration renumbered to 12.
• Example app: Image Inpainting screen with a brush-painted mask (MaskCanvas), result/original toggle. app.json gains NSLocalNetworkUsageDescription so a dev build on a physical iPhone can reach Metro and local files.

The exports come from export-scripts MR !31 and are published to software-mansion/react-native-executorch-migan at v0.11.0. Native libs are pinned to 0.10.7, which the XNNPACK exports need.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

1. apps/computer-vision → Image Inpainting.
2. Pick a photo, paint over an object with the brush (cover it fully, with a margin), Run Async / Run Sync.
3. Toggle Show Original to compare.

Library: yarn workspace react-native-executorch test (12 new tests in inpainter.test.ts), scripts/run-native-tests.sh -R CvOpsTest (needs OpenCV 4; Homebrew's opencv is 5.x and does not link against the vendored 4.x headers, use opencv@4 with OpenCV_DIR).

Measured with the published v0.11.0 files and native libs 0.10.7, forward only, medians after warm-up:

Device	Model	Backend	forward
iPhone 17	MI-GAN 512	Core ML	29 ms
iPhone 17	MI-GAN 512	XNNPACK	81 ms
iPhone 17	MI-GAN 256	Core ML	11 ms
iPhone 17	MI-GAN 256	XNNPACK	47 ms
Galaxy S23	MI-GAN 512	Vulkan	175 ms
Galaxy S23	MI-GAN 512	XNNPACK	194 ms
Galaxy S23	MI-GAN 256	Vulkan	101 ms
Galaxy S23	MI-GAN 256	XNNPACK	116 ms

Faces 256 runs at the same speed as 256. Whole inpaint() with MI-GAN 512 on a 708x424 photo: ~35 ms Core ML / ~89 ms XNNPACK on the iPhone 17, ~200-260 ms Vulkan / ~260-300 ms XNNPACK on the S23. At 12 MP the full-resolution resize and blend dominate: 100-250 ms on the iPhone, 0.4-1.6 s on the S23 with a large run-to-run spread.

End-to-end on both devices (all six variants per platform, eight real photos incl. two at 12 MP, synthetic edge cases down to 1x1 and a single-pixel hole, RGB vs RGBA input, sync vs async, two calls in flight, dispose): pixels outside the mask come back unchanged on every case, outputs are deterministic, and inside the mask they are within 1-9 levels (mean) of upstream's reference results on the places2_512 examples. Vulkan and XNNPACK outputs on the S23 differ by about one level on average (fp16 on GPU vs CPU). With upstream's sheep_mask.png example the output matches upstream's sheep_out.png.

Screenshots

(to be added)

Related issues

#1373

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

• A mask that cuts through an object makes MI-GAN extend that object into the hole rather than remove it; the docs carry a tip about covering the whole object.
• @shopify/react-native-skia deprecates SkPath.moveTo/lineTo; the mask component uses PathBuilder.
• Pre-commit hooks locally need cspell on PATH (node_modules/.bin) and all workspaces built (yarn prepare), otherwise spell-check-other and types fail for reasons unrelated to the change.

[msluszniak]

@borys3kk I uploaded the models on HF: https://huggingface.co/software-mansion/react-native-executorch-migan/tree/main, please wire them and check if everything works correctly and the speed you measured is the same. Also please update PR description.

[barhanc]

Haven't tested the app yet, will do that first thing tomorrow.

Edit: Oh and please fix the cspell warnings.

[borys3kk]

Addressed the review in five commits (3120d76 … 28b136d), see the inline replies. On top of that, an end-to-end run on an iPhone 17 against the published HF models (all six iOS variants, eight real photos incl. two 12 MP, plus synthetic edge cases) turned up two problems in the blend, both fixed in 3120d76:

• Strokes thinner than 3 px were silently dropped. blendOpts.shrink: 1 eroded the hole with a 3x3 kernel, so a 1–2 px mask eroded to nothing and inpaint() returned the input unchanged. With the full-frame pipeline the fill is upsampled and its edge already soft, so the presets now use shrink: 0, and blendMasked falls back to the un-eroded region when erosion would empty it.
• The feather leaked about 1 px outside the mask (up to 23 levels), while the docs promise pixels outside the mask come back unchanged. The weight is now zeroed outside the mask, so the ramp runs inward only. Verified on device: 0 pixels differ outside the mask on every case, down to a single-pixel hole.

Also in the docs: an empty mask returns the input unchanged but still runs the model (the early return went away with the crop).

Whole inpaint() on the iPhone 17 with the HF v0.11.0 files: ~89 ms XNNPACK / ~35 ms Core ML on a 708 px photo (forward alone 79 / 29 ms for MI-GAN 512); on 12 MP 150–250 ms / 100–150 ms, dominated by the full-resolution resize and blend. Inside the mask the output is within 1–9 levels (mean) of upstream's reference results on the places2_512 examples.

[msluszniak]

PR description: inpainter.test.ts has 12 tests now (it says 17), and the end-to-end numbers from your comment could go there too.

[borys3kk]

PR description updated: 12 tests, and the forward table re-measured on both devices with the published v0.11.0 files (the old rows were the pre-release exports), plus the end-to-end numbers. The S23 went through the same e2e set today as the iPhone yesterday, all six Android variants, zero failures, 0 pixels changed outside the mask on every case. Also fixed the cspell warning (a4b1d60).