Skip to content

Add MPMask.getAsImageBitmap() for Web segmentation - #6343

Open
a2105z wants to merge 1 commit into
google-ai-edge:masterfrom
a2105z:mpmask-get-as-image-bitmap
Open

Add MPMask.getAsImageBitmap() for Web segmentation#6343
a2105z wants to merge 1 commit into
google-ai-edge:masterfrom
a2105z:mpmask-get-as-image-bitmap

Conversation

@a2105z

@a2105z a2105z commented Aug 20, 2026

Copy link
Copy Markdown

Summary

MPImage already has getAsImageBitmap(). MPMask (confidence / category masks from ImageSegmenter, Interactive Segmenter, PoseLandmarker) did not. Segmentation effects on Insertable Streams / BreakoutBox / Web Workers want:

new VideoFrame(result.confidenceMasks[0].getAsImageBitmap(), { timestamp: frame.timestamp });

Workarounds today are either getAsFloat32Array() (GPU→CPU readPixels) or poking mask.canvas['GLctxObject']. Schmidt noted this was already on the internal list in 2023.

Fixes #4798

API

mask.hasImageBitmap(): boolean
mask.getAsImageBitmap(): ImageBitmap  // synchronous
  • GPU path: sample the RED channel, write grayscale RGBA vec4(m, m, m, m) to the OffscreenCanvas, transferToImageBitmap() — same sync GPU path as MPImage, no float readback.
  • CPU path: upload once to a texture, then the same transfer.
  • Requires the task to be created with an OffscreenCanvas (same limitation / error as MPImage.getAsImageBitmap()).
  • Cached on the MPMask; close() frees an owned bitmap.

Test plan

  • mask.test.ts: hasImageBitmap() after conversion; grayscale pixels from Uint8Array and WebGLTexture; cache identity; clone owns a distinct bitmap; throws without a canvas
  • bazel test //mediapipe/tasks/web/vision/core:mask_test (browser; skipped in Node like the existing mask tests)

@schmidt-sebastian @kuaashish @HarishPermal @tyrmullen @kalyan2789g @akashvverma1995 @ayushgdev @lu-wang-g

Insertable Streams / BreakoutBox need a GPU mask as ImageBitmap for
VideoFrame without a Float32Array readback. Mirror MPImage: draw the
RED channel as grayscale RGBA on an OffscreenCanvas and
transferToImageBitmap().

Fixes google-ai-edge#4798
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add mask.getAsImageBitmap() method to vision tools like image segmentation

1 participant