animation support for IIP

[jerch]

IIP supports the following image formats: PNG, JPEG, GIF, QOI (and new WEBP, AVIF)

Of these formats GIF, PNG, WEBP and AVIF may contain multiple frames for animations. QOI does not know about animations, also JPEG does not support animations (although there exists an M-JPEG derivate).

To support animations from those formats, several obstacles in xterm.js have to be dealt with:

getting the animation frames
This is already a big bummer, as there is no convenient API standardized yet. There is a draft of ImageDecoder in the making, but we cannot rely on that until it gets stable. Also it gives frame access promisified and as a full video frame, not to partial subframes. Both works against xterm.js logic, where a rAF should be sync code and the RAM is precious. So maybe it will never be useable for xterm.js.

Alternatively we can do things manually with custom format parsers:

• GIF is relatively easy to handle (see omggif)
• APNG can be done with a simple frame trick and letting the browser do the heavy lifting (apng-js)
• WEBP and AVIF are on a different scale and most likely too heavy for JS/wasm lifting

image addon storage
Animation frames create a big memory pressure on the addon storage. Additionally the frame metainfo needs to be preserved (offsets, dimensions, composition type, delay etc).

render logic issues
Image rendering is currently passive, thus an image portion gets only updated from an onRender event from xterm.js itself. With animation support we need a self contained render cycle.

[jerch]

Have tested several ideas with omggif. The library takes its time to parse things. I think a wasm replacement can at least speed it up by a factor of 4 (lots of bit manipulations due to LZW), but it still chokes on bigger GIFs with lots of frames (in 10-100ms). So we cannot do the frame decoding in mainthread, or have to budget the runtime. There is a second issue - GIF is very stateful and might merge frames from previous ones with a local palette, thus an RGBA frame might exceed the 256ish palette, which weighs heavy on RAM consumption (one of my test GIFs explodes from 800kB filesize to 30MB RGBA in RAM).

Both facts indicate, that we should resort to a worker solution for the frame creation and the delivery to mainthread. Basic idea:

• on receive place a still on the terminal (as already done currently)
• if the image contains animations, transfer it to a worker decoding the frames
• frames are either stored as indexed8 or full RGBA in the worker
• when decoding is done the worker:
  • worker notifies mainthread about the animation to start with a fixed timeline
  • worker start transferring frames to mainthread in a pooled ringbuffer with frame IDs
  • mainthread holds only a few frames in the future to be drawn soon, sends older ones back to worker
  • if mainthread lags due to heavy other work, the frame pool might starve
    • stops worker automatically
    • mainthread skips through old frames without rendering, thus worker can continue normally
• animations should be stoppable:
  • automatically when out of viewport
  • by some sort of a stop action

This setup has the advantage that an early still will be shown, while the heavy lifting happens in a worker, which has its own RAM budget. Furthermore it can be used for ImageDecoder as well, thus would give us direct support for other animation formats while keeping the image rendering synchronous (all browsers except Safari).

[jerch]

First PoC:

(Note: The tiny stuttering is an artifact of the screen recorder from a close drawing to sampling frequency.)

[jerch]

Turns out that ImageDecoder is very convenient to use, as it automatically gives us access to all image formats the browser engine supports.

[jerch]

Well seems Safari is getting ImageDecoder support soon. commit history

[jerch]

Have a working PoC with ImageDecoder built. It is astonishingly snappy (hardly get it over 20% CPU time), even for resuming decoding from scratch, which gives us the chance to cleanup animated images to a single RGBA picture when scrolled out. Proper eviction handling, font resizing and optimized animation screen update are still missing, but the results are already impressive on my oldish Haswell laptop.

It already uses blobs as backing storage to have low impact on precious JS heap. That is a detail I also want to bring to ImageStorage for scrolled out images in general as a step in eviction strategy. The current FIFO cap (default 128MB) evicts images hard, which gives poor user experience for images further up in the scroll buffer.

With this and the fact that Safari is working on ImageDecoder support my thoughts above get obsolete. Writing our own format handlers with workers is cumbersome while ImageDecoder gives us direct support for all browser supported formats.