Video Super Resolution for Moonlight iOS/tvOS - #704
Open
linckosz wants to merge 1 commit into
Open
Conversation
This commit adds an optional Video Super Resolution path for iOS/tvOS. - Introduce a VideoSuperResolution renderer backed by Metal and Core Image. - Convert decoded YUV frames to RGB and upscale them with MetalFX SpatialScaler. - Preserve the existing rendering path as the default fallback. - Add a user-facing toggle and hide it when MetalFX. - Show "Video Enhancement: MetalFX " in the performance overlay when active. - Run the VSR path on a dedicated queue and cache session resources for better latency. - Preserve HDR/color attachments through the converted output frames.
This was referenced Apr 23, 2026
bartleman
added a commit
to bartleman/moonlight-ios
that referenced
this pull request
Aug 17, 2026
…am#704 Adds an optional MetalFX spatial upscaler to the tvOS video pipeline, ported from upstream PR moonlight-stream#704 with tvOS-specific fixes and hardening. Off by default; when enabled it decodes on a dedicated queue, converts YUV->RGB via Core Image, and MetalFX-upscales the stream to the display size before enqueuing to the display layer. Only engages when the display is larger than the stream (e.g. 1080p -> 4K); at native 4K it is bypassed. tvOS integration the PR omitted / got wrong: - Add the "Video Super Resolution (MetalFX)" toggle to Moonlight TV Settings.bundle (key videoSuperResolution, default off). Without this the feature was permanently off on Apple TV as shipped. - Skip the PR's iOS-only SettingsViewController + storyboard changes. Hardening over the raw PR: - +[VideoSuperResolution isDeviceSupported] gates the feature on actual MetalFX MTLFXSpatialScaler support so unsupported GPUs fall back to the normal pipeline instead of dropping every frame (black screen). - Disable VSR for the session when the display is not larger than the stream, or when scaler configuration fails, instead of running a pointless second decode/convert or dropping frames. - Runtime per-frame fallback: after repeated failures, revert to direct enqueue and request a fresh IDR (avoids mixing decoded-RGB and mid-GOP compressed samples on one layer). - Wait for GPU completion before handing the upscaled buffer to the display layer (runs on the VSR queue, not main, so no input-latency cost) to fix a tearing / pixel-buffer-pool recycle race. - Make the HDR reconfigure path serialize on the VSR queue without a queue->main->queue deadlock against stop(). MetalFX auto weak-links (introduced tvOS 16 > deployment target), so no extra linker flags are needed. Private-API-free; fork/self-signed only is not a concern here (this is standard framework usage). Claude-Session: https://claude.ai/code/session_01QiVtyqxowFKZc8gKnNjevy
bartleman
added a commit
to bartleman/moonlight-ios
that referenced
this pull request
Aug 17, 2026
…am#704 Adds an optional MetalFX spatial upscaler to the tvOS video pipeline, ported from upstream PR moonlight-stream#704 with tvOS-specific fixes and hardening. Off by default; when enabled it decodes on a dedicated queue, converts YUV->RGB via Core Image, and MetalFX-upscales the stream to the display size before enqueuing to the display layer. Only engages when the display is larger than the stream (e.g. 1080p -> 4K); at native 4K it is bypassed. tvOS integration the PR omitted / got wrong: - Add the "Video Super Resolution (MetalFX)" toggle to Moonlight TV Settings.bundle (key videoSuperResolution, default off). Without this the feature was permanently off on Apple TV as shipped. - Skip the PR's iOS-only SettingsViewController + storyboard changes. Hardening over the raw PR: - +[VideoSuperResolution isDeviceSupported] gates the feature on actual MetalFX MTLFXSpatialScaler support so unsupported GPUs fall back to the normal pipeline instead of dropping every frame (black screen). - Disable VSR for the session when the display is not larger than the stream, or when scaler configuration fails, instead of running a pointless second decode/convert or dropping frames. - Runtime per-frame fallback: after repeated failures, revert to direct enqueue and request a fresh IDR (avoids mixing decoded-RGB and mid-GOP compressed samples on one layer). - Wait for GPU completion before handing the upscaled buffer to the display layer (runs on the VSR queue, not main, so no input-latency cost) to fix a tearing / pixel-buffer-pool recycle race. - Make the HDR reconfigure path serialize on the VSR queue without a queue->main->queue deadlock against stop(). MetalFX auto weak-links (introduced tvOS 16 > deployment target), so no extra linker flags are needed. Private-API-free; fork/self-signed only is not a concern here (this is standard framework usage).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Video Super Resolution (VSR) brings to video what DLSS/FSR bring to 3D rendering.
This PR introduces high-quality spatial upscaling on iOS/tvOS using Metal, Core Image, and MetalFX SpatialScaler.
Mobile devices and Apple TVs can benefit from receiving a lower base stream resolution while reconstructing a sharper final image locally on-device.
This helps reduce bandwidth pressure and decoder workload while preserving visual quality at display resolution.
The Apple ecosystem requires an upscaling solution that is:
MetalFX SpatialScaler is a strong fit for these constraints.
It provides a hardware-accelerated spatial upscaling path designed for Apple GPUs, with good quality and limited CPU overhead.
In this implementation, the decoded frame is first converted from YUV to RGB on the GPU, then optionally upscaled to the display resolution before being submitted to the existing display pipeline.
The observed power impact was very small, around +20%.
On an iPhone 13 Pro, during a 720p60 streaming session, battery usage was about 15% without VSR and about 18% with VSR enabled.
Result
400% Zoom:
Original:
Video Super Resolution:
Existing VSR Implementations
Video Super Resolution is also available in my other Moonlight ports:
Windows, macOS, Linux: Add Video Super Resolution for Windows x64 (AMD, Intel and NVIDIA), Windows ARM (Snapdragon), Linux and MacOS moonlight-qt#1557
Android: Video Super Resolution for Moonlight iOS/tvOS #704
Xbox: Add Video Super Resolution moonlight-android#1567