Fix H.265 decoder crushing 10-bit output and discarding bitstream colorimetry (HDR10 renders washed out) - #267
Conversation
…orimetry The VideoToolbox decompression session pinned its output to 8-bit NV12, and overrideColorSpaceAttachments() unconditionally stamped BT.709/sRGB attachments on every frame. Together these break HEVC Main10/HDR10 reception: 10-bit content is bit-crushed and PQ/BT.2020 colorimetry from the bitstream VUI is replaced, so HDR streams render washed out. Request a 10-bit biplanar output format when the hvcC configuration record signals a luma bit depth above 8, and only apply the BT.709/sRGB fallback attachments when the format description carries no colour information from the bitstream. 8-bit streams without VUI colour info are byte-for-byte unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-preserving H.265 decoder (#55) * feat(hdr): make HDR streaming work end-to-end HDR never actually reached the screen despite #49's groundwork. Three independent gaps, each verified with on-device session logs: 1. Automatic color mode could never request HDR: colorRequest() was always called with default game/account/server HDR signals, and the automatic gate required all three positively true. Derive the account entitlement from the MES membership tier (Ultimate/Performance -> HDR) and thread it through session create, resume, and connect; the automatic gate now requires confirmed entitlement and is permissive on unknown game/server support (GFN falls back to an SDR encode inside an HDR-capable session for SDR titles). 2. H.265 Main10 was never negotiated: LKRTCDefaultVideoDecoderFactory only advertises Main (profile-id=1), so createAnswer dropped GFN's Main10 payload before SDPMunger.preferCodec(preferTenBit:) could front-load it, and HDR sessions streamed an 8-bit pipe. GFNVideoDecoderFactory additionally advertises profile-id=2. 3. The bundled LiveKit H.265 decoder pins its VideoToolbox output to 8-bit NV12 and force-stamps BT.709/sRGB attachments on every frame, crushing HDR10 to washed-out SDR. GFNVideoDecoderH265 decodes via VideoToolbox without the pixel-format pin and propagates the bitstream's VUI colorimetry (PQ/BT.2020); an upstream fix will be proposed to webrtc-sdk/webrtc so this class can eventually be removed. Also teach DecodedVideoFormatInspector Apple's packed 10-bit 'p420' format, which VideoToolbox emits as its native 10-bit HEVC output. Verified on Apple TV 4K: Automatic requests hdr10, the answer carries Main10 first, and enabling HDR in-game yields decoded frames with SMPTE_ST_2084_PQ / ITU_R_2020 attachments and a correct HDR picture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(hdr): reference upstream decoder fix webrtc-sdk/webrtc#267 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
| // primaries) already have correct attachments propagated by VideoToolbox | ||
| // from the format description; overriding them mistags the frames as | ||
| // BT.709/sRGB and HDR content renders washed out. | ||
| if (![decoder bitstreamCarriesColorInfo]) { |
There was a problem hiding this comment.
This runs on VideoToolbox's async callback thread and reads _videoFormat, which decodeData: releases (via setVideoFormat:) before destroyDecompressionSession waits for in-flight frames — a use-after-free window on every mid-stream format change; consider capturing this as a bool in RTCH265FrameDecodeParams at decode time instead.
There was a problem hiding this comment.
Good catch — the callback could indeed read _videoFormat in the window where decodeData: had already released the old format while the previous session's frames were still in flight (and even after the swap, in-flight frames would be judged against the wrong format's colour info). Fixed as you suggested in d4af43d: the flag is now captured in RTCH265FrameDecodeParams on the decode thread, so the callback no longer touches decoder state and each frame is judged against the format it was actually decoded with.
|
Thanks — verified this end-to-end with a Main10/PQ stream and the colorimetry + 10-bit output work as described. One question on rollout: since x420 output breaks consumers that assume 8-bit NV12 ( |
…ate from the VideoToolbox callback The decompression output callback runs on VideoToolbox's async callback thread and read _videoFormat via bitstreamCarriesColorInfo. decodeData: releases _videoFormat (through setVideoFormat:) on a mid-stream format change before destroyDecompressionSession waits for in-flight frames, leaving a use-after-free window; frames from the old session could also be tagged against the new format's colour info. Capture the flag in RTCH265FrameDecodeParams on the decode thread at decode time, so the callback never touches decoder state and each frame is judged against the format it was actually decoded with. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Emitting kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange breaks consumers that assume 8-bit NV12 output: RTCMTLVideoView's R8/RG8 shaders render black and RTCCVPixelBuffer's I420 conversion paths hit RTC_DCHECK_NOTREACHED. Keep the historical NV12 downconversion as the default and add RTCVideoDecoderH265.preferHighBitDepthOutput so apps whose render path handles 10-bit biplanar buffers can opt in. The colorimetry fix is unaffected and stays on for everyone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for verifying end-to-end! Agreed on the rollout concern — ff6af96 keeps 8-bit NV12 downconversion as the default and gates the x420 output behind an opt-in class property, I went with a class property over a field trial because it's the easiest to reach for apps consuming the prebuilt xcframework — happy to switch if you'd prefer the field-trial route. For the lossy 10→8 fallback in |
2 participants
Problem
RTCVideoDecoderH265breaks HEVC Main10 / HDR10 reception in two ways:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, so 10-bit output is crushed to 8 bits.overrideColorSpaceAttachments()runs unconditionally on every decoded frame, replacing genuine colorimetry from the bitstream VUI (e.g. SMPTE ST 2084 PQ transfer + BT.2020 primaries) with BT.709/sRGB.Together, any HDR10 stream (HEVC Main10 with PQ/BT.2020 VUI) decodes as washed-out 8-bit SDR: bit depth is lost at the decoder boundary, and downstream renderers see frames mistagged as sRGB/709.
Fix
kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange) when the hvcC decoder configuration record signalsbit_depth_luma_minus8 > 0; 8-bit streams keep NV12 exactly as before.Compatibility
x420output with true colorimetry. Consumers rendering viaAVSampleBufferDisplayLayerdisplay HDR correctly;RTCMTLVideoView's shaders may need 10-bit support to benefit, but Main10 content was already colour-broken before this change. Happy to gate the pixel-format change further if you'd prefer.Testing
The behaviour change was verified on-device in a receive-only tvOS WebRTC client with 4K HEVC streams: Main10/HDR10 streams now decode as 10-bit with SMPTE ST 2084 / BT.2020 attachments and render correctly, and 8-bit streams are unaffected. I could not build the full webrtc tree locally, so this patch relies on CI for compile verification — happy to iterate on review feedback.
🤖 Generated with Claude Code