Skip to content

Parallelize build-time shader compilation in the descriptor generators - #932

Draft
rickbrew wants to merge 1 commit into
Sergio0694:mainfrom
rickbrew:feature/parallel-shader-compilation
Draft

Parallelize build-time shader compilation in the descriptor generators#932
rickbrew wants to merge 1 commit into
Sergio0694:mainfrom
rickbrew:feature/parallel-shader-compilation

Conversation

@rickbrew

@rickbrew rickbrew commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

via Claude. I have not yet reviewed this code yet.

Closes #931

Description

Moves FXC/DXC compilation out of the ForAttributeWithMetadataName transform callbacks (which the incremental driver invokes sequentially, serializing all shader compilations) and into a dedicated node per generator that compiles every shader in the project in parallel. Applies to both D2DPixelShaderDescriptorGenerator and ComputeShaderDescriptorGenerator; the shared HlslBytecodeSyntaxProcessor carries the new logic for both.

Pipeline shape (identical in both generators):

  1. Transform: unchanged up to building HlslBytecodeInfoKey, but no longer compiles. It also captures the info needed to later synthesize the compile diagnostics (HlslBytecodeDiagnosticsInfo: type name, type location, [D2DRequiresDoublePrecisionSupport] presence + location), since symbols must not escape the transform.
  2. Warm-up node: Select(key).Collect()CompileAllInParallel(keys, token), a Parallel.ForEach over the distinct keys into the existing DynamicCache (which is already thread-safe and needed no changes; the FXC and DXC wrappers are also both safe for concurrent use, DXC via its existing [ThreadStatic] instance).
  3. Join node: Combine with the warm-up node's output (this edge is what guarantees the ordering, via the dataflow graph rather than driver scheduling assumptions), then a Select that fetches the bytecode (a guaranteed cache hit), synthesizes the deferred diagnostics, and drops the captured diagnostics info from the model.

Locations are captured by value in the new LocationInfo model (file path + TextSpan + LinePositionSpan, materialized via Location.Create(path, span, lineSpan)), deliberately not as a SyntaxTree reference: this keeps transform outputs equatable across unrelated edits (comment-only edits still produce value-equal models), and avoids ever holding stale trees in cached models. All existing incrementality tests pass unchanged.

Numbers from Paint.NET (warm --no-incremental Release x64 rebuilds, second of two runs, vs. stock 3.2.0):

Project serial parallel delta
Effects.Gpu (51 shaders) 21.4s 13.5s -7.9s (-37%)
Windows.Direct2D1.Managed 12.6s 9.0s -3.6s (-29%)
Windows.Framework (546 shaders) 11.2s 7.6s -3.6s (-32%)

API breakdown

No public API changes — internal generator restructuring only.

Additional context

  • Generated output verified byte-for-byte identical to 3.2.0 across Paint.NET's ~660 shaders (329 emitted .g.cs files compared, generator version stamp aside), and the full Paint.NET solution builds clean against a locally packed build of this branch.
  • ComputeSharp.D2D1.Tests.SourceGenerators (113) and ComputeSharp.Tests.SourceGenerators (110) all pass. The WinUI source generator test project currently fails to restore for me on stock main as well (NU1603 warnings-as-errors from feed version drift), so I couldn't run that suite locally — but it only exercises CanvasEffectPropertyGenerator, which this PR doesn't touch.
  • D2DPixelShaderSourceGenerator (raw HLSL string literals) intentionally keeps compiling inline; it can get the same treatment in a follow-up if you want it.
  • As mentioned in Parallelize build-time shader compilation in the descriptor generators #931, I'd like to follow up separately with an opt-in persistent on-disk bytecode cache proposal, which composes with this change (parallelism helps the first compile; a disk cache would make unchanged CLI rebuilds nearly free).

🤖 Generated with Claude Code

Move FXC/DXC compilation out of the ForAttributeWithMetadataName transform
callbacks (which the incremental driver invokes sequentially) into a dedicated
node that warms the shared bytecode cache for all shaders in parallel. A join
node (ordered after the warm-up by its dataflow edge) then folds each compiled
shader back into its model and synthesizes the compile diagnostics, using
location info captured by value during the transform.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rickbrew
rickbrew marked this pull request as draft August 31, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelize build-time shader compilation in the descriptor generators

1 participant