Where generation time actually goes on the reference card, what was tried to reduce it, and how the numbers translate to other hardware. Every measured figure here comes from the worker's own engine on the reference box, not from vendor claims or literature.
Companion documents: third-party-models.md for licensing, scripts/BENCHMARK.md for the harness, decisions.md for the decisions these measurements informed.
| Property | Value |
|---|---|
| Model | AMD Radeon RX 7600 XT |
| Architecture | RDNA 3, Navi 33, gfx1102 |
| Compute units | 32 CUs (16 WGPs), 2048 stream processors |
| Boost clock | 2755 MHz rated, 2520 MHz observed under sustained load |
| FP32 | ~11.3 TFLOPS peak |
| FP16 | ~22.6 TFLOPS peak (RDNA 3 dual-issue) |
| VRAM | 16 GB GDDR6, 128-bit bus, 18 Gbps, ~288 GB/s |
| Host link | PCIe 4.0 x8 |
| Board power | 165 W |
| Stack | torch 2.9.1+rocm6.3, diffusers 0.39.0 |
Two caveats worth knowing before reading any number below.
torch.cuda.get_device_properties().multi_processor_count reports 16 on
this card. That is WGPs, not CUs. The card has 32 CUs. Do not use the torch
figure for throughput math.
Usable VRAM is not 16 GB. The card reports 15.98 GiB total and the desktop session holds 1.2 to 2.0 GiB of it, so roughly 14.3 GiB is available to a worker. Several conclusions below turn on that difference.
RDNA 3 has WMMA instructions but nothing equivalent to the dedicated tensor cores in NVIDIA's datacenter parts. The FP16 figure above is shader throughput, which is why it sits an order of magnitude below an A100's quoted tensor number and why those two numbers must never be compared directly.
Every text-to-image model at the highest steps and largest resolution its
manifest allows, across the 60-prompt suite in
scripts/benchmark-prompts-60.json. 600 images, 600 successes, no failures.
Measured 2026-07-27 on the reference card through the API and worker, not
through the engine directly, so these include the real dispatch path.
| Model | Res | Steps | Median | Min | Max | Load |
|---|---|---|---|---|---|---|
| sd-turbo | 512 | 4 | 0.43 s | 0.43 s | 0.64 s | 5.0 s |
| sdxl-turbo | 512 | 4 | 0.64 s | 0.64 s | 0.85 s | 15.1 s |
| vega-rt | 1024 | 8 | 1.96 s | 1.91 s | 2.37 s | 6.3 s |
| ssd-1b-lightning | 1024 | 8 | 2.61 s | 2.58 s | 2.90 s | 8.3 s |
| sdxl-hypersd | 1024 | 8 | 3.77 s | 3.73 s | 4.08 s | 11.1 s |
| sdxl-fast | 1024 | 8 | 3.77 s | 3.74 s | 4.08 s | 10.4 s |
| dreamshaper-lcm | 768 | 15 | 5.57 s | 5.56 s | 6.05 s | 4.6 s |
| ssd-1b | 1024 | 40 | 18.55 s | 18.20 s | 18.61 s | 9.8 s |
| sdxl-base | 1024 | 50 | 37.34 s | 37.23 s | 37.68 s | 15.1 s |
| sd35-medium | 1024 | 50 | 110.94 s | 110.52 s | 119.08 s | 2.2 s |
This is a ceiling comparison, not a defaults comparison. Several of these
models ship far lower defaults because their distillation does not benefit
from more steps: vega-rt runs 2 steps at 512 in the studio, not 8 at 1024.
For the numbers the studio picker actually shows, see
backend/app/model_timings.json.
Read across the table and the spread is 258x, from sd-turbo at 0.43 s to
sd35-medium at 110.94 s. The distilled 8-step models cluster tightly around
2 to 4 seconds and are the practical working tier. sdxl-fast and
sdxl-hypersd are within 0.01 s of each other, which is worth remembering
given sdxl-hypersd remains benchmark-only: there is nothing to gain by
promoting it. This measurement is the reason, not licensing; the Hyper-SD SDXL
LoRA is CreativeML Open RAIL++-M (see
third-party-models.md).
sd35-medium is 3x its own 20-step default for a quality difference that did
not survive inspection at a fixed seed, which is why the manifest defaults to
20. Its load time of 2.2 s looks anomalous next to sdxl-base at 15.1 s only
because the offload rung leaves the weights on the host: it never pays a full
transfer to VRAM at load, it pays it during every generation instead.
All figures at 1024x1024, 20 steps, fp16, warm (second run discarded the
first), sd35-medium, prompt fixed and seed fixed.
| Configuration | Time | Peak VRAM | Note |
|---|---|---|---|
model_offload, T5 resident (previous) |
49.5 s | 12.09 GB | The previous 16 GB configuration |
group_offload |
46.5 s | 2.51 GB | Faster and 4.8x lighter |
| 768 px instead of 1024 | 27.8 s | 12.09 GB | 1.78x fewer pixels, 1.78x less time |
| Flash attention backend | 49.9 s | 12.09 GB | No effect; fused attention was already on |
No T5, full residency |
40.9 s | 8.83 GB | Loses the long-prompt window |
No T5, full + torch.compile |
26.0 s | 8.84 GB | Fastest measured, but no T5 |
full residency with T5 |
OOM | - | 15.15 GB of weights, 14.3 GiB available |
Step scaling on the previous configuration is roughly a fixed floor plus a per-step cost: 56 s at 20 steps, 65 s at 28, 89 s at 40 on first-run timings, which works out near 1.7 s per step over a fixed overhead in the low twenties of seconds. The per-step cost is transformer compute and the floor is text encoding, component transfers and VAE decode.
Guidance above 1.0 enables classifier-free guidance, which runs the
transformer on a doubled batch and therefore roughly doubles per-step cost.
sd35-medium defaults to guidance 4.5 because it needs CFG for quality, so
that doubling is paid on every step. This is not a defect to optimize away;
it is the cost of the quality the model is selected for.
Generation time on this card decomposes cleanly. For sdxl-base at 1024 px,
measured by running the same pipeline at 10 and 20 steps and solving for the
constant, then repeating with output_type="latent" to remove the decode:
total = 717 ms fixed + steps x 750 ms
| |
| +-- 2 UNet evaluations at ~375 ms (CFG doubles it)
+-- 667 ms VAE decode + 50 ms text encode and setup
Both halves were verified against the measurements they predict: 10 steps gives 8218 ms and 20 gives 15719 ms, and the fit reproduces them exactly. Step scaling is perfectly linear from 10 to 50 steps.
Three consequences worth internalising.
The fixed cost is almost entirely VAE decode. 667 ms of the 717 ms floor
is turning the final latent into pixels, and it is paid once per image no
matter how many steps ran. Text encoding is 50 ms, which is noise. For a
50-step render the floor is 2% of runtime and irrelevant; for vega-rt at
1.87 s it is 36% of the total, and on the fast tier it is the single
largest optimisation target left. AutoencoderTiny (TAESD) decodes in
roughly 10 ms instead of 667 ms at some fidelity cost, and the deferred
realtime ladder in decisions.md already anticipates exactly
this ("tiny-autoencoder decode for the live preview with full VAE on refine").
This measurement is the quantitative case for it.
Guidance above 1.0 doubles per-step cost. Diffusers sets
do_classifier_free_guidance = guidance_scale > 1.0, so every step runs the
UNet on a batch of two, conditional and unconditional. That is why the roster
spans 258x: sdxl-fast at 8 steps with guidance 0 runs 8 UNet evaluations,
sdxl-base at 50 steps with guidance 6 runs 100, on the same 3.5B UNet.
Distilled models set guidance 0 and cap it at 2 in the manifest precisely
because their distillation bakes the guidance effect into the weights;
applying CFG on top oversaturates.
Steps are linear and buy less than they cost. See the sweep below.
sdxl-base and ssd-1b already ship dpmsolver (DPM++ 2M Karras), a
fast-converging solver. The question is therefore not which scheduler but how
few steps it needs. Same prompt, same seed 100, 1024 px, guidance 6:
| Steps | dpmsolver | euler-trailing | stock Euler |
|---|---|---|---|
| 10 | 8.29 s | - | - |
| 15 | 12.02 s | - | - |
| 20 | 15.76 s | 15.72 s | 15.70 s |
| 25 | 19.51 s | - | - |
| 30 | 23.27 s | 23.17 s | 23.15 s |
| 40 | 30.66 s | - | - |
| 50 | 38.03 s | - | - |
Scheduler choice does not change runtime at matched steps, to within 60 ms. Solvers change the trajectory through latent space, not the arithmetic per step, so any speed argument between them is really an argument about how few steps each needs to converge.
On quality, 20 and 50 steps are effectively indistinguishable at a fixed seed: same composition, same anatomy, marginally cleaner background detail at 50 for 2.4x the time. Even 10 steps produces a sharp, well-formed image. Note that composition shifts with step count because the trajectory differs, so this is not a strict quality ladder, but there is no degradation to point at.
The practical conclusion: the shipped default of 20 is sound and the 50-step
manifest ceiling exists for headroom rather than because anyone should use it.
The roster benchmark ran ceilings deliberately, which is why sdxl-base shows
37.34 s there against 15.76 s at its default.
The obvious way to spend spare VRAM is to generate several images per denoise
loop. Measured with num_images_per_prompt, per-image time and peak VRAM:
| Model | batch 1 | batch 2 | batch 4 | batch 8 |
|---|---|---|---|---|
| sdxl-fast | 3.69 s / 9.44 GB | 3.67 s / 14.22 GB | OOM | - |
| ssd-1b-lightning | 2.55 s / 6.91 GB | 2.50 s / 9.38 GB | 3.15 s / 14.44 GB | OOM |
| vega-rt | 1.87 s / 5.71 GB | 1.83 s / 8.18 GB | 1.82 s / 13.12 GB | OOM |
| sdxl-base | 15.34 s / 9.07 GB | 15.12 s / 11.54 GB | OOM | - |
Best case is a 3% improvement. ssd-1b-lightning at batch 4 is 19% worse
than batch 1. Everything OOMs by batch 4 or 8, because VRAM grows steeply
(9.44 to 14.22 GB going from one image to two on sdxl-fast).
The flat result above is easy to misread as "GPUs cannot process images concurrently". They can, it is how every inference provider operates, and the result here is a property of this card rather than of GPUs.
A GPU runs one kernel across thousands of threads grouped into workgroups, scheduled onto compute units. Throughput depends on occupancy: whether there is enough independent work in flight to keep every CU busy and to hide memory latency behind arithmetic. Batching adds independent work along the batch dimension, so it raises occupancy whenever occupancy is the thing you are short of.
That is the whole story. Batching helps exactly when the GPU is starved, and this card is not starved:
- A 1024 px SDXL latent is 128x128. Every convolution in the UNet already unfolds into millions of independent output elements, which is far more parallel work than 32 CUs can consume at once. The machine is saturated by one image, so a second image simply queues behind the first. Time doubles, per-image time does not improve, and that is precisely what the table shows.
- An A100 has 108 SMs and roughly 14x the FP16 throughput. There, one 1024 px image genuinely leaves the machine partly idle, and batch 4 or 8 costs far less than 4x or 8x the time of batch 1. This is why datacenter inference is quoted in images per second per GPU rather than seconds per image.
Two further mechanisms matter at enterprise scale and neither applies to a single consumer card:
Continuous batching across requests. Providers do not batch one user's four images; they batch four different users' requests into one denoise loop, refilling slots as requests complete. That converts idle occupancy into throughput at high request volume. It needs many concurrent requests to be worth it, which a self-hosted install does not have.
Multi-GPU and partitioning. Large deployments run many GPUs, and a single A100 or H100 can be partitioned (MIG) into instances serving separate streams. Concurrency there comes from having more silicon, not from making one image faster.
So the honest statement is not "batching does not work" but "batching converts spare occupancy into throughput, and this card has no spare occupancy". The same experiment on a rented A100 would very likely show large gains, and the cloud fleet should measure it rather than inherit this conclusion. What it does mean concretely: implementing queued-job micro-batching for a 16 GB self-hosted box would deliver nothing, which is consistent with that work already being deferred in decisions.md.
One caveat on the numbers above: batching also does not help latency even on a large GPU. Every image in a batch finishes when the slowest does, so per-image latency never improves; only images-per-second does. For an interactive studio, latency is what a user feels.
group_offload instead of model_offload. Worth knowing about. The
memory ladder assumes rungs trade speed for VRAM monotonically, so
group_offload sits at the bottom as the slow-but-survivable rung. For SD3
that assumption does not hold: leaf-level streaming with prefetch overlaps
transfer with compute, and a 9.12 GB text encoder that runs once per image
then sits idle is exactly the shape that suits. It came out marginally faster
while using a fifth of the memory. The ladder still selects model_offload
automatically, and that is left alone deliberately, because one model
contradicting the ordering does not justify rewriting rung selection for all
of them. Recorded here so the next person does not assume the bottom rung is
always the slow one.
torch.compile. The largest single win measured anywhere: 40.9 s down to
26.0 s, a 36% reduction. It was unreachable in the previous configuration.
Compile is applied only to full-resident pipelines because accelerate's
offload hooks and Inductor fight each other, so a model forced onto an offload
rung cannot have it. On this card T5-XXL forces the offload rung, so the 36%
stays locked away.
Dropping T5-XXL. Frees 9.12 GB, brings full residency within reach, and unlocks compile. It also discards the entire reason for choosing SD 3.5 over SDXL: the >77-token prompt window that answers issues #147 and #148. Measured for diagnosis, not proposed as a product configuration.
Lower resolution. 768 px costs 27.8 s against 49.5 s, tracking the pixel ratio almost exactly. This is the one lever a user can pull today, and the manifest exposes 768 alongside 1024 for that reason.
Flash attention backend. No measurable change, and the reason is that
there was nothing to gain. DiffusersEngine.__init__ already sets
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 for DEVICE=rocm, which is what
gates the fused attention kernels on RDNA 3; without it torch falls back to
math attention, which is several times slower. The baseline therefore already
runs fused attention through torch SDPA, and an explicit backend can only
match it or regress.
Any standalone ROCm test script must set
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 before SDPA first dispatch. Without
it SDPA falls back to math attention and allocates a 3.51 GiB attention matrix.
Of the backends diffusers 0.39 exposes, none improves on that here. The
FlashAttention-3 entries target Hopper. aiter is AMD's own kernel library
and is the one plausible candidate, but it is not installed, as are
flash_attn, xformers and sageattention. Adding attention backends to
this card is a dependency decision with no demonstrated headroom behind it.
Quantizing T5. The one that paid off, covered in full above. Two practical notes for anyone repeating it.
bitsandbytes does not work on this card. It installs, warns that it is
substituting a ROCm 6.4 binary for ROCm 6.3, and then fails on the GPU with
Error invalid device function at line 432 in file /src/csrc/ops.cu. Its
int8 kernels are not built for gfx1102. Do not spend time on it.
torchao does work, with Int8WeightOnlyConfig, despite warning at import
that its C++ extensions want torch >= 2.11 (this environment has 2.9.1). A
quantised linear layer matches its fp16 counterpart to five decimal places.
The upstream t5xxl_fp8_e4m3fn.safetensors at 4.56 GB is a red herring:
RDNA 3 has no native fp8 matmul, so torch upcasts it on load and the saving
evaporates. int8 weight-only is the format that works here.
An earlier revision of this document concluded that a 16 GB card cannot have
all three of T5-XXL, full residency and torch.compile, because 15.15 GB of
fp16 weights do not fit in ~14.3 GiB. That conclusion was correct about fp16
and wrong as a general statement. Quantising T5 to int8 with torchao fits
everything, and the payoff is large:
| Configuration | Time | Peak VRAM | T5? |
|---|---|---|---|
fp16 T5, model_offload (previous) |
49.5 s | 12.09 GB | Yes |
| int8 T5, full residency, eager | 43.1 s | 13.44 GB | Yes |
int8 T5, full residency, torch.compile |
28.0 s | 13.44 GB | Yes |
| fp16, no T5, full residency + compile | 26.0 s | 8.84 GB | No |
1.77x faster than the previous configuration, with the long-prompt window
intact. int8 brings T5 from 9.12 GB to 4.57 GB, which puts the whole
pipeline at 10.93 GB resident against 15.15 GB before. Full residency then
makes torch.compile available, and compile is worth 35% here (43.1 s to
28.0 s), consistent with the 36% measured on the no-T5 pipeline.
The 28.0 s figure is steady state, not startup latency. A second and later
generation measured 27.2 s. The first generation in a fresh worker measured
138.8 s with the Inductor cache populated, and 528.9 s when compiling from
cold for the first time on the machine. Model load itself takes 220 to 239 s
at full residency because quantisation happens during load. Against the
previous 49.5 s model_offload path with no compile warm-up, the cached int8
path is still behind after five images in a worker's life and pulls ahead on
the sixth. One image after a restart therefore waits longer than before,
while a session of twelve finishes sooner.
It lands within 2 s of the no-T5 configuration while keeping the feature that no-T5 throws away.
Quality survives. On a 90-word prompt the int8 pipeline still renders detail from past CLIP's 77-token cutoff, including a cat and a lamp named only in the tail. Weight-only int8 perturbs the text encoder slightly; it did not cost prompt comprehension, which is the property that matters for choosing SD 3.5 over SDXL in the first place.
Two implementation constraints follow from the measurements:
- 13.44 GB peak against ~14.3 GiB usable is tight. A desktop session
holding more VRAM than usual can push it back to OOM. Under
MEMORY_MODE=auto, the worker now descends one rung after its generation eviction retry fails, reloads and retries the job once. - It adds a
torchaodependency to the worker and makestorch.compilerequired for the quantized full-resident pipeline, which is a change to the engine's failure surface, not just its speed.
Issue #155 adopts this path for sd35-medium only. CUDA workers still need
their own acceptance measurement.
int8 helps sd35-medium because that model does not fit. The other models
already fit. This section records what int8 does to them, so nobody repeats
the test.
Measured at 1024 px, sdxl-base at 20 steps and guidance 6, sdxl-fast at
8 steps and guidance 0. Times are per image.
| Model | Weights | batch 1 | batch 2 | batch 4 |
|---|---|---|---|---|
| sdxl-base | fp16 | 15.66 s / 9.07 GB | 15.23 s / 11.54 GB | OOM |
| sdxl-base | int8 | 15.84 s / 6.98 GB | 15.10 s / 9.45 GB | OOM |
| sdxl-fast | fp16 | 3.87 s / 11.62 GB | OOM | OOM |
| sdxl-fast | int8 | 4.01 s / 7.22 GB | 3.77 s / 9.69 GB | OOM |
Three results follow from the table.
int8 costs a small amount of speed at batch 1. sdxl-base slows by 1.1 percent
and sdxl-fast by 3.6 percent. Int8WeightOnlyConfig stores the weights as
int8 and converts them back to fp16 for each matrix multiply. RDNA 3 has no
fast int8 matmul, so the conversion adds work and returns nothing.
int8 frees real memory. sdxl-base drops 2.09 GB and sdxl-fast drops
4.40 GB. This is the same mechanism that rescues sd35-medium.
The freed memory buys almost no speed. sdxl-fast at batch 2 becomes possible
under int8, where fp16 runs out of memory. It then delivers 3.77 s per image
against 3.87 s for fp16 at batch 1, which is 2.6 percent. The card stays
compute-bound, so a larger batch still finds no idle capacity to use.
The conclusion is narrow and worth stating plainly. int8 solves a memory
problem. Only sd35-medium has a memory problem. For every other model in the
roster, int8 trades a little speed for memory that the model does not need.
Every slow thing measured above falls into one of three categories, and the category determines whether spending money, changing code, or neither will help.
| Limit | Category | Escapable? |
|---|---|---|
| CFG doubles per-step compute | Fundamental | No, not while you want CFG |
| Steps multiply evaluations linearly | Fundamental | No |
| Denoising cannot be parallelised across steps | Fundamental | No |
| SD 3.5 cannot be fully resident in fp16 | This GPU | Yes, int8 T5 fixes it today |
torch.compile unavailable for SD 3.5 |
This GPU | Yes, follows from int8 residency |
| Batching gains nothing | This GPU | Yes, a larger GPU |
| ~375 ms per SDXL UNet evaluation | This GPU | Yes, faster silicon |
| No fp8 quantisation path | This GPU | No, RDNA 3 lacks fp8 matmul |
| int8 quantisation | Stack only | Solved, torchao works |
| T5-XXL costs 9.12 GB | The model | No, inherent to SD 3.5 |
| Distilled models refuse CFG | The model | No, by design |
| 667 ms VAE decode per image | The model | Yes, and it is the best target left |
| More steps stop helping around 20 | The model | No, that is convergence |
Diffusion is sequential. Step N needs step N-1's latent. You cannot spend parallelism, VRAM or money to compute steps concurrently. This is the root reason the batching result came out flat and why latency has a hard floor independent of how large a GPU you buy.
Classifier-free guidance costs exactly 2x. Any guidance_scale > 1.0
evaluates the network twice per step. The only escape is not using CFG, which
is what distillation does, and that is a different model rather than a faster
one.
Steps are linear. Measured across 10 to 50 steps with no sublinearity to exploit. Fewer steps is the only lever, which makes solver convergence and distillation the real levers.
16 GB is the binding constraint, more than compute is. SD 3.5 Medium needs
15.15 GB of fp16 weights against ~14.3 GiB usable after the desktop. That
single fact forces the offload rung, and the offload rung forfeits
torch.compile. A 24 GB card would give back both at once, and so, as it
turns out, does int8 quantisation on this card: 10.93 GB resident, 28.0 s
against 49.5 s. The constraint is real but it is a memory constraint, which
means it has software answers as well as hardware ones. The projection table
below makes the same point from the other side: a V100 32 GB is expected to
roughly double a V100 16 GB on identical compute, purely because the pipeline
fits.
Batching is dead here specifically because the card is small. 32 CUs are saturated by one 1024 px image. The same experiment on an A100 would very likely show real gains, because there a single image leaves the machine idle. Do not generalise the flat result above to other hardware.
Raw throughput is what it is. ~375 ms per SDXL UNet evaluation at 1024 px on ~11.3 TFLOPS fp32 / ~22.6 TFLOPS fp16 shader throughput, with no datacenter-class tensor cores. Faster silicon is the only answer.
Quantisation is limited by both hardware and stack. torchao's
Int8WeightOnlyConfig works on the reference card through its pure PyTorch
path. bitsandbytes does not: its int8 kernel is not built for gfx1102. RDNA 3
still has no native fp8 matmul, so the upstream t5xxl_fp8 file is upcast on
load and saves nothing.
T5-XXL is 9.12 GB and that is the point of it. It is what gives SD 3.5 a prompt window past CLIP's 77 tokens, which is the entire reason the model was chosen over SDXL. Removing it measured 26.0 s against 49.5 s, and would discard the feature.
Distilled models cannot take CFG. Lightning, Hyper-SD, Turbo and LCM bake guidance into the weights. Their manifests cap guidance at 2 as a guard rail. This is not a limitation to fix; it is the trade that makes them 8-step models.
Convergence stops around 20 steps. DPM++ has essentially converged by then, so the remaining 30 steps of the manifest ceiling buy nothing.
VAE decode is the exception, and it is the best remaining target. 667 ms
per image regardless of resolution-independent work, model, or step count. On
a 50-step render it is 2% and invisible; on vega-rt at 1.87 s it is 36% of
total runtime. AutoencoderTiny decodes in roughly 10 ms. Nothing about the
hardware forces this cost, and the deferred realtime ladder already names the
approach. Of everything measured in this document, this is the one large win
that is neither blocked by VRAM nor by physics.
Two independent factors decide the time on another card, and the VRAM one usually dominates:
- Does the pipeline fit fully resident? Fitting removes the offload
overhead and unlocks
torch.compile, worth 36% on its own. SD 3.5 Medium in fp16 needs roughly 15.2 GB of weights plus a few GB of activations, so it wants 20 GB or more to sit resident comfortably. Anything at 16 GB or below is forced onto an offload rung regardless of how fast it computes. - Compute throughput, which sets the per-step cost.
There is no "Tesla T100"; NVIDIA's datacenter line runs T4, V100, A100, H100, so the table uses real parts.
| GPU | VRAM | FP16 throughput | Fits resident? | Estimated 1024/20 |
|---|---|---|---|---|
| RX 7600 XT (this card) | 16 GB | ~22.6 TFLOPS shader | Yes, with int8 T5 | 28.0 s measured |
| Tesla T4 | 16 GB | ~65 TFLOPS tensor | No, offload forced | 40 to 70 s |
| Tesla V100 16 GB | 16 GB | ~125 TFLOPS tensor | No, offload forced | 25 to 40 s |
| Tesla V100 32 GB | 32 GB | ~125 TFLOPS tensor | Yes | 10 to 18 s |
| RTX 4090 | 24 GB | ~165 TFLOPS tensor (dense) | Yes | 6 to 12 s |
| A100 40/80 GB | 40/80 GB | ~312 TFLOPS tensor | Yes | 4 to 8 s |
| H100 80 GB | 80 GB | ~756 TFLOPS tensor (dense) | Yes | 2 to 5 s |
Every figure in the last column except the first is an estimate, not a measurement, and the ranges are deliberately wide. The estimates for the other cards remain fp16 projections because the int8 path has not been accepted on CUDA. Diffusion rarely scales with peak FLOPS: real speedups land well below the paper ratio because attention is partly memory-bound, and the fixed costs of text encoding and VAE decode do not shrink with tensor throughput. Treat the ordering as reliable and the absolute numbers as rough.
The interesting row is the two V100 variants. Identical compute, and the 32 GB part is expected to be roughly twice as fast purely because the pipeline fits. On a card this size, capacity beats throughput.
For self-hosters on 16 GB or less, group_offload at 2.51 GB peak means
sd35-medium will run on far smaller cards than its min_vram_gb suggests;
it will simply be slow.
The int8 pipeline is 10.93 GB resident and reaches 13.44 GB during generation.
min_vram_gb is the full-residency requirement, so 14 rounds up from the
measured peak rather than from the weight-only resident figure. This lets a
reference-card worker with about 14.3 GiB free select full residency while a
busier desktop can select model offload before attempting a tight load.
Free memory can change between rung selection and the generation peak. The automatic fallback therefore remains required even with the measured value: after the existing eviction retry fails, generation descends one rung, reloads and retries once.
The stack must be up: make deps, then make api (with BENCHMARK_API=1)
and make worker-rocm.
# Every model at its manifest ceiling across the 60-prompt suite
backend/.venv/bin/python scripts/benchmark.py \
--prompts scripts/benchmark-prompts-60.json \
--matrix scripts/benchmark-matrix-best.json \
--include-capped --continue-on-errorStop competing GPU workloads first. An earlier issue #75 run was invalidated by an unrelated process holding ~14 GB, and on a card this size anything resident changes which rung the ladder picks.
Issue #155 adds one worker-only manifest field naming a component and scheme.
sd35-medium declares text_encoder_3:int8; no other model is quantized.
The engine applies torchao immediately after _from_pretrained, before the
device move, and requires compile when that quantized pipeline is fully
resident.
Load-time out-of-memory errors under MEMORY_MODE=auto descend from full
residency to model offload, then to group offload. Generation first evicts
other residents and retries as before. If that still fails, it descends one
rung, reloads and retries the job once. Explicitly pinned memory modes never
descend.
The 50-step ceiling buys nothing over 20. A wider sweep shows that 20 also buys little over 15. Six prompts, four step counts, two models, seed 100, 1024 px, dpmsolver.
| Model | 12 steps | 15 steps | 20 steps | 30 steps |
|---|---|---|---|---|
| sdxl-base | 9.73 s | 11.96 s | 15.69 s | 23.09 s |
| ssd-1b | 6.21 s | 7.58 s | 9.81 s | 14.27 s |
Both models moved from a 20-step default to 15. That returns 24 percent of the time on every image the studio makes with them.
The visual check covered a portrait with hands and a face, a botanical illustration with fine linework, and a forge scene with sparks. Faces, hands and crosshatching stay clean at 15 steps. They also stay clean at 12, which suggests more headroom, but the sample does not support a further cut yet.
Read this as a sample, not a proof. The sweep produced 48 images and a person looked at four of them. Step count changes the path through latent space, so two step counts give different compositions rather than the same image at two levels of polish. That rules out a pixel metric and leaves human judgment.
The cosmic suite (scripts/benchmark-prompts-cosmic.json, 60 prompts on black
holes, deep space and cosmic fantasy) ran the whole roster twice over: once at
manifest ceilings, once at the settings this project now believes are correct.
600 images each time, no failures either time.
| Model | Ceiling | Tuned | Change |
|---|---|---|---|
| sdxl-base | 37.34 s at 50 steps | 11.69 s at 15 | 3.2x faster |
| ssd-1b | 18.55 s at 40 steps | 7.37 s at 15 | 2.5x faster |
| sd35-medium | 110.94 s at 50 steps | 48.33 s at 20 | 2.3x faster |
| sdxl-fast | 3.77 s at 8 steps | 3.77 s at 8 | unchanged |
| sdxl-hypersd | 3.77 s at 8 steps | 3.77 s at 8 | unchanged |
| ssd-1b-lightning | 2.61 s at 8 steps | 2.62 s at 8 | unchanged |
| vega-rt | 1.96 s at 8 steps | 1.96 s at 8 | unchanged |
| dreamshaper-lcm | 5.57 s at 15 steps | 5.58 s at 15 | unchanged |
| sdxl-turbo | 0.64 s at 4 steps | 0.64 s at 4 | unchanged |
| sd-turbo | 0.43 s at 4 steps | 0.43 s at 4 | unchanged |
The distilled models do not move, because their design point already is their ceiling. Only the three models that take real step counts had anything to give back, and they gave back a lot.
The whole suite fell from 3 hours 7 minutes to 1 hour 31 minutes. Nothing in the roster got worse.
Read the two runs together and the point is simple. The ceilings were never a recommendation. They were headroom that cost 2.3 to 3.2 times the time and returned quality that did not survive inspection at a fixed seed.