SDXL LoRA: SIGSEGV entering step 2 on RDNA2 (gfx1030, ROCm 7.2) — gradient-checkpoint recompute path, survives native-math

[bls9707-claude]

Summary

On AMD RDNA2 (gfx1030, ROCm 7.2), an SDXL LoRA train runs cleanly through data prep and completes training step 1 with a valid loss, then the training subprocess dies with SIGSEGV (null instruction pointer) entering step 2. The crash is in the gradient-checkpointing recompute/backward path, not attention: forcing attention_mechanism: native-math (PyTorch MATH SDPA, the backend this arch is stable on) does not change it. Gradient checkpointing is required to fit the model in 16 GB, so there is currently no config on a 16 GB RDNA2 card that both fits and survives past step 1.

This is a hardware/kernel-path report, not a request for support — filing it so the RDNA2 data point is on record with a clean repro.

Environment

• GPU: AMD Radeon RX 6800 (Navi 21, gfx1030, 16 GB). Also reproduced on an RX 6900 XT (same arch/VRAM).
• ROCm: 7.2.1 (torch HIP 7.2.53211)
• torch: 2.13.0+rocm7.2 · diffusers: 0.40.0 · accelerate: 1.14.0
• SimpleTuner: 4.9.1, installed pip install -e '.[rocm]' --extra-index-url https://download.pytorch.org/whl/rocm7.2
• Python: 3.12.3, single GPU

Required workaround just to reach training (separate, minor)

Text-encoder GEMMs crash immediately with RuntimeError: hipblaslt error: HIPBLAS_STATUS_INVALID_VALUE when calling hipblaslt_ext::getAllAlgos(...) during caption caching. Forcing rocBLAS avoids it:

TORCH_BLAS_PREFER_HIPBLASLT=0 DISABLE_ADDMM_HIP_LT=1 PYTORCH_TUNABLEOP_ENABLED=0

(Worth noting since the ROCm quickstart auto-enables TunableOp; on gfx1030 + ROCm 7.2 that path is what trips hipBLASLt here.) With these set, training proceeds to the crash below.

Repro

Minimal SDXL LoRA config (rank 16, 12 images, filename captions):

{
  "model_type": "lora", "model_family": "sdxl", "model_flavour": "base-1.0",
  "max_train_steps": 60, "train_batch_size": 1, "learning_rate": 1e-4,
  "optimizer": "adamw_bf16", "lora_rank": 16,
  "gradient_checkpointing": true,
  "resolution": 1024, "resolution_type": "pixel_area",
  "attention_mechanism": "native-math",
  "report_to": "none"
}

TORCH_BLAS_PREFER_HIPBLASLT=0 DISABLE_ADDMM_HIP_LT=1 PYTORCH_TUNABLEOP_ENABLED=0 \
  simpletuner train --env <env>

Observed

Epoch 1/1, Steps:   2%|▏  | 1/60 [00:11<11:44, 11.95s/it, lr=0.0001, step_loss=0.0018]
...
subprocess.CalledProcessError: Command '[.../python', '.../simpletuner/train.py']' died with <Signals.SIGSEGV: 11>

dmesg at the crash:

python[…]: segfault at 0 ip 0000000000000000 sp … error 14 in python3.12  (null-ip jump)

• Step 1 completes and reports a plausible step_loss=0.0018 — forward + first backward work.
• The segfault lands entering step 2 (the first step whose backward recomputes activations under gradient checkpointing).
• Not OOM: peak was ~12.5 GB of 16 GB with checkpointing on.

What was ruled out (8 runs)

Change	Result
default (diffusers) attention, bf16, ckpt, 1024²	step 1 trains, SIGSEGV → step 2
attention_mechanism: native-math, bf16, ckpt, 1024²	same SIGSEGV → step 2 (rules out the SDPA kernel)
bf16, no gradient checkpointing, 1024² / 768²	OOM (>16 GB activations)
base_model_precision: int8-quanto (via cpu), 1024² and 512²	OOM during the Quanto "Freezing model weights only" pass, before step 1 — the frozen model stays resident on-GPU

int8-torchao was not usable at all here — torchao/_C.abi3.so fails to import on this ROCm build (Unable to import torchao Tensor objects).

Conclusion / ask

The SIGSEGV survives native-math, so it is not the attention backend — it appears to be the gradient-checkpoint recompute path miscompiling/dispatching a bad kernel on gfx1030 under ROCm 7.2. Because checkpointing is required to fit 16 GB, RDNA2 users have no working SDXL LoRA path today. Any pointer to a known-good gfx1030 recompute/backward config, or a way to disable checkpointing recompute selectively, would help — and if this is a torch/ROCm-level issue rather than SimpleTuner's, that confirmation is useful too.