amd-rocm-ollama

Why this exists

On a stock Ubuntu 24.04 install of Ollama on Strix Halo (gfx1151), one of two things happens, neither of them "the GPU is doing the work":

1. Silent CPU fallback. ollama run answers, but every token is generated on the CPU. docker compose exec ollama ollama ps reports 100% CPU / 0% GPU, and make logs shows:

   ... level=INFO source=types.go msg="inference compute" id=cpu library=cpu ...
   ... level=INFO msg="vram-based default context" total_vram="0 B" default_num_ctx=4096
   

2. Hard GPU page fault. The runner finds the GPU, starts loading rocBLAS, and the kernel page-faults on first dispatch:

   ggml_cuda_init: found 1 ROCm devices:
     Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
   ggml_cuda_init: initializing rocBLAS on device 0
   Memory access fault by GPU node-1 (Agent handle: 0x...) on address 0x... .
      Reason: Page not present or supervisor privilege.
   ... source=runner.go msg="failure during GPU discovery" error="runner crashed"
   ... source=types.go msg="inference compute" id=cpu library=cpu ...
   

Behind those symptoms are several distinct root causes that all look identical in the logs:

• MES 0x83 firmware regression — the kernel/driver pair ships an MES build that page-faults on first dispatch. Detect with make mes-check.
• IOMMU not in passthrough mode — missing amd_iommu=on iommu=pt on the kernel cmdline.
• Over-aggressive rocBLAS prune — keeping only *gfx1151* files deletes the 54 arch-agnostic fallback .dat files and faults at first kernel call (see docs/rocblas-prune.md).
• Ollama upstream not building gfx1151 natively until a recent release.
• Permissions / wrong user. On Linux, the default ollama systemd user must pick up the host's render and video supplementary groups (so /dev/kfd and /dev/dri/renderD* are accessible). If initgroups(3) doesn't get them — custom unit overrides, missing groups, or a manually spawned ollama serve outside the service — GPU discovery silently fails and inference falls back to CPU. Switching to User=root (or running the binary under sudo) is a blunt but effective workaround when the group setup is broken. On Windows, the same class of bug appears when the standard installer or ollama.exe is launched without "Run as administrator": the API comes up, but the unelevated process never sees the GPU and every request runs on CPU.

This repo is the documented fix for each Linux/Strix Halo variant — pinned versions, the working build, a 9-layer make validate ladder that names the failing layer (e.g. Layer 5: FAIL_CPU / FAIL_VULKAN), and the docs that map each symptom back to its actual cause in docs/build-fixes.md.

If you saw any of the log lines above on your own Strix Halo box, you are in the right place.

---

Quickstart

git clone --recursive https://github.com/MaxusAI/ryzen-ai-max-rocm-ollama-testbench
cd ryzen-ai-max-rocm-ollama-testbench

./quickstart.sh --build      # FIRST run: build the ROCm image (~30 min), then up + validate
./quickstart.sh              # AFTER first build: skip build, just up + validate
./quickstart.sh --skip-up    # validate the host-installed ollama instead of the container
./quickstart.sh --no-pull    # don't auto-pull llama3.2:latest if no models are installed

./quickstart.sh is the one command a brand-new box needs. It does, in order:

1. Prereq check (docker, docker compose, /dev/kfd, /dev/dri/renderD*, host video/render groups).
2. git submodule update --init --recursive (idempotent).
3. Scaffold .env from .env.example, auto-patching VIDEO_GID/RENDER_GID from your host if they differ from the Ubuntu 24.04 defaults.
4. Image presence check (amd-rocm-ollama:7.2.2). Fail fast if not built — --build is the explicit opt-in for the slow ROCm compile.
5. docker compose up --detach and wait up to 90s for /api/tags.
6. Auto-pull llama3.2:latest (~2 GiB) only if no models are installed, so the smoke test has something to load. Suppress with --no-pull.
7. ./scripts/validate.sh --skip-long-ctx — Layers 0-7 of the 9-layer ladder. Layer 8 (~200K-token prefill, 4-25 min) stays opt-in via make validate-full.

make quickstart is the same script (use ARGS=... to pass flags through: make quickstart ARGS="--build").

The validator picks the smoke and long-context models dynamically: it uses the historical defaults (llama3.2:latest, gemma4:e4b-it-q4_K_M) if present, otherwise falls back to the smallest installed model (smoke) and the largest installed model with declared context ≥ 128K (long-ctx). Set SMOKE_MODEL=... / LONG_CTX_MODEL=... to pin specific tags.

---

Docker stack that builds ollama v0.21.0 from source against ROCm 7.2.2 with native gfx1151 (Strix Halo) support, and serves Gemma 4 with up to a 256K context window on an AMD Ryzen AI MAX+ 395 / Radeon 8060S APU.

Who this is for: AMD Strix Halo (gfx1151), Ubuntu 24.04-style host, Docker, and the stack versions pinned in this README. Not for: other GPUs (Navi, CDNA, Phoenix, …), prebuilt images for every distro, or generic “any ROCm” support.

Clone: use git clone --recursive <url> or run make submodules before make build / docker compose build so external/ollama/ is populated.

Per-host overrides: copy .env.example to .env and edit VIDEO_GID, RENDER_GID, OLLAMA_MODEL_STORE if your machine differs from the defaults (Ubuntu 24.04, video=44, render=992, model store at /usr/share/ollama/.ollama). .env is gitignored.

• No Vulkan, no CUDA, no NVIDIA paths.
• Mounts the host model store at /usr/share/ollama/.ollama so existing pulls (e.g. gemma4:31b-it-q4_K_M) work immediately.
• Runtime image keeps the full ROCm SDK so rocminfo / rocm-smi work inside the container for live debugging.
• Aggressive gfx1151-only rocBLAS pruning keeps image size manageable.

Quick verification: make validate runs the 9-layer test ladder automatically (host kernel, MES firmware, HIP smoke, container health, Ollama GPU discovery, small inference). make validate-full adds the ~200K-token prefill at the end. See docs/validation-tests.md for what each layer means and how to fix failures.

Hit a Memory access fault by GPU or library=cpu? Run make mes-check first — odds are it's the MES 0x83 firmware regression. Fix is one command: make install-mes-firmware && sudo reboot.

---

Hardware tested

Item	Value
GPU	AMD Ryzen AI MAX+ 395 / Radeon 8060S (Strix Halo APU)
ISA	gfx1151 (RDNA 3.5)
Total UMA (LPDDR5X)	~128 GiB (BIOS splits 96 GiB to GPU pool, 31 GiB to system)
GPU VRAM (UMA)	96 GiB (BIOS UMA split)
System RAM	31 GiB
ROCk module	6.16.13 / HSA Runtime 1.18
Base image	rocm/dev-ubuntu-24.04:7.2.2-complete (container build)
Host ROCm	7.2.1 (curl-installed for the host-Ollama path; one minor below the container)
ollama	v0.21.0 (git submodule at external/ollama)
Go (auto)	1.24.1 from external/ollama/go.mod

The host ROCm trails the container by one patch (7.2.1 vs 7.2.2); both produce working gfx1151 runners. Container is the recommended path (version pinned in the Dockerfile, reproducible across hosts).

---

Prerequisites (one-time, on host)

1. Replace Ubuntu's broken MES firmware on Strix Halo

Single most important host-side fix. Ubuntu Noble's current linux-firmware ships an MES (Micro Engine Scheduler) blob at version 0x83 for gfx11_5_1 (Strix Halo). It mismatches the KFD driver's compute VA layout and every compute kernel — host or container — faults at the first dispatch with [gfxhub] page fault … CPF (0x4) WALKER_ERROR=1 MAPPING_ERROR=1. Container side it surfaces as Memory access fault by GPU node-1 followed by Ollama silently falling back to library=cpu total_vram="0 B".

Detect:

sudo cat /sys/kernel/debug/dri/1/amdgpu_firmware_info | grep '^MES feature'
# OK:      MES feature version: 1, firmware version: 0x00000080  (or lower)
# BROKEN:  MES feature version: 1, firmware version: 0x00000083

Fix (one command, then reboot):

make install-mes-firmware    # = sudo ./scripts/install-mes-firmware.sh
sudo reboot
make mes-check               # expect: PASS, MES firmware < 0x83

The installer downloads pre-regression gc_11_5_1_* blobs from upstream linux-firmware git, md5-verifies them, drops them in /lib/firmware/updates/amdgpu/ (precedence over the package dir, survives apt upgrade linux-firmware), and rebuilds initramfs. Idempotent. Manual procedure, alternative upstream commits, and --check / --uninstall flags are in scripts/README.md. Full diagnostic story (false trails, AMD/Ubuntu tracker links, kernel-side MES bugs, playbook for the next regression) is in docs/build-fixes.md Fix 4. Run ./scripts/install-mes-firmware.sh --list-known for the live table of community-tested versions.

2. Kernel cmdline baseline (recommended)

Not strictly required for compute (§1 is what unblocks GPU dispatch), but the AMD-recommended baseline for a UMA APU. Removes one variable from future debugging.

cat /proc/cmdline | tr ' ' '\n' | grep --extended-regexp 'iommu|amdgpu'
# Recommended GRUB_CMDLINE_LINUX_DEFAULT:
#   "quiet splash thunderbolt.host_reset=0 amd_iommu=on iommu=pt"

If you see amd_iommu=off or amdgpu.cwsr_enable=0, fix /etc/default/grub, run sudo update-grub, and reboot. Background on why iommu=pt (passthrough) is preferred on UMA APUs is in docs/validation-tests.md Layer 0 and docs/build-fixes.md Fix 3.

3. Stop the host ollama systemd service

The host has Ubuntu's bundled ollama systemd service. It collides with this container on port 11434 and on the model store at /usr/share/ollama/.ollama. Stop and disable it:

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo ss --tcp --listening --numeric --processes | grep 11434   # should be empty

4. Container runtime sanity

Confirm the user is in the docker group, the host has /dev/kfd and /dev/dri, and the AMDGPU kernel module is loaded:

groups | grep --quiet docker || sudo usermod --append --groups docker "$USER"
ls /dev/kfd /dev/dri
lsmod | grep amdgpu
rocminfo | grep --extended-regexp 'Marketing Name|Name:[[:space:]]+gfx'

The base image must already be present:

docker pull rocm/dev-ubuntu-24.04:7.2.2-complete

Re-enabling the host ollama later: the container runs as root, so any models/manifests it pulls become root-owned in /usr/share/ollama/.ollama. Before re-enabling the host service:

sudo chown --recursive ollama:ollama /usr/share/ollama/.ollama
sudo systemctl enable --now ollama

---

Build & run

make submodules    # init external/ollama at v0.21.0
make build         # docker compose build (slow first time: ~10-25 min)
make up            # docker compose up --detach
make logs          # confirm ROCm discovery: 'discovered N AMD GPUs ... gfx1151'
make gpu-check     # rocminfo + rocm-smi from inside the container

API smoke test:

curl http://localhost:11434/api/tags                    # list installed models
curl http://localhost:11434/api/generate -d '{
  "model":"gemma4:31b-it-q4_K_M",
  "prompt":"Write a haiku about Strix Halo",
  "stream":false,
  "options":{"num_ctx":262144}
}'

docker compose exec ollama ollama ps should report 100% GPU for the loaded model.

Compose / security: docker-compose.yml enables SYS_PTRACE, seccomp=unconfined, and ipc: host to keep ROCm debugging practical. Treat that as an intentional trade-off for a single-user lab host, not a minimal production hardening profile.

---

Setting the 256K context

There is intentionally no derived "gemma4-256k" Modelfile - context length is set by the caller per request:

Client	How
OpenWebUI	Chat or model settings → Advanced Params → Context Length
Raw API	options.num_ctx: 262144 in the JSON body
ollama run	/set parameter num_ctx 262144

OLLAMA_CONTEXT_LENGTH is intentionally not set in docker-compose.yml. The default (0) lets ollama auto-pick 4k/32k/256k based on detected VRAM (see ContextLength / OLLAMA_CONTEXT_LENGTH in external/ollama/envconfig/config.go) so smaller models like llama3.2 aren't unnecessarily inflated. Set context per request, not server-wide.

---

OpenWebUI integration

This stack does not include OpenWebUI - run it separately and point it at the ollama port:

• OpenWebUI on the same host (native install): set OLLAMA_BASE_URL=http://localhost:11434.

• OpenWebUI in another docker compose stack:

  services:
    open-webui:
      image: ghcr.io/open-webui/open-webui:main
      extra_hosts:
        - "host.docker.internal:host-gateway"
      environment:
        OLLAMA_BASE_URL: "http://host.docker.internal:11434"

---

KV-cache memory math (informational, 96 GiB VRAM confirmed)

rocminfo reports Pool 1 at ~96 GiB on this BIOS UMA split. For gemma4:31b-it-q4_K_M at num_ctx=262144 the worst case (FA off, KV f16) is ~43 GiB total: ~20 GiB weights + ~12-20 GiB KV f16 + ~2-3 GiB overhead. With FA on + KV q8_0 it drops to ~30 GiB. Either way 50+ GiB free, enough for 512K context on the 31B model or two medium models concurrently (OLLAMA_MAX_LOADED_MODELS=2). Full operating envelope and limit modes: docs/break-modes.md.

---

Flash attention reality check (gfx1151 + Gemma 4)

OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 are set in compose. On gfx1151 + Gemma 4 there are three possible runtime outcomes; make test-fa classifies which one you're in by greping the server log:

Branch	What happens	Action
(a)	FA works, KV-q8_0 takes effect. Log: enabling flash attention + kv cache type: q8_0. Fastest.	Nothing — this is the goal.
(b)	FA rejected at load. Log: flash attention enabled but not supported by gpu. KV silently downgrades to f16.	Nothing — 96 GiB VRAM absorbs the 12-20 GiB f16 KV easily.
(c)	FA accepted but the 512-dim MMA kernels abort at runtime. Runner crashes; healthcheck reports unhealthy.	Set OLLAMA_FLASH_ATTENTION: "0" in compose, make restart, retry.

Why the outcome is uncertain (gating in external/ollama/ml/device.go, llm/server.go, the Gemma-4 patches in external/ollama/llama/patches/0036-backport-kernels-for-gemma4.patch, and the rocWMMA path being CDNA/RDNA4-only): see scripts/README.md → "Reading Ollama's runtime config & state".

---

Troubleshooting

total_vram="0 B" and Ollama runs on CPU even though rocminfo works

Symptom in make logs: Memory access fault by GPU node-1, library=cpu, total_vram="0 B". In dmesg: [gfxhub] page fault … CPF (0x4) WALKER_ERROR=1 MAPPING_ERROR=1.

This is almost always the buggy 0x83 MES firmware on the host (the container's ROCm install is fine). Re-do Prerequisites § 1 and reboot. A distant secondary cause is amd_iommu=off on the host cmdline (see Prerequisites § 2). Full diagnostic story: docs/build-fixes.md Fix 4.

Container starts but rocminfo shows no GPU agent

make gpu-check                  # rocminfo + rocm-smi from inside the container
make logs | grep --extended-regexp 'discovered|rocm|hip|gfx'

If rocminfo inside the container shows no agent but works on the host, you likely have a permission issue on /dev/kfd or a render node under /dev/dri/ (e.g. renderD128; run ls /dev/dri — the index varies by machine). Confirm your host's render and video group ids and update group_add: in docker-compose.yml:

getent group render video           # find the numeric ids

Host-installed Ollama runs on Vulkan or CPU instead of ROCm

Container users can skip this — the compose image is ROCm-only.

The official curl ... | sh installer succeeds and the API answers, but inference reports library=Vulkan or library=cpu. Almost always the actual cause is the MES 0x83 regression (Fix 4): when the rocm/ runner page-faults during init, Ollama's auto-selector silently falls back. Diagnose in order:

make mes-check                              # 1. firmware safe?
make install-mes-firmware && sudo reboot    # 2. fix it if not
./scripts/validate.sh --mode host           # 3. re-validate; Layer 5 should PASS

The minimal sustainable systemd override is purely operational — OLLAMA_HOST, OLLAMA_DEBUG, OLLAMA_MODELS. No User=root change, no OLLAMA_ROCM=1, no *_VISIBLE_DEVICES are needed; Ollama 0.21.0 picks ROCm on its own when the runner is healthy. Full story including the override snippet, the user/group story, and the audit trail of two retracted theories: docs/build-fixes.md Fix 5.

Error: HSA_STATUS_ERROR or random hangs during generation

Strix Halo + ROCm 7.2 is generally stable, but if you see SDMA-related hangs add this to compose environment: and restart:

HSA_ENABLE_SDMA: "0"

rocBLAS error: Cannot read TensileLibrary*.dat / page fault during init

The prune step in docker/Dockerfile deleted too much. The current safe pattern deletes by other-arch name (gfx908, gfx90a, gfx942, gfx950, gfx10xx, gfx11xx≠1151, gfx12xx) so anything without an arch tag survives. This preserves:

• *gfx1151* per-arch lazy index + code objects (HSA-CO and .dat)
• TensileLibrary_Type_*_fallback.dat (54 arch-agnostic fallback files that rocBLAS reads at init - deleting these triggers the page fault above)
• TensileManifest.txt

If you see "Cannot read" with a specific filename, add a ! -name 'pattern' exclusion. See docs/rocblas-prune.md.

Out of VRAM at 256K

You shouldn't hit this on this hardware (worst case ≈ 43 GiB of 96 GiB VRAM), but if you do:

• Drop num_ctx to 131072 (128K) - same model, half the KV cache.
• Switch to gemma4:e4b-it-q4_K_M at full 256K.
• Confirm the BIOS UMA split is at the expected 96 GiB: rocminfo | grep --before-context 5 'POOL 1' | head -n 30.

Need to rebuild the C++ backends from scratch

make clean-image                # remove the built image
docker builder prune --filter type=exec.cachemount   # drop the ccache mount
make build

---

Repo layout

.
├── docker/
│   ├── Dockerfile                # multi-stage: base / build / runtime (ROCm 7.2.2 + ollama)
│   └── entrypoint.sh             # logs ROCm + ENV discovery, then execs ollama
├── docker-compose.yml            # single ollama service; KFD/DRI; FA env; healthcheck
├── docs/                         # maintainer notes (see docs/README.md for index)
│   ├── break-modes.md            #   what fails first under load: VRAM/GTT/MES
│   ├── build-fixes.md            #   first-build failures + fixes applied
│   ├── rocblas-prune.md          #   what the gfx1151-only rocBLAS prune keeps
│   └── validation-tests.md       #   per-layer spec for the 9-layer validate ladder
├── external/ollama/              # git submodule (URL in .gitmodules; commit = v0.21.0 in parent repo)
├── logs/                         # gitignored: per-machine JSONL run history
├── scripts/                      # see scripts/README.md
│   ├── lib/                      #   sourceable bash + python helpers (api, dmesg, pretty, snapshot, parse_*)
│   ├── hip-kernel-test.cpp       #   tiny HIP smoke kernel built by validate Layer 2
│   ├── install-mes-firmware.sh   #   roll back the broken 0x83 MES blob
│   ├── log-run.sh                #   JSONL wrapper around any test run
│   ├── stress-test.sh            #   VRAM/GTT/MES stress
│   ├── torture.sh                #   escalating torture ladder
│   └── validate.sh               #   9-layer validation ladder
├── Makefile                      # `make help` for the full target list
└── README.md

---

License and upstream

The repository overlay (everything outside external/ollama/) is licensed under the MIT License; see LICENSE. The Ollama submodule keeps its own copyright: external/ollama/LICENSE.

Out of scope

• No OpenWebUI service in this compose stack.
• No Vulkan, CUDA, or NVIDIA paths.
• No automatic model pulling - the host already has the models mounted.
• No GGML_HIP_ROCWMMA_FATTN (RDNA 3.5 isn't on the supported path).
• No multi-arch (linux/amd64 only - Strix Halo is x86_64).
• No HSA_OVERRIDE_GFX_VERSION workaround - gfx1151 is native in the v0.21.0 ROCm 7 preset (CMakePresets.json:85).