fix(llamacpp): make the default CUDA image build (slim toolkit + nvcc on PATH) - #1531
Open
mhotan wants to merge 1 commit into
Open
fix(llamacpp): make the default CUDA image build (slim toolkit + nvcc on PATH)#1531mhotan wants to merge 1 commit into
mhotan wants to merge 1 commit into
Conversation
mhotan
force-pushed
the
mike/llamacpp-cuda-image-fixes
branch
from
September 4, 2026 03:56
e1819a8 to
66e641d
Compare
… on PATH) build_llama_cpp_image's default CUDA image failed to build on constrained image builders, for two independent reasons found driving a real GPU serve end to end: 1. Full `cuda-toolkit-12-8` metapackage. It drags in cuda-nsight-systems (a GUI profiler: Java/GTK/X11, ~2 GB) that a headless build never uses, bloating the image and OOM-killing resource-constrained builders mid-install. Replace it with the compile-only subset (cuda-nvcc + cudart/driver stubs + cuBLAS/cuRAND dev, ~2.8 GB). 2. `No CMAKE_CUDA_COMPILER could be found`. The image's PATH env (which includes $CUDA_HOME/bin, where nvcc lives) is applied *after* the cmake configure/build RUN steps, so nvcc is not on PATH during the build. Set PATH inline on the cmake configure and build commands. With both fixes the default `build_llama_cpp_image()` compiles llama-server and the app serves an OpenAI-compatible /v1 endpoint on a GPU node (verified on a live control plane). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Michael Hotan <mike@union.ai>
mhotan
force-pushed
the
mike/llamacpp-cuda-image-fixes
branch
from
September 4, 2026 11:44
66e641d to
fdfa9b6
Compare
mhotan
changed the base branch from
main
to
mike/prefetch-hf-model-file-selection
September 4, 2026 11:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
flyteplugins.llamacpp's defaultbuild_llama_cpp_image()(CUDA mode) does not build on resource-constrained image builders. Both failures were found while driving a real GGUF GPU serve end-to-end on a live control plane; with these two fixes the default image compilesllama-serverand the app serves an OpenAI-compatible/v1on a GPU node.What changed
_constants.py).cuda-toolkit-12-8is a metapackage that drags incuda-nsight-systems— a GUI profiler pulling Java/GTK/X11 (~2 GB) that a headless build never uses. It bloats the image and OOM-kills constrained builders mid-install. Replaced with the compile-only subset:cuda-nvcc-12-8 cuda-cudart-dev-12-8 cuda-driver-dev-12-8 libcublas-dev-12-8 libcurand-dev-12-8(~2.8 GB).nvcconPATHfor the build (_image.py). The image'sPATHenv (which includes$CUDA_HOME/bin) is applied after the cmake configure/build RUN steps, sonvccisn't onPATHduring the build and cmake fails withNo CMAKE_CUDA_COMPILER could be found. SetPATH=$CUDA_HOME/bin:$PATHinline on the cmake configure and build commands.How it was tested
Built the default image via
build_llama_cpp_image()on a live dataplane and deployed aLlamaCppAppEnvironment(Qwen3.8-27B GGUF, single L40S). Before: the build failed at ~4 min (OOM during CUDA install) or immediately (No CMAKE_CUDA_COMPILER). After: the image compiles and the app reaches a live/v1endpoint on the GPU node.Note
Separately,
flyteplugins-llamacppisn't published to PyPI yet, sobuild_llama_cpp_image'swith_pip_packages("flyteplugins-llamacpp", pre=True)can't resolve — a release is needed for the default image to be buildable off the shelf. (Worked around locally with a source wheel.)Stack
Linear stack, merge in order:
main← #1528 (prefetchallow_patterns) ← #1531 (this PR) ← #1532 (fuse delivery). Based on #1528; #1532 needs this PR's image build fix to compile the CUDA image.