Add flyteplugins-llamacpp: serve GGUF models with llama.cpp - #1521
Merged
Conversation
Adds a LlamaCppAppEnvironment (adapted from internal-union-apps' llm-service) following the vllm/sglang plugin structure: - LlamaCppAppEnvironment serves GGUF weights from object storage (model_path / RunOutput / ArtifactValue) or straight from a Hugging Face GGUF repo (--hf-repo), with draft-model speculative decoding support. - llama-cpp-fserve shim resolves the concrete .gguf inside the mounted model directory at startup (first-shard aware) and execs llama-server. - build_llama_cpp_image compiles llama-server from source with CUDA (and the embedded Web UI), with knobs for CUDA arch, repo/ref pin, and a CPU-only variant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmFSWkpi4eoke3vBE64Wuc
cosmicBboy
marked this pull request as ready for review
September 2, 2026 15:55
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmFSWkpi4eoke3vBE64Wuc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
flyteplugins-llamacppplugin for serving GGUF models with llama.cpp'sllama-serverbehind Flyte Apps, adapted frominternal-union-apps/llm-serviceand following the existing vllm/sglang plugin structure.LlamaCppAppEnvironment: OpenAI-compatible endpoint (/v1) + embedded Web UI. Serves weights from object storage (model_pathas remote path /RunOutput/ArtifactValue, downloaded and mounted) or straight from a Hugging Face GGUF repo (model_hf_path→--hf-repo, downloaded by llama-server at startup). Supports speculative decoding viadraft_model_path/draft_model_hf_path(→--model-draft/--hf-repo-draft), with tuning throughextra_args.llama-cpp-fserveshim (console script): the GGUF filename inside a mounted directory is unknown at deploy time, so the shim rewrites--model-dir/--draft-model-dirinto concrete--model/--model-draftpaths at startup (first-shard aware for sharded models) and execsllama-server.build_llama_cpp_image: llama.cpp has no GPU pip wheel, so the default image compilesllama-serverfrom source with CUDA 12.8 and the embedded SvelteKit Web UI (Node build), keeping the internal app's stub-linking and base64 RUN-script workarounds. Knobs:cuda_arch(fat binaries),repo/refpinning, andcuda=Falsefor a CPU-only image.Not carried over from the internal app: the preset catalog, subdomain derivation, and deployment-config machinery (internal concerns, not plugin API).
CI needs no changes: unit tests, lint, and publish workflows auto-discover plugins.
Test plan
make unit_test_pluginswithFLYTE_PLUGIN=plugins/llamacpp): env validation, argv construction, HF vs mounted weights, draft models, shell quoting,clone_with, and shim GGUF resolution.🤖 Generated with Claude Code
https://claude.ai/code/session_01MmFSWkpi4eoke3vBE64Wuc