Skip to content

Commit 434003c

Browse files
cosmicBboyclaude
cosmicBboy
and
claude
authored
Add flyteplugins-llamacpp: serve GGUF models with llama.cpp (#1521)
## Summary Adds a new `flyteplugins-llamacpp` plugin for serving GGUF models with [llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` behind Flyte Apps, adapted from `internal-union-apps/llm-service` and following the existing vllm/sglang plugin structure. - **`LlamaCppAppEnvironment`**: OpenAI-compatible endpoint (`/v1`) + embedded Web UI. Serves weights from object storage (`model_path` as remote path / `RunOutput` / `ArtifactValue`, downloaded and mounted) or straight from a Hugging Face GGUF repo (`model_hf_path` → `--hf-repo`, downloaded by llama-server at startup). Supports speculative decoding via `draft_model_path` / `draft_model_hf_path` (→ `--model-draft` / `--hf-repo-draft`), with tuning through `extra_args`. - **`llama-cpp-fserve` shim** (console script): the GGUF filename inside a mounted directory is unknown at deploy time, so the shim rewrites `--model-dir`/`--draft-model-dir` into concrete `--model`/`--model-draft` paths at startup (first-shard aware for sharded models) and execs `llama-server`. - **`build_llama_cpp_image`**: llama.cpp has no GPU pip wheel, so the default image compiles `llama-server` from source with CUDA 12.8 and the embedded SvelteKit Web UI (Node build), keeping the internal app's stub-linking and base64 RUN-script workarounds. Knobs: `cuda_arch` (fat binaries), `repo`/`ref` pinning, and `cuda=False` for a CPU-only image. Not carried over from the internal app: the preset catalog, subdomain derivation, and deployment-config machinery (internal concerns, not plugin API). CI needs no changes: unit tests, lint, and publish workflows auto-discover plugins. ## Test plan - [x] 39 unit tests (`make unit_test_plugins` with `FLYTE_PLUGIN=plugins/llamacpp`): env validation, argv construction, HF vs mounted weights, draft models, shell quoting, `clone_with`, and shim GGUF resolution. - [x] ruff check/format clean; pre-commit (mypy, ty) passed. - [ ] Deploy smoke test on a live cluster. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MmFSWkpi4eoke3vBE64Wuc --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent ee21132 commit 434003c

12 files changed

Lines changed: 2683 additions & 0 deletions

File tree

‎plugins/llamacpp/README.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Union llama.cpp Plugin
2+
3+
Serve GGUF models with [llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` behind Flyte Apps.
4+
5+
This plugin provides the `LlamaCppAppEnvironment` class for deploying quantized (GGUF) LLMs
6+
with an OpenAI-compatible API (under `/v1`) and the built-in llama.cpp Web UI. llama.cpp shines
7+
where vLLM and SGLang don't fit: quantized GGUF weights, partial CPU offload of models larger
8+
than VRAM, and CPU-only serving.
9+
10+
## Installation
11+
12+
```bash
13+
pip install --pre flyteplugins-llamacpp
14+
```
15+
16+
## Usage
17+
18+
```python
19+
import flyte
20+
import flyte.app
21+
from flyteplugins.llamacpp import LlamaCppAppEnvironment
22+
23+
llama_app = LlamaCppAppEnvironment(
24+
name="my-llm-app",
25+
# A directory (or direct path) of GGUF weights in object storage...
26+
model_path="s3://your-bucket/models/your-model-gguf",
27+
model_id="your-model-id",
28+
resources=flyte.Resources(cpu="4", memory="32Gi", gpu="L40s:1", disk="100Gi"),
29+
scaling=flyte.app.Scaling(replicas=(0, 1), scaledown_after=300),
30+
)
31+
32+
if __name__ == "__main__":
33+
flyte.init_from_config()
34+
app = flyte.serve(llama_app)
35+
print(f"Deployed llama.cpp app: {app.url}")
36+
```
37+
38+
`model_path` accepts a remote directory or file path, a `RunOutput` (e.g. from a prefetch task
39+
that downloaded the GGUF), or an `ArtifactValue`. The weights are downloaded into the container
40+
and the served `.gguf` is located at startup; for sharded models the `-00001-of-` shard is
41+
selected and llama-server discovers the rest.
42+
43+
Alternatively, point directly at a Hugging Face GGUF repo (with an optional quant tag) and let
44+
llama-server download it at startup:
45+
46+
```python
47+
llama_app = LlamaCppAppEnvironment(
48+
name="gemma-app",
49+
model_hf_path="ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
50+
model_id="gemma-3-4b-it",
51+
resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L4:1", disk="50Gi"),
52+
)
53+
```
54+
55+
## The default image
56+
57+
llama.cpp ships no GPU pip wheel, so the default image compiles `llama-server` from source with
58+
CUDA enabled (plus the embedded Web UI). The default targets compute capability 8.9 (L4/L40S);
59+
use `build_llama_cpp_image` to target other GPUs, pin a llama.cpp release for reproducible
60+
builds, or build a CPU-only image:
61+
62+
```python
63+
from flyteplugins.llamacpp import LlamaCppAppEnvironment, build_llama_cpp_image
64+
65+
llama_app = LlamaCppAppEnvironment(
66+
name="my-llm-app",
67+
image=build_llama_cpp_image(
68+
cuda_arch="80;86;89;90", # fat binary: A100, A10, L4/L40S, H100
69+
ref="b6148", # pin a llama.cpp release tag
70+
),
71+
...
72+
)
73+
```
74+
75+
`build_llama_cpp_image(cuda=False)` produces a CPU-only image for serving small quantized
76+
models without a GPU.
77+
78+
## Speculative decoding
79+
80+
Point `draft_model_path` (object storage, `RunOutput`, or `ArtifactValue`) or
81+
`draft_model_hf_path` at a small draft GGUF and it is passed to llama-server as
82+
`--model-draft` / `--hf-repo-draft`. Tune the speculation via `extra_args`:
83+
84+
```python
85+
llama_app = LlamaCppAppEnvironment(
86+
name="qwen3-spec",
87+
model_path="s3://your-bucket/models/qwen3-32b-gguf",
88+
model_id="qwen3-32b",
89+
draft_model_hf_path="ggml-org/Qwen3-0.6B-GGUF:Q8_0",
90+
extra_args="--draft-max 16 --draft-min 1 --gpu-layers-draft 99",
91+
resources=flyte.Resources(cpu="8", memory="64Gi", gpu="L40s:1", disk="120Gi"),
92+
)
93+
```
94+
95+
## Extra arguments
96+
97+
`extra_args` is appended to `llama-server`, as either a string or a list:
98+
99+
```python
100+
llama_app = LlamaCppAppEnvironment(
101+
name="my-llm-app",
102+
model_path="s3://your-bucket/models/your-model-gguf",
103+
model_id="your-model-id",
104+
extra_args="--ctx-size 32768 --parallel 4 --jinja",
105+
)
106+
```
107+
108+
Useful flags: `--ctx-size` (context length), `--parallel` (concurrent request slots),
109+
`--jinja` (enable the model's chat template, needed for tool calling), `--n-gpu-layers`
110+
(limit GPU offload for models larger than VRAM; recent llama.cpp offloads everything by
111+
default), `--cache-type-k/--cache-type-v` (quantized KV cache), `--flash-attn`.
112+
113+
Arguments are quoted before they reach the server, so values containing spaces or JSON survive
114+
intact. Arguments of the form `$MY_VAR` are left unquoted so that Flyte still expands them from
115+
the app's environment.
116+
117+
Run `llama-server --help` or see the
118+
[llama-server docs](https://github.com/ggml-org/llama.cpp/tree/master/tools/server)
119+
for all options.

‎plugins/llamacpp/pyproject.toml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
[project]
2+
name = "flyteplugins-llamacpp"
3+
dynamic = ["version"]
4+
description = "llama.cpp plugin for flyte"
5+
readme = "README.md"
6+
authors = [{ name = "Niels Bantilan", email = "cosmicbboy@users.noreply.github.com" }]
7+
requires-python = ">=3.10"
8+
dependencies = [
9+
"flyte>=2.0.0b43",
10+
]
11+
12+
[build-system]
13+
requires = ["setuptools", "setuptools_scm"]
14+
build-backend = "setuptools.build_meta"
15+
16+
[dependency-groups]
17+
dev = [
18+
"pytest>=8.3.5",
19+
"pytest-asyncio>=0.26.0",
20+
]
21+
22+
[tool.setuptools]
23+
include-package-data = true
24+
license-files = ["licenses/*.txt", "LICENSE"]
25+
26+
[tool.setuptools.packages.find]
27+
where = ["src"]
28+
include = ["flyteplugins*"]
29+
30+
[tool.setuptools_scm]
31+
root = "../../"
32+
33+
[project.scripts]
34+
llama-cpp-fserve = "flyteplugins.llamacpp._server:main"
35+
36+
[tool.pytest.ini_options]
37+
norecursedirs = []
38+
log_cli = true
39+
log_cli_level = 20
40+
markers = []
41+
asyncio_default_fixture_loop_scope = "function"
42+
43+
[tool.coverage.run]
44+
branch = true
45+
46+
[tool.ruff]
47+
line-length = 120
48+
49+
[tool.ruff.lint]
50+
select = [
51+
"E",
52+
"W",
53+
"F",
54+
"I",
55+
"PLW",
56+
"YTT",
57+
"ASYNC",
58+
"C4",
59+
"T10",
60+
"EXE",
61+
"ISC",
62+
"LOG",
63+
"PIE",
64+
"Q",
65+
"RSE",
66+
"FLY",
67+
"PGH",
68+
"PLC",
69+
"PLE",
70+
"PLW",
71+
"FURB",
72+
"RUF",
73+
]
74+
ignore = ["PGH003", "PLC0415"]
75+
76+
[tool.ruff.lint.per-file-ignores]
77+
"examples/*" = ["E402"]
78+
79+
[tool.uv.sources]
80+
flyte = { path = "../../", editable = true }

‎plugins/llamacpp/src/flyteplugins/llamacpp/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
__all__ = ["DEFAULT_LLAMA_CPP_IMAGE", "LlamaCppAppEnvironment", "build_llama_cpp_image"]
2+
3+
from flyteplugins.llamacpp._app_environment import LlamaCppAppEnvironment
4+
from flyteplugins.llamacpp._image import DEFAULT_LLAMA_CPP_IMAGE, build_llama_cpp_image

0 commit comments

Comments
 (0)