feat(prefetch): add file selection (allow_patterns/ignore_patterns) to hf_model

examples/genai/llamacpp/README.md

@@ -0,0 +1,55 @@
+# llama.cpp serving
+
+Serve quantized **GGUF** models with [llama.cpp](https://github.com/ggml-org/llama.cpp)'s
+`llama-server` behind a Flyte App, with an OpenAI-compatible endpoint at `/v1`.
+
+This is the GGUF counterpart to [`../vllm`](../vllm) and [`../sglang`](../sglang). Those serve
+safetensors weights and stream them straight to the GPU; llama.cpp serves the quantized GGUF
+format they don't take, and runs where they don't fit: quantized weights, partial CPU offload
+of models larger than VRAM, and CPU-only serving. It builds on the `flyteplugins.llamacpp`
+plugin — the example is thin: prefetch → artifact → serve.
+
+## The two levers this example shows
+
+**1. Object-store model delivery as a versioned artifact.** `flyte.prefetch.hf_model`
+downloads the weights once to blob storage and publishes them as a model **artifact**
+(versioned by the HuggingFace commit). The app binds the artifact by name with
+`ArtifactValue`, so it is complete at module scope and deploys with a bare `flyte deploy` —
+no run name to thread. The app scales to zero when idle and remounts the same weights on the
+next request.
+
+**2. File selection for GGUF.** A GGUF repo ships many quantizations at one commit; you serve
+exactly one. `hf_model(..., allow_patterns=["*q4_k_m*"])` prefetches only that quant instead
+of the whole repo, and records the selected pattern in the artifact metadata so the stored
+subset is identifiable. Pull a different quant by changing `QUANT` — each is published as its
+own artifact.
+
+## Run it
+
+```bash
+# 1. Prefetch one quant and publish the artifact
+python examples/genai/llamacpp/llamacpp_app.py
+
+# 2. Deploy the app (resolves the artifact at deploy time)
+flyte deploy examples/genai/llamacpp/llamacpp_app.py llamacpp_app
+
+# 3. Call it
+python examples/genai/llamacpp/client.py --endpoint <app-endpoint> --api_key <api-key>
+```
+
+The default model is [`Qwen/Qwen2.5-0.5B-Instruct-GGUF`](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF)
+at `q4_k_m` (~0.4 GB) — small enough to iterate on quickly.
+
+## Variations
+
+- **CPU-only serving.** Drop `gpu` from `resources` and pass a CPU image:
+  ```python
+  from flyteplugins.llamacpp import LlamaCppAppEnvironment, build_llama_cpp_image
+  llamacpp_app = LlamaCppAppEnvironment(..., image=build_llama_cpp_image(cuda=False))
+  ```
+- **A different quant or model.** Change `QUANT` / `MODEL_REPO` and the `allow_patterns` glob;
+  bump `resources` for larger weights.
+- **Serving tuning.** `extra_args` is appended to `llama-server` (e.g. `--ctx-size`, `--parallel`,
+  `--jinja` for tool-calling, `--flash-attn`). See the
+  [llama-server docs](https://github.com/ggml-org/llama.cpp/tree/master/tools/server).
+- **Speculative decoding.** Point `draft_model_hf_path` at a small draft GGUF (see the plugin README).

examples/genai/llamacpp/__init__.py

@@ -0,0 +1 @@
+# llama.cpp app examples

examples/genai/llamacpp/client.py

@@ -0,0 +1,22 @@
+import argparse
+
+from openai import OpenAI
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--endpoint", type=str, required=True, help="The app endpoint URL (without /v1).")
+parser.add_argument("--model_id", type=str, default="qwen2.5-0.5b-instruct")
+parser.add_argument("--api_key", type=str, default="<your-api-key>")
+args = parser.parse_args()
+
+client = OpenAI(base_url=f"{args.endpoint}/v1", api_key=args.api_key)
+
+response = client.chat.completions.create(
+    model=args.model_id,
+    messages=[{"role": "user", "content": "Write a one-line hello in Python."}],
+    stream=True,
+)
+for chunk in response:
+    delta = chunk.choices[0].delta.content
+    if delta:
+        print(delta, end="", flush=True)
+print()

examples/genai/llamacpp/llamacpp_app.py

@@ -0,0 +1,113 @@
+"""
+Serve a GGUF model with llama.cpp, delivered as a prefetched model **artifact**.
+
+This is the llama.cpp counterpart to `examples/genai/vllm` and `examples/genai/sglang`.
+Where those serve safetensors weights with a GPU-streaming loader, llama.cpp serves
+quantized **GGUF** weights -- the format vLLM and SGLang don't take -- and shines exactly
+where they don't fit: quantized models, partial CPU offload of models larger than VRAM,
+and CPU-only serving.
+
+The delivery pattern mirrors `sglang_app_artifact.py`: `flyte.prefetch.hf_model` publishes
+the model as a versioned artifact, and the app binds it by artifact name -- so the app is
+complete at module scope and deployable with a plain `flyte deploy`, no run name to thread.
+
+The one llama.cpp-specific wrinkle is **file selection**. A GGUF repo ships many
+quantizations (q3, q4, q5, q6, q8, ...) and you want exactly one. `hf_model`'s
+`allow_patterns` prefetches just that quant instead of the whole repo, and the stored
+subset is what the app serves.
+
+Step 1 -- prefetch one quant (publishes the artifact)
+-----------------------------------------------------
+
+```
+python examples/genai/llamacpp/llamacpp_app.py
+```
+
+`allow_patterns=["*q4_k_m*"]` stores only the Q4_K_M GGUF (~0.4 GB) out of a repo that also
+ships q3/q5/q6/q8, so the artifact is the single file the server loads. Inspect it with:
+
+```
+flyte get artifact qwen2-5-0-5b-instruct-q4-k-m
+```
+
+The repo is public, so `hf_token_key=None` prefetches anonymously -- no HF_TOKEN secret.
+
+Step 2 -- deploy the app
+------------------------
+
+```
+flyte deploy examples/genai/llamacpp/llamacpp_app.py llamacpp_app
+```
+
+`ArtifactValue` resolves at deploy time and pins the app to the artifact version, so a later
+re-prefetch does not swap the weights under a running app. Redeploy to move forward.
+
+Usage
+-----
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="<your-app-endpoint>/v1", api_key="<your-api-key>")
+
+response = client.chat.completions.create(
+    model="qwen2.5-0.5b-instruct",
+    messages=[{"role": "user", "content": "Write a one-line hello in Python."}],
+)
+print(response.choices[0].message.content)
+```
+"""
+
+from flyteplugins.llamacpp import LlamaCppAppEnvironment
+
+import flyte
+import flyte.app
+
+MODEL_REPO = "Qwen/Qwen2.5-0.5B-Instruct-GGUF"
+QUANT = "q4_k_m"
+
+# hf_model requires an artifact name of [alnum_-] only, and a GGUF repo holds many
+# quants at one commit -- so the quant is encoded into the artifact name to keep each
+# prefetched quant a distinct, addressable artifact.
+ARTIFACT_NAME = "qwen2-5-0-5b-instruct-q4-k-m"
+
+llamacpp_app = LlamaCppAppEnvironment(
+    name="qwen2-5-0-5b-instruct-llamacpp",
+    model_id="qwen2.5-0.5b-instruct",
+    # Bind the prefetched artifact, not a run -- deployable with a bare `flyte deploy`.
+    model_path=flyte.app.ArtifactValue(name=ARTIFACT_NAME, type="directory"),
+    # A 0.5B Q4_K_M GGUF runs comfortably on a single L4; for CPU-only serving, drop the
+    # gpu and build a CPU image with `build_llama_cpp_image(cuda=False)` (see the README).
+    resources=flyte.Resources(cpu="2", memory="8Gi", gpu="L4:1", disk="10Gi"),
+    scaling=flyte.app.Scaling(
+        replicas=(0, 1),
+        scaledown_after=300,  # scale to zero after 5 minutes idle
+    ),
+    requires_auth=True,
+    extra_args="--ctx-size 8192",
+)
+
+
+if __name__ == "__main__":
+    import flyte.prefetch
+    from flyte.remote import Run
+
+    flyte.init_from_config()
+
+    # Prefetch ONE quant out of the multi-quant GGUF repo. Without allow_patterns this
+    # would pull every quant in the repo; with it, only the Q4_K_M file is stored -- and
+    # published as the artifact the app binds above.
+    run: Run = flyte.prefetch.hf_model(
+        repo=MODEL_REPO,
+        artifact_name=ARTIFACT_NAME,
+        allow_patterns=[f"*{QUANT}*"],
+        hf_token_key=None,  # public repo: prefetch anonymously
+        resources=flyte.Resources(cpu="2", memory="4Gi", disk="10Gi"),
+    )
+    print(f"Prefetching {MODEL_REPO} ({QUANT}): {run.url}")
+    run.wait()
+
+    # Nothing needs to be passed from the run to the app -- `llamacpp_app` already names
+    # the artifact, and deploy resolves it.
+    app = flyte.serve(llamacpp_app)
+    print(f"Deployed llama.cpp app: {app.url}")

src/flyte/cli/_prefetch.py

@@ -87,6 +87,24 @@ def prefetch():
     type=str,
     help="Short description of the model.",
 )
+@click.option(
+    "--allow-pattern",
+    "allow_patterns",
+    type=str,
+    multiple=True,
+    help=(
+        "Glob pattern selecting which repo files to prefetch, e.g. `*Q4_K_M*` to pull one GGUF "
+        "quant out of a repo that ships many. Can be specified multiple times. Omit to prefetch "
+        "the whole repo. Ignored when `--shard-config` is set."
+    ),
+)
+@click.option(
+    "--ignore-pattern",
+    "ignore_patterns",
+    type=str,
+    multiple=True,
+    help="Glob pattern excluded from the prefetch, applied after `--allow-pattern`. Can be specified multiple times.",
+)
 @click.option(
     "--force",
     type=int,
@@ -162,6 +180,8 @@ def hf_model(
     serial_format: str | None,
     model_type: str | None,
     short_description: str | None,
+    allow_patterns: tuple[str, ...],
+    ignore_patterns: tuple[str, ...],
     force: int,
     wait: bool,
     hf_token_key: str,
@@ -270,6 +290,8 @@ def hf_model(
         model_type=model_type,
         short_description=short_description,
         shard_config=parsed_shard_config,
+        allow_patterns=list(allow_patterns) or None,
+        ignore_patterns=list(ignore_patterns) or None,
         hf_token_key=hf_token_key,
         resources=Resources(cpu=parsed_cpu, memory=parsed_mem, disk=disk, gpu=gpu, shm=shm),
         force=force,