feat(llamacpp): add fuse delivery mode (read-only object-store PVC mount)

[mhotan]

Intent

Give LlamaCppAppEnvironment a scale-to-zero-clean way to serve large GGUF weights. Today the plugin only downloads the bound model into the pod's local disk before llama-server starts — the whole GGUF (up to ~18 GB) lands on ephemeral disk and the copy is on the cold-start path. This adds a lazy FUSE-mount delivery so scale-from-zero is bounded by GPU node cold-start, not by re-downloading the model.

Root cause it works around: flyte/app/_parameter.py forces download=True whenever mount= is set for a Dir/ArtifactValue/RunOutput, so there is no way to make a bound model available at a path without copying it to local disk. Rather than reworking that core primitive, this re-leverages the proven, Knative-friendly path: a static, CSI-backed read-only PVC attached via pod_template.

What changed

• model_delivery: Literal["download", "fuse"] on LlamaCppAppEnvironment (default "download" — existing behavior unchanged).
• "fuse" mode: reads weights in place from a read-only, object-store-backed PVC (gcsfuse on GKE, Mountpoint-S3 on EKS). New knobs: model_pvc, model_mount_path (default /tmp/models), fuse_pod_annotations. model_path/draft_model_path become relative subpaths under the mount.
• Attaches the PVC volume + read-only mount to the primary app container and emits no download Parameter. Idempotent under clone_with (which re-runs __post_init__). Validation rejects model_hf_path, bound ArtifactValue/RunOutput, or a missing model_pvc in fuse mode; and the fuse-only knobs in download mode.
• Why a PVC and not an inline CSI volume / device-plugin: a PersistentVolumeClaim is on Knative's podspec allow-list and releases cleanly on scale-down; a node device-plugin (JuiceFS / Union Volume) pins the node and blocks scale-to-zero.
• Two examples, one per CSI driver: examples/genai/llamacpp/llamacpp_app_gcsfuse.py (GKE — sets the gke-gcsfuse/volumes: "true" sidecar annotation) and llamacpp_app_mountpoint_s3.py (EKS — mounts the static PV directly, no annotation). Each demonstrates the full loop: a Flyte run (hf_model) creates a versioned Model artifact under the FUSE-visible prefix, the run's artifact URI is stripped of the mounted prefix to derive the subpath, and the app streams exactly that artifact in place via FUSE — no download. README adds a download-vs-fuse comparison.

How it was tested

• pytest plugins/llamacpp/tests → 48 passed (9 new fuse tests; existing 39 unchanged).
• Serde smoke: the fuse pod_spec sanitizes to the exact k8s shape the CSI driver needs (RO PVC volume + RO volumeMount on app, gcsfuse sidecar annotation).
• Both examples construct at module scope; ruff check / ruff format --check / check_docstring_style.py all clean.
• Backing infra path proven end-to-end on a live GKE dataplane via internal-flyte-apps (gcsfuse RO PVC serving a real GGUF, scale-to-zero-clean).

Stack

Part of a linear stack — merge in order:

main ← #1528 (prefetch allow_patterns) ← #1531 (CUDA image build fix) ← #1532 (this PR).

This PR is based on #1531: fuse mode reuses the plugin's build_llama_cpp_image (fixed in #1531), and the examples compose #1528's hf_model(allow_patterns=…) filtered prefetch with this PR's fuse mount.