Skip to content

feat(prefetch): add file selection (allow_patterns/ignore_patterns) to hf_model - #1528

Open
mhotan wants to merge 1 commit into
mainfrom
mike/prefetch-hf-model-file-selection
Open

feat(prefetch): add file selection (allow_patterns/ignore_patterns) to hf_model#1528
mhotan wants to merge 1 commit into
mainfrom
mike/prefetch-hf-model-file-selection

Conversation

@mhotan

@mhotan mhotan commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Intent

flyte.prefetch.hf_model prefetches the entire HuggingFace repo. That makes it unusable for GGUF repos, which ship many quantizations at one commit (100s of GB) when a caller wants exactly one (~18 GB). This adds file selection so hf_model can prefetch just the files you want — the missing piece for using the prefetch → model-artifact pattern with flyteplugins-llamacpp (#1521).

What changed

  • allow_patterns / ignore_patterns threaded through HuggingFaceModelInfohf_model()store_hf_model_task into both download paths.
  • Streaming path (_stream_to_remote_dir) now lists the repo recursively (list_repo_files) and filters with huggingface_hub.utils.filter_repo_objects, preserving each file's repo-relative subpath — so nested quant layouts are fully captured, not just top-level files. (Previously a flat ls silently skipped subdirectories.)
  • Snapshot-download fallback forwards the patterns; the unfiltered call is byte-for-byte unchanged.
  • Artifact metadata records the selected patterns (allow_patterns/ignore_patterns attrs), so a stored subset (e.g. one quant) is identifiable beyond the source commit.
  • CLI: flyte prefetch hf-model gains --allow-pattern / --ignore-pattern (repeatable).
  • Sharding intentionally ignores the patterns (it needs the full weights).
  • Examples: new examples/genai/llamacpp/ — the GGUF counterpart to vllm/sglang: prefetch one quant (allow_patterns=["*q4_k_m*"]) → model artifact → serve via the llama.cpp plugin, bound by ArtifactValue.

Backward compatible: default (None) prefetches the whole repo exactly as before — existing callers and the exact-kwargs snapshot test are unaffected.

How it was tested

  • pytest tests/flyte/prefetch/test_hf_model.py — 65 passed, 1 skipped. New tests cover: pattern fields default None + json round-trip; streaming filters to matching files (incl. nested) and preserves subpaths; snapshot fallback forwards patterns (and omits the kwargs when unset); metadata records the patterns; hf_model threads patterns into the info.
  • pytest tests/cli/test_prefetch.py — 16 passed.
  • ruff check + ruff format --check clean on all changed files.

Notes

First of a two-PR stack; a follow-up adds a lazy object-store FUSE serving mode for GGUF artifacts (the scale-to-zero-clean delivery that download/stream_model don't cover). This PR stands alone.

…o hf_model

hf_model prefetched the entire HuggingFace repo, which makes it unusable for GGUF
repos: those ship many quantizations at one commit (100s of GB) when a caller wants
exactly one (~18 GB). Thread allow_patterns/ignore_patterns through
HuggingFaceModelInfo -> hf_model() -> store_hf_model_task into both download paths so
a caller can prefetch just the files they want:

- Streaming path now lists the repo recursively (list_repo_files) and filters with
  filter_repo_objects, preserving each file's repo-relative subpath, so nested quant
  layouts are fully captured rather than only top-level files.
- Snapshot-download fallback forwards the patterns; the unfiltered call is unchanged.
- The selected patterns are recorded in the artifact metadata, so a stored subset
  (e.g. one quant) is identifiable beyond the source commit.
- `flyte prefetch hf-model` gains --allow-pattern/--ignore-pattern.
- Sharding ignores patterns (it needs the full weights).

Backward compatible: the default (None) prefetches the whole repo as before.

Also adds examples/genai/llamacpp/, the GGUF counterpart to the vllm/sglang serving
examples: prefetch one quant -> model artifact -> serve with the llama.cpp plugin.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Michael Hotan <mike@union.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant