feat(prefetch): add file selection (allow_patterns/ignore_patterns) to hf_model - #1528
Open
mhotan wants to merge 1 commit into
Open
feat(prefetch): add file selection (allow_patterns/ignore_patterns) to hf_model#1528mhotan wants to merge 1 commit into
mhotan wants to merge 1 commit into
Conversation
…o hf_model hf_model prefetched the entire HuggingFace repo, which makes it unusable for GGUF repos: those ship many quantizations at one commit (100s of GB) when a caller wants exactly one (~18 GB). Thread allow_patterns/ignore_patterns through HuggingFaceModelInfo -> hf_model() -> store_hf_model_task into both download paths so a caller can prefetch just the files they want: - Streaming path now lists the repo recursively (list_repo_files) and filters with filter_repo_objects, preserving each file's repo-relative subpath, so nested quant layouts are fully captured rather than only top-level files. - Snapshot-download fallback forwards the patterns; the unfiltered call is unchanged. - The selected patterns are recorded in the artifact metadata, so a stored subset (e.g. one quant) is identifiable beyond the source commit. - `flyte prefetch hf-model` gains --allow-pattern/--ignore-pattern. - Sharding ignores patterns (it needs the full weights). Backward compatible: the default (None) prefetches the whole repo as before. Also adds examples/genai/llamacpp/, the GGUF counterpart to the vllm/sglang serving examples: prefetch one quant -> model artifact -> serve with the llama.cpp plugin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Michael Hotan <mike@union.ai>
mhotan
force-pushed
the
mike/prefetch-hf-model-file-selection
branch
from
September 4, 2026 03:55
15284ff to
1b5b01c
Compare
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
flyte.prefetch.hf_modelprefetches the entire HuggingFace repo. That makes it unusable for GGUF repos, which ship many quantizations at one commit (100s of GB) when a caller wants exactly one (~18 GB). This adds file selection sohf_modelcan prefetch just the files you want — the missing piece for using the prefetch → model-artifact pattern withflyteplugins-llamacpp(#1521).What changed
allow_patterns/ignore_patternsthreaded throughHuggingFaceModelInfo→hf_model()→store_hf_model_taskinto both download paths._stream_to_remote_dir) now lists the repo recursively (list_repo_files) and filters withhuggingface_hub.utils.filter_repo_objects, preserving each file's repo-relative subpath — so nested quant layouts are fully captured, not just top-level files. (Previously a flatlssilently skipped subdirectories.)allow_patterns/ignore_patternsattrs), so a stored subset (e.g. one quant) is identifiable beyond the source commit.flyte prefetch hf-modelgains--allow-pattern/--ignore-pattern(repeatable).examples/genai/llamacpp/— the GGUF counterpart tovllm/sglang: prefetch one quant (allow_patterns=["*q4_k_m*"]) → model artifact → serve via the llama.cpp plugin, bound byArtifactValue.Backward compatible: default (
None) prefetches the whole repo exactly as before — existing callers and the exact-kwargs snapshot test are unaffected.How it was tested
pytest tests/flyte/prefetch/test_hf_model.py— 65 passed, 1 skipped. New tests cover: pattern fields defaultNone+ json round-trip; streaming filters to matching files (incl. nested) and preserves subpaths; snapshot fallback forwards patterns (and omits the kwargs when unset); metadata records the patterns;hf_modelthreads patterns into the info.pytest tests/cli/test_prefetch.py— 16 passed.ruff check+ruff format --checkclean on all changed files.Notes
First of a two-PR stack; a follow-up adds a lazy object-store FUSE serving mode for GGUF artifacts (the scale-to-zero-clean delivery that download/
stream_modeldon't cover). This PR stands alone.