Skip to content

Commit 1d280bb

Browse files
authored
Import transformers only where the LLM runner needs it (pytorch#22268)
## Summary Importing the LLM runner from a plain wheel install fails: ``` >>> import executorch.extension.llm.runner ModuleNotFoundError: No module named 'transformers' ``` `transformers` is not a dependency of this package and the wheel does not install it, but `extension/llm/runner/__init__.py` imported `BatchFeature` from it at module scope. So the entire runner was unreachable, including the compiled `_llm_runner` extension the wheel ships. That extension is around 3 MB and has nothing to do with HuggingFace. ## The change Only two functions need that class: `generate_hf` and `generate_text_hf`. Both exist specifically to accept output from a HuggingFace `AutoProcessor`, so anyone calling them already has `transformers` installed. The class is now looked up when one of those functions runs, rather than when the module loads. The annotations that mention it became strings under `TYPE_CHECKING`, so type checkers still resolve it normally. Nothing changes for users who have `transformers`: the same class object is used, so the `isinstance` checks and the conversion behave exactly as before. For users who do not, the runner imports fine and only the two `hf` helpers raise, naming the package that is missing. ## Test plan Ran against the installed wheel on macOS arm64, from a directory containing no checkout. Without `transformers`: | | result | | --- | --- | | before | `ModuleNotFoundError: No module named 'transformers'` | | after | import OK, `MultimodalRunner` present, both `hf` helpers attached | | after, calling `generate_hf` | still raises, naming `transformers` | With `transformers` installed, confirming behaviour is unchanged: ``` _batch_feature_type() is transformers' BatchFeature True isinstance(BatchFeature(...), resolved) True _hf_to_multimodal_inputs(BatchFeature(...)) 1 MultimodalInput ```
1 parent f715ad2 commit 1d280bb

1 file changed

Lines changed: 44 additions & 15 deletions

File tree

‎extension/llm/runner/__init__.py‎

Lines changed: 44 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,36 @@
3535

3636

3737
import logging
38-
from typing import Callable, List, Optional, Union
38+
from typing import Any, Callable, List, Optional, TYPE_CHECKING, Union
3939

40-
from transformers.feature_extraction_utils import BatchFeature
40+
if TYPE_CHECKING:
41+
from transformers.feature_extraction_utils import BatchFeature
42+
else:
43+
# A runtime stand-in, so the annotations below stay resolvable without transformers. They
44+
# name BatchFeature as a string, and typing.get_type_hints evaluates those strings in this
45+
# module's namespace, so a name that only exists under TYPE_CHECKING makes that call raise
46+
# NameError, even on an install that has transformers.
47+
BatchFeature = Any
48+
49+
50+
def _is_batch_feature(inputs) -> bool:
51+
"""Whether `inputs` is a HuggingFace BatchFeature.
52+
53+
transformers is not a dependency of this package, and only the generate_hf helpers below
54+
need it, so importing it at module scope made the whole runner unimportable on an install
55+
that has no transformers, including the compiled extension imported above that has nothing
56+
to do with HuggingFace.
57+
58+
A missing transformers answers no rather than raising. Nothing can be a BatchFeature when
59+
the class does not exist, and raising here would turn "this input is the wrong type" into a
60+
missing dependency error for a caller who never intended to use HuggingFace at all.
61+
"""
62+
try:
63+
from transformers.feature_extraction_utils import BatchFeature
64+
except ImportError:
65+
return False
66+
67+
return isinstance(inputs, BatchFeature)
4168

4269

4370
def _find_image_token_runs(
@@ -68,7 +95,7 @@ def _find_image_token_runs(
6895

6996

7097
def _hf_to_multimodal_inputs( # noqa: C901
71-
inputs: BatchFeature, image_token_id: Optional[int] = None
98+
inputs: "BatchFeature", image_token_id: Optional[int] = None
7299
) -> List[MultimodalInput]:
73100
"""Convert a HuggingFace AutoProcessor dict to ExecuTorch MultimodalInputs.
74101
Currently only support 1 image inside the input.
@@ -171,22 +198,23 @@ def _hf_to_multimodal_inputs( # noqa: C901
171198

172199
def generate_hf(
173200
runner: MultimodalRunner,
174-
inputs: Union[BatchFeature, List[MultimodalInput]],
201+
inputs: Union["BatchFeature", List[MultimodalInput]],
175202
config: GenerationConfig,
176203
image_token_id: Optional[int] = None,
177204
token_callback: Optional[Callable[[str], None]] = None,
178205
stats_callback: Optional[Callable[[Stats], None]] = None,
179206
) -> None:
180207
"""Generate using an BatchFeature by converting to multimodal inputs internally, or using a list of MultimodalInput."""
181-
if isinstance(inputs, BatchFeature):
208+
# The list form is tested first because it needs nothing from HuggingFace. Resolving
209+
# BatchFeature to compare against it would import transformers, which this package does not
210+
# declare, so a caller passing a list would fail on a dependency its own input never needs.
211+
if isinstance(inputs, list) and all(isinstance(i, MultimodalInput) for i in inputs):
212+
converted = inputs
213+
elif _is_batch_feature(inputs):
182214
logging.info(
183215
"Input is a BatchFeature, assuming it's coming from HF AutoProcessor.apply_chat_template(). Converting to multimodal inputs."
184216
)
185217
converted = _hf_to_multimodal_inputs(inputs, image_token_id=image_token_id)
186-
elif isinstance(inputs, list) and all(
187-
isinstance(i, MultimodalInput) for i in inputs
188-
):
189-
converted = inputs
190218
else:
191219
raise RuntimeError(
192220
"inputs must be either a BatchFeature (from HF AutoProcessor) or a list of MultimodalInput"
@@ -197,20 +225,21 @@ def generate_hf(
197225

198226
def generate_text_hf(
199227
runner: MultimodalRunner,
200-
inputs: Union[BatchFeature, List[MultimodalInput]],
228+
inputs: Union["BatchFeature", List[MultimodalInput]],
201229
config: GenerationConfig,
202230
image_token_id: Optional[int] = None,
203231
) -> str:
204232
"""Generate using an BatchFeature by converting to multimodal inputs internally, or using a list of MultimodalInput."""
205-
if isinstance(inputs, BatchFeature):
233+
# The list form is tested first because it needs nothing from HuggingFace. Resolving
234+
# BatchFeature to compare against it would import transformers, which this package does not
235+
# declare, so a caller passing a list would fail on a dependency its own input never needs.
236+
if isinstance(inputs, list) and all(isinstance(i, MultimodalInput) for i in inputs):
237+
converted = inputs
238+
elif _is_batch_feature(inputs):
206239
logging.info(
207240
"Input is a BatchFeature, assuming it's coming from HF AutoProcessor.apply_chat_template(). Converting to multimodal inputs."
208241
)
209242
converted = _hf_to_multimodal_inputs(inputs, image_token_id=image_token_id)
210-
elif isinstance(inputs, list) and all(
211-
isinstance(i, MultimodalInput) for i in inputs
212-
):
213-
converted = inputs
214243
else:
215244
raise RuntimeError(
216245
"inputs must be either a BatchFeature (from HF AutoProcessor) or a list of MultimodalInput"

0 commit comments

Comments
 (0)