Not compatible with HF transformers >=5.0.0

[moritzharmel-doordash]

In our environment we use mosaic streaming and hugging face transformers libraries.

In order to run latest qwen models I need transformers >= 5.2.0
https://github.com/huggingface/transformers/releases/tag/v5.2.0

However, mosaic requires transformers<5 here
https://github.com/mosaicml/streaming/blame/6a4e12f66ee64d28bd4c5b308cec7fd825222205/setup.py#L55

This was set 4 years ago but probably is not a strict requirement.

Could you do some testing with newer transformers version and if it works remove the upper limit on the transformers version?

Thank you!

[miguelalba96]

same issue, this makes the framework unusable for new pre-trained models

[dmal-msft]

anyone from the authors to review this, there is an open PR about it.

[miguelalba96]

in theory you can bypass this dependency issue by overriding the dependencies I did this with uv:

default-groups = ["dev"]
override-dependencies = ["transformers>=5.2.0"]