Repository navigation
RFC: Improve developer experience by anchoring on multimodal use-case聽#7093
Description
馃殌 The feature, motivation and pitch
Let's build an example demo app, perhaps in pytorch-labs, which will become a forcing function to improve developer experience from a user perspective. A positive outcome of this demo app is to define and build new higher level abstractions (e.g., similar to Pipelines).
On a high-level, here's the app we would like to build: LLM based on voice input and output. In terms of implementation, it's a three step process:
- Given a voice input, convert to text (e.g., Whisper)
- Run text based LLM (e.g., Llama 1B)
- Convert text output to voice (e.g., using T5)
Here are the requirements:
- Be able to run on iOS, Android and Desktop app.
- Be able to prototype e2e flow in Python first and HuggingFace
- Be able to deploy on laptop without Python runtime easily for testing purposes.
- Be able to swap underlying models easily (e.g., Whisper -> Seamless, Llama 1B -> Qwen)
- Easy to swap Sampler/Tokenizer/KVCache implementations in LLM, perhaps, use this issue
- Easy deployment process to mobile and desktop app.
- Everything in OSS
- Easy to improve performance optimization and debugging (e.g., use mobile accelerators, quantization)
Here's a positive outcome of this demo app:
- Define and build new higher level abstractions to make these possible.
- ExecuTorch and torchchat uses this abstraction for text-based LLMs.
- Llava and multimodal image uses this abstraction.
- Community can build completely new apps using this new abstractions
Alternatives
No response
Additional context
Already another RFC, but specifically in the context of LLMs
RFC (Optional)
No response
cc @cccclai @helunwencser @dvorjackz
Metadata
Metadata
Labels
Type
Projects
- Status
Activity
shoumikhin commentedon Nov 26, 2024
Another success story for iOS specifically (and hopefully for Android too) should look roughly like on this video, where the clients could just add some
executorch-llmSwift PM package, write a few lines to create a Pipeline using an exported .pte file, add a text-edit field and a button to UI, and then just run LLaMA inference out-the-box.mergennachin commentedon Nov 26, 2024
@shoumikhin
yeah, that's cool. two additional comments:
iseeyuan commentedon Nov 26, 2024
It's great to have this kind of experience in general, but we may need to think more on how the framework can really help. For multimodal we need to learn more on the common pattern. Note that different components may not work together directly out of box due to:
As a framework we may think about how we can help users to hook their components with a working and robust pipeline.
kimishpatel commentedon Dec 2, 2024
I think it would be great if the issue/pain points, solution space and potential way to validate this actually came from users a level higher than framework devs. My fear is that we will do incremental improvements that aesthetically please us based on our own experience. Even if such issues and/or solution space is not driven by other users or product engineers, such personas have to be close part of iterating over solution space. Maybe this would be people from Paris hackathon.
shoumikhin commentedon Dec 2, 2024
@kimishpatel I imagine if for users it looks similar to HF transformers or OpenAI API, that should be good enough?
kimishpatel commentedon Dec 2, 2024
WHat is OpenAI API?
Why do you believe users similar to HF transformers covers spans of users that interact with torchchat/ET? Any examples?
shoumikhin commentedon Dec 2, 2024
HF transformers or OpenAI API are sorta de-facto standards how devs and clients interact with LLMs these days. I guess if TC provides a similar interface it at least wouldn't be worse.
The real question is can we do better and what such better would be? Agree that's something the researchers or consumers can help us to define, along with our own iterations.
kimishpatel commentedon Dec 2, 2024
@shoumikhin OpenAI API, AFAIU, is related to end point APIs while building pipeline of components, with customizability across different aspects such as tokenizer, kv cache management, long context management etc. might be different. Dont know enough about HF in this space. I assume that would more closely align with some of the objectives here.
And generally @mergennachin, I would probably want to also understand how different end-users envision deploying models. Requirements listed here make sense but I cant place them in larger context and where they are coming from. @shoumikhin's comment regarding HF users do make sense though but does the same exist for on-device use-cases?