Skip to content

Commit 25fb705

Browse files
committed
documentation update
1 parent b523e57 commit 25fb705

6 files changed

Lines changed: 1527 additions & 22 deletions

File tree

‎docs/docs/03-hooks/01-natural-language-processing/useLLM.md‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -494,13 +494,13 @@ Depending on selected model and the user's device generation speed can be above
494494

495495
## Vision-Language Models (VLM)
496496

497-
Some models support multimodal input — text and images together. To use them, pass a `capabilities` array when loading the model.
497+
Some models support multimodal input — text, images and/or audio together. To use them, pass a `capabilities` array when loading the model.
498498

499499
### Loading a VLM
500500

501501
```tsx
502502
import { models, useLLM } from 'react-native-executorch';
503-
const llm = useLLM({ model: models.llm.lfm2_5_vl_1_6b() });
503+
const llm = useLLM({ model: models.llm.gemma4_e2b_multimodal() });
504504
```
505505

506506
The `capabilities` field is already set on the model constant. You can also construct the model object explicitly:
@@ -511,22 +511,26 @@ const llm = useLLM({
511511
modelSource: '...',
512512
tokenizerSource: '...',
513513
tokenizerConfigSource: '...',
514-
capabilities: ['vision'],
514+
capabilities: ['vision', 'audio'],
515515
},
516516
});
517517
```
518518

519519
Passing `capabilities` unlocks the typed `media` argument on `sendMessage`.
520520

521-
### Sending a message with an image
521+
### Sending a message with an image or audio recording
522522

523523
```tsx
524-
const llm = useLLM({ model: models.llm.lfm2_5_vl_1_6b() });
524+
const llm = useLLM({ model: models.llm.gemma4_e2b_multimodal() });
525525

526526
const send = () => {
527527
llm.sendMessage('What is in this image?', {
528528
imagePath: '/path/to/image.jpg',
529529
});
530+
// or
531+
llm.sendMessage('What can you hear?', {
532+
audioBuffer: audioRecording,
533+
});
530534
};
531535

532536
return (
@@ -538,6 +542,7 @@ return (
538542
```
539543

540544
The `imagePath` should be a local file path on the device.
545+
The `audioBuffer` should be a `Float32Array` with 16kHz waveform.
541546

542547
### Functional generation with images
543548

‎docs/docs/04-typescript-api/01-natural-language-processing/LLMModule.md‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -118,12 +118,12 @@ Model presets expose an optional `generationConfig` that `LLMModule.fromModelNam
118118

119119
## Vision-Language Models (VLM)
120120

121-
Some models support multimodal input — text and images together. To use them, pass `capabilities` in the model object when calling [`fromModelName`](../../06-api-reference/classes/LLMModule.md#frommodelname):
121+
Some models support multimodal input — text, images and/or audio together. To use them, pass `capabilities` in the model object when calling [`fromModelName`](../../06-api-reference/classes/LLMModule.md#frommodelname):
122122

123123
```typescript
124124
import { models, LLMModule } from 'react-native-executorch';
125125
const llm = await LLMModule.fromModelName(
126-
models.llm.lfm2_5_vl_1_6b(),
126+
models.llm.gemma4_e2b_multimodal(),
127127
undefined,
128128
(token) => console.log(token)
129129
);
@@ -133,20 +133,24 @@ The `capabilities` field is already set on the model constant. You can also cons
133133

134134
```typescript
135135
const llm = await LLMModule.fromModelName({
136-
modelName: 'lfm2.5-vl-1.6b-quantized',
136+
modelName: 'gemma4-e2b-multimodal',
137137
modelSource: require('./path/to/model.pte'),
138138
tokenizerSource: require('./path/to/tokenizer.json'),
139139
tokenizerConfigSource: require('./path/to/tokenizer_config.json'),
140-
capabilities: ['vision'],
140+
capabilities: ['vision', 'audio'],
141141
});
142142
```
143143

144-
Once loaded, pass `imagePath` to [`sendMessage`](../../06-api-reference/classes/LLMModule.md#sendmessage):
144+
Once loaded, pass `imagePath`Once loaded, pass `imagePath` or `audioBuffer` to [`sendMessage`](../../06-api-reference/classes/LLMModule.md#sendmessage):
145145

146146
```typescript
147147
const response = await llm.sendMessage('What is in this image?', {
148148
imagePath: '/path/to/image.jpg',
149149
});
150+
// or
151+
const response = await llm.sendMessage('What can you hear?', {
152+
audioBuffer: audioRecording, //expected as waveform 16kHz
153+
});
150154
```
151155

152156
Or use [`generate`](../../06-api-reference/classes/LLMModule.md#generate) with `mediaPath` on the message:
@@ -159,7 +163,14 @@ const chat: Message[] = [
159163
mediaPath: '/path/to/image.jpg',
160164
},
161165
];
162-
166+
// or
167+
const chat: Message[] = [
168+
{
169+
role: 'user',
170+
content: 'Transcribe the recording.',
171+
audioWaveform: audioRecording,
172+
},
173+
];
163174
const response = await llm.generate(chat);
164175
```
165176

0 commit comments

Comments
 (0)