Skip to content

Commit cc7b4ec

Browse files
barhancmsluszniak
andauthored
feat: add enable_thinking option to template rendering and new MiniCPM5-2B model (#1571)
## Description Adds option to switch-on/off thinking (for chat templates that respect that setting) and new MiniCPM5-2B https://huggingface.co/openbmb/MiniCPM5-2B as a substitute for Qwen 3.5 whose hybrid architecture doesn't work well with our runtime. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [ ] Android ### Testing instructions - [ ] Run the NLP example app and test new LLM - [ ] Check the HF repo https://huggingface.co/software-mansion/react-native-executorch-minicpm-5 ### Screenshots <!-- Add screenshots here, if applicable --> ### Related issues Closes #935 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes <!-- Include any additional information, assumptions, or context that reviewers might need to understand this PR. --> --------- Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>
1 parent 3cdbaf4 commit cc7b4ec

9 files changed

Lines changed: 322 additions & 15 deletions

File tree

‎.cspell-wordlist.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ execu
1616
Execu
1717
torch
1818
huggingface
19+
apos
1920
bbox
2021
bboxes
2122
chatbots
@@ -33,6 +34,7 @@ microcontrollers
3334
notimestamps
3435
seqs
3536
smollm
37+
minicpm
3638
llms
3739
qwen
3840
XNNPACK

‎apps/nlp/app/llm/index.tsx‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import {
1212
Alert,
1313
Image as RNImage,
1414
Modal,
15+
Switch,
1516
} from 'react-native';
1617
import { Skia } from '@shopify/react-native-skia';
1718
import RNBlobUtil from 'react-native-blob-util';
@@ -22,7 +23,12 @@ import ScreenWrapper from '../../components/ScreenWrapper';
2223
import { ModelPicker, type ModelOption } from '../../components/ModelPicker';
2324
import { Button } from '../../components/Button';
2425
import { getImage, skImageToBuffer } from '../../utils';
25-
import { LLM_MODELS, type LLMModelConfig, type LLMBackend } from '../../constants/llm';
26+
import {
27+
LLM_MODELS,
28+
parseMiniCpmToolCalls,
29+
type LLMModelConfig,
30+
type LLMBackend,
31+
} from '../../constants/llm';
2632

2733
type Turn = {
2834
role: 'user' | 'assistant' | 'tool';
@@ -93,12 +99,23 @@ function LLMContent() {
9399
[activeModel]
94100
);
95101

102+
const [enableThinking, setEnableThinking] = useState(false);
103+
const thinkingRef = useRef(enableThinking);
104+
const supportsThinkingSwitch = activeModel.supportsThinkingSwitch !== false;
105+
const toolOpts = useMemo(() => {
106+
if (activeModel.toolOpts?.parseToolCalls !== parseMiniCpmToolCalls) return activeModel.toolOpts;
107+
return {
108+
...activeModel.toolOpts,
109+
parseToolCalls: (text: string) => parseMiniCpmToolCalls(text, thinkingRef.current),
110+
};
111+
}, [activeModel]);
112+
96113
const { isReady, downloadProgress, error, sendMessage, stop, resource, getKVCacheState } =
97114
useLLMChatSession(activeModel.model, {
98115
initialMessages,
99116
generationConfig: activeModel.generationConfig,
100117
stopRegex: activeModel.stopRegex,
101-
toolOpts: activeModel.toolOpts,
118+
toolOpts,
102119
preventLoad: !isDownloadStarted,
103120
});
104121

@@ -286,9 +303,13 @@ function LLMContent() {
286303
payload = textMessage;
287304
}
288305

289-
const result = await sendMessage(payload, (token) => {
290-
setStreamingResponse((prev) => (prev !== null ? prev + token : token));
291-
});
306+
const result = await sendMessage(
307+
payload,
308+
(token) => {
309+
setStreamingResponse((prev) => (prev !== null ? prev + token : token));
310+
},
311+
{ enableThinking: supportsThinkingSwitch ? enableThinking : undefined }
312+
);
292313

293314
let assistantIdx = 0;
294315
const generatedTurns: Turn[] = result.messages
@@ -343,6 +364,20 @@ function LLMContent() {
343364
selectedValue={selectedModelId}
344365
onValueChange={handleModelChange}
345366
/>
367+
<View style={styles.thinkingRow}>
368+
<Text style={styles.thinkingLabel}>
369+
{supportsThinkingSwitch ? 'Thinking' : 'Thinking (always on)'}
370+
</Text>
371+
<Switch
372+
accessibilityLabel="Thinking"
373+
value={supportsThinkingSwitch ? enableThinking : true}
374+
onValueChange={(value) => {
375+
thinkingRef.current = value;
376+
setEnableThinking(value);
377+
}}
378+
disabled={isGenerating || !supportsThinkingSwitch}
379+
/>
380+
</View>
346381
{resource && (
347382
<TouchableOpacity
348383
style={styles.unlinkButton}
@@ -674,6 +709,13 @@ const styles = StyleSheet.create({
674709
marginBottom: 8,
675710
marginLeft: 4,
676711
},
712+
thinkingRow: {
713+
flexDirection: 'row',
714+
alignItems: 'center',
715+
justifyContent: 'space-between',
716+
marginBottom: 8,
717+
},
718+
thinkingLabel: { color: '#212529', fontSize: 14, fontWeight: '600' },
677719
unlinkButtonText: {
678720
color: '#e03131',
679721
fontSize: 12,

‎apps/nlp/constants/llm.ts‎

Lines changed: 112 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,80 @@ export function parseLfmToolCalls(text: string): llm.ToolParserResult | undefine
284284
return { toolCalls, textContent: textContent || undefined };
285285
}
286286

287+
function decodeMiniCpmArgument(value: string): string {
288+
const trimmed = value.trim();
289+
if (trimmed.startsWith('<![CDATA[') && trimmed.endsWith(']]>')) {
290+
return trimmed.slice(9, -3);
291+
}
292+
const entities: Record<string, string> = {
293+
amp: '&',
294+
lt: '<',
295+
gt: '>',
296+
quot: '"',
297+
apos: "'",
298+
};
299+
return trimmed.replace(/&(#x[\da-f]+|#\d+|amp|lt|gt|quot|apos);/gi, (_, entity: string) => {
300+
if (entity.startsWith('#')) {
301+
const code = entity.slice(1).toLowerCase();
302+
return String.fromCodePoint(
303+
code.startsWith('x') ? parseInt(code.slice(1), 16) : Number(code)
304+
);
305+
}
306+
return entities[entity.toLowerCase()]!;
307+
});
308+
}
309+
310+
export function parseMiniCpmToolCalls(
311+
text: string,
312+
enableThinking = true
313+
): llm.ToolParserResult | undefined {
314+
const thinkingEnd = text.lastIndexOf('</think>');
315+
if (thinkingEnd === -1 && (enableThinking || text.includes('<think>'))) return undefined;
316+
const answer = thinkingEnd === -1 ? text : text.slice(thinkingEnd + '</think>'.length);
317+
const callRegex =
318+
/<function\s+name=(["'])([^"']+)\1\s*>((?:<!\[CDATA\[[\s\S]*?\]\]>|[\s\S])*?)<\/function>/g;
319+
const toolCalls: llm.ToolCall[] = [];
320+
321+
for (const match of answer.matchAll(callRegex)) {
322+
const name = match[2]!;
323+
const body = match[3]!;
324+
const tool = TOOLS.find((candidate) => candidate.function.name === name);
325+
if (!tool) return undefined;
326+
const params = tool.function.parameters;
327+
const paramRegex =
328+
/<param\s+name=(["'])([^"']+)\1\s*>((?:<!\[CDATA\[[\s\S]*?\]\]>|[\s\S])*?)<\/param>/g;
329+
const args: Record<string, unknown> = {};
330+
331+
try {
332+
for (const param of body.matchAll(paramRegex)) {
333+
const key = param[2]!;
334+
const schema = params?.properties?.[key] as { type?: string } | undefined;
335+
if (!schema || Object.prototype.hasOwnProperty.call(args, key)) return undefined;
336+
const value = decodeMiniCpmArgument(param[3]!);
337+
const parsed = schema.type === 'string' ? value : JSON.parse(value);
338+
if (schema.type === 'number' && (typeof parsed !== 'number' || !Number.isFinite(parsed))) {
339+
return undefined;
340+
}
341+
args[key] = parsed;
342+
}
343+
} catch {
344+
return undefined;
345+
}
346+
if (body.replace(paramRegex, '').trim()) return undefined;
347+
if (params?.required?.some((key) => !Object.prototype.hasOwnProperty.call(args, key))) {
348+
return undefined;
349+
}
350+
toolCalls.push({ type: 'function', function: { name, arguments: args } });
351+
}
352+
353+
if (toolCalls.length === 0) return undefined;
354+
const textContent = answer
355+
.replace(callRegex, '')
356+
.replace(/<\|im_end\|>/g, '')
357+
.trim();
358+
return { toolCalls, textContent: textContent || undefined };
359+
}
360+
287361
export const QWEN_TOOL_STOP_REGEX = /(?:<\/tool_call>|<\|im_end\|>)/;
288362

289363
export function parseQwenToolCalls(text: string): llm.ToolParserResult | undefined {
@@ -375,6 +449,14 @@ export function parseLlamaToolCalls(text: string): llm.ToolParserResult | undefi
375449

376450
// --- Model Family Generation Configurations ---
377451

452+
/** Sampling defaults recommended by the MiniCPM5-2B model card. */
453+
export const MINICPM5_GENERATION_CONFIG: llm.LLMGenerationConfig = {
454+
temperature: 1.0,
455+
topP: 0.95,
456+
minP: 0.0,
457+
maxNewTokens: 512,
458+
};
459+
378460
/**
379461
* Liquid AI LFM 2.5 recommended generation parameters.
380462
* Sampling parameters are specified in the official Liquid AI GGUF model cards
@@ -532,6 +614,11 @@ export const LFM_TOOL_OPTS: LLMToolOpts = {
532614
parseToolCalls: parseLfmToolCalls,
533615
};
534616

617+
export const MINICPM5_TOOL_OPTS: LLMToolOpts = {
618+
tools: TOOLS,
619+
parseToolCalls: parseMiniCpmToolCalls,
620+
};
621+
535622
export const GEMMA_TOOL_OPTS: LLMToolOpts = {
536623
tools: TOOLS,
537624
parseToolCalls: parseGemmaToolCalls,
@@ -566,14 +653,37 @@ export interface LLMModelConfig {
566653
stopRegex?: RegExp;
567654
toolOpts: LLMToolOpts | undefined;
568655
iosOnly?: boolean;
656+
supportsThinkingSwitch?: boolean;
569657
}
570658

571659
// --- Quantized LLM Models List ---
572660

573661
export const LLM_MODELS: LLMModelConfig[] = [
662+
// MiniCPM5
663+
{
664+
id: 'minicpm-5-2b-xnnpack',
665+
name: 'MiniCPM5 2B',
666+
backend: 'xnnpack',
667+
model: models.llm.MINICPM5_2B.XNNPACK_8DA4W,
668+
generationConfig: MINICPM5_GENERATION_CONFIG,
669+
systemPrompt: `${DEFAULT_SYSTEM_PROMPT}. Always reply in the language of the user's latest message.`,
670+
toolOpts: MINICPM5_TOOL_OPTS,
671+
},
672+
{
673+
id: 'minicpm-5-2b-mlx',
674+
name: 'MiniCPM5 2B',
675+
backend: 'mlx',
676+
model: models.llm.MINICPM5_2B.MLX_INT4,
677+
generationConfig: MINICPM5_GENERATION_CONFIG,
678+
systemPrompt: `${DEFAULT_SYSTEM_PROMPT}. Always reply in the language of the user's latest message.`,
679+
toolOpts: MINICPM5_TOOL_OPTS,
680+
iosOnly: true,
681+
},
682+
574683
// LFM 2.5 family
575684
{
576685
id: 'lfm-2.5-2.6b-xnnpack',
686+
supportsThinkingSwitch: false,
577687
name: 'LFM 2.5 2.6B',
578688
backend: 'xnnpack',
579689
model: models.llm.LFM2_5_2_6B.XNNPACK_8DA4W,
@@ -584,6 +694,7 @@ export const LLM_MODELS: LLMModelConfig[] = [
584694
},
585695
{
586696
id: 'lfm-2.5-2.6b-mlx',
697+
supportsThinkingSwitch: false,
587698
name: 'LFM 2.5 2.6B',
588699
backend: 'mlx',
589700
model: models.llm.LFM2_5_2_6B.MLX_INT4,
@@ -1107,7 +1218,7 @@ export const LLM_MODELS: LLMModelConfig[] = [
11071218
backend: 'xnnpack',
11081219
model: models.llm.BIELIK_V3_1_5B.XNNPACK_8DA4W,
11091220
generationConfig: BIELIK_V3_GENERATION_CONFIG,
1110-
// eslint-disable-next-line @cspell/spellchecker
1221+
// cspell:disable-next-line
11111222
systemPrompt: 'Jesteś pomocnym asystentem AI mówiącym po polsku.',
11121223
stopRegex: undefined,
11131224
toolOpts: undefined,

‎docs/docs/02-extensions/natural-language/02-llm-chat-and-generation.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,21 @@ type LLMChatTurnResult = {
151151

152152
Under the hood, [`createChatPreprocessor`](../../06-api-reference/react-native-executorch/namespaces/llm/functions/createChatPreprocessor.md) renders [`ChatMessage[]`](../../06-api-reference/react-native-executorch/namespaces/llm/type-aliases/ChatMessage.md) arrays using the model's official Jinja2 template from `tokenizer_config.json` (formatting special tokens, roles, and generation headers).
153153

154+
For models that support answering without thinking, set
155+
`generationConfig: { enableThinking: false }` in `useLLMChatSession` or
156+
`createLLMChatSession`. Override it per turn with
157+
`session.sendMessage(message, onToken, { enableThinking: true })`. Direct
158+
preprocessor users can pass `enableThinking` in the options to `render()` or
159+
`process()`.
160+
161+
This exposes `enable_thinking` to Jinja. The template must honor this switch;
162+
a template that always appends `<think>` needs to be corrected separately. Omit
163+
the option to preserve the template's default. The raw runner accepts already
164+
formatted prompts and ignores this field. When thinking is disabled, tool parsers
165+
must accept answers without a generated `</think>` tag. Switching thinking modes
166+
between turns assumes the template preserves previously cached messages; use
167+
`resetOnTurn: true` if the switch also changes how earlier messages are rendered.
168+
154169
To keep multi-turn chat responsive without re-encoding past history on every message, the preprocessor uses incremental prompt diffing:
155170

156171
1. It verifies that the rendered prefix of previously committed turns is an exact substring match of the newly updated conversation.
@@ -381,6 +396,7 @@ The library provides ready-to-use models from the [Software Mansion HuggingFace
381396
| **Microsoft Phi-4 Mini** | [`3.8B`](../../06-api-reference/variables/models.md#llmphi4_mini) | 2.62 GB – 7.15 GB | XNNPACK (CPU), MLX (Apple) | High-density reasoning model for STEM problem solving & coding. |
382397
| **HuggingFace SmolLM2** | [`135M`](../../06-api-reference/variables/models.md#llmsmollm2_135m), [`360M`](../../06-api-reference/variables/models.md#llmsmollm2_360m), [`1.7B`](../../06-api-reference/variables/models.md#llmsmollm2_1_7b) | 158 MB – 1.83 GB | XNNPACK (CPU), MLX (Apple) | Smallest footprint family, for constrained devices and fast drafting. |
383398
| **SpeakLeash Bielik v3** | [`1.5B`](../../06-api-reference/variables/models.md#llmbielik_v3_1_5b) | 923 MB – 2.97 GB | XNNPACK (CPU) | Bilingual Polish & English instruction model. |
399+
| **OpenBMB MiniCPM5** | [`2B`](../../06-api-reference/variables/models.md#llmminicpm5_2b) | 1.36 GB – 1.42 GB | XNNPACK (CPU), MLX (Apple) | Compact reasoning model with thinking and tool calling. |
384400

385401
:::tip Using Custom Models
386402
To use your own fine-tuned LLM `.pte` model, pass an [`LLMModel`](../../06-api-reference/type-aliases/LLMModel.md) configuration object to [`useLLMChatSession`](../../06-api-reference/functions/useLLMChatSession.md) or [`createLLMChatSession`](../../06-api-reference/functions/createLLMChatSession.md):

0 commit comments

Comments
 (0)