Skip to content

fix(glm): vendor a minja-compatible chat template — tool calling works, and the #1254 mechanism was wrong #1255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
noonghunna merged 1 commit into from
Sep 11, 2026
Merged

fix(glm): vendor a minja-compatible chat template — tool calling works, and the #1254 mechanism was wrong #1255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 36 additions & 17 deletions models/glm-5.3-flash/llamacpp-club3090/compose/dual/devquasar-q2k/moecache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,26 +89,34 @@
# Status: 🧪 Experimental
# Best for: long-context MoE serving experiments on the moe-cache engine
# Caveats:
# • ⛔ TOOL CALLING IS BROKEN ON THIS ENGINE — template/minja, not the quant.
# Any request carrying `tools` returns HTTP 400 "Unable to generate parser for
# this template. Automatic parser generation failed: While executing
# CallExpression at line 163, column 57". llama.cpp builds a tool parser by
# statically walking the chat template; column 57 of line 163 is the `(` of
# `_args.items()`, which minja cannot evaluate during that pass. It fails at
# template-COMPILE time, so it is deterministic on every tools request, and the
# downstream symptom is `<tool_call>` left inline in content with `tool_calls[]`
# empty. Reported club-3090#1250 (@paulp83).
# • ✅ TOOL CALLING WORKS — via the vendored template override wired below.
# ⚠️ It does NOT work on the GGUF's own embedded template: every request carrying
# `tools` returns HTTP 400 "Unable to generate parser for this template …
# CallExpression at line 163, column 57", and /props reports supports_tools:false.
# Reported club-3090#1250 (@paulp83) and upstream ggml-org/llama.cpp#27754.
# ⛔⛔ THE LINE IN THAT ERROR IS THE CRASH SITE, NOT THE CAUSE. Two earlier readings
# of this bug were WRONG and are recorded so nobody re-derives them:
# ❌ "minja cannot evaluate .items()" — GLM-4.7-Flash.jinja has the identical
# construct, is exercised by llama.cpp's own autoparser tests, and passes.
# ❌ "the caps probe under-detects supports_object_arguments because the template
# iterates instead of indexing" — adding a named access changes nothing; the
# probe never gets that far.
# ⭐ ACTUAL CAUSE: minja does not implement NUMERIC DOTTED ATTRIBUTE ACCESS (`x.0`).
# Jinja2 defines `x.0` as `x[0]`; minja parses it as a non-computed member whose
# property is a number literal and throws. The GGUF template uses it in 4 places, so
# the caps probe throws, infers NOTHING, and leaves every capability false —
# including supports_object_arguments, which gates the JSON-string -> object
# conversion of tool arguments (chat.cpp). Arguments therefore stay a string, and
# the autoparser then dies on `.items()` at line 163. That is why supports_string_content
# is false too, which no tools-only explanation accounts for.
# ⇒ The override is the vendor template with `.0.` -> `[0].` in 4 places plus a
# null-tool guard. Wire format UNCHANGED. Verified with llama.cpp's own
# llama-template-analysis: all four capabilities flip false -> true.
# ⚠️ NOT club-3090#1195 — that is an INTERMITTENT 500 parsing the model's OUTPUT
# ("Failed to parse tool call arguments as JSON", 2/25 turns, payload-dependent).
# Same subsystem, opposite end. Do not merge the two.
# ⚠️ The vendor's CURRENT template does NOT fix it: zai-org/GLM-5.3-Flash
# chat_template.jinja (last modified 2026-09-07) still carries `_args.items()`
# at line 163, byte-identical. It DOES fix other things (null-safe content,
# `~` concat, break guards in the sort loops), so vendoring it is worth doing —
# just not as a fix for this.
# ⇒ Everything NOT passing `tools` is unaffected (completion, streaming,
# reasoning, long-context all pass). Use SKIP_TOOLS=1 with verify-full to get
# the rest of the run without the tool checks dominating it.
# ⚠️ NOT live-booted: verified by static analysis + the engine's own autoparser, not
# by serving 140 GiB end-to-end. Detail: patches/glm53-minja-numeric-index/README.md
# • ⚠️ ATTRIBUTION: the MoE expert cache is **leloch's** work
# (github.com/leloch/llama.cpp, RFC ggml-org#24528). This slug packages
# it; it does not originate it.
Expand Down Expand Up @@ -239,6 +247,9 @@ services:
- "${BIND_HOST:-0.0.0.0}:${ESTATE_PORT:-${PORT:-8130}}:8080"
volumes:
- "${MODEL_DIR:-../../../../../../models-cache}:/models:ro"
# minja-compatible chat template — WITHOUT this every tools request 400s and
# /props reports supports_tools:false. See patches/glm53-minja-numeric-index/README.md
- ../../../patches/glm53-minja-numeric-index/chat_template.jinja:/etc/glm53-chat-template.jinja:ro
environment:
# Drafter toggle - docker forwards only what is declared here.
- SPEC_N=${SPEC_N:-}
Expand Down Expand Up @@ -627,6 +638,14 @@ services:
- '--reasoning-effort'
- '${REASONING_EFFORT:-max}'
# Route thoughts to message.reasoning_content (never inline in content).
# ⭐ minja cannot parse `x.0` (numeric dotted attribute access); the GGUF's own
# template uses it in 4 places, which makes the jinja caps probe throw and report
# EVERY capability false — so tool arguments are never converted from JSON string
# to object and the autoparser dies. This override is the vendor template with
# `.0.` -> `[0].` (semantics-identical in Jinja2) + a null-tool guard. Wire format
# unchanged. club-3090#1250.
- '--chat-template-file'
- '/etc/glm53-chat-template.jinja'
- '--reasoning-format'
- '${REASONING_FORMAT:-deepseek}'
# ✅ What DOES work is not paying for it twice. This GGUF's template sets
Expand Down
53 changes: 36 additions & 17 deletions models/glm-5.3-flash/llamacpp-club3090/compose/dual/devquasar-q2k/offload.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,26 +270,34 @@
# Best for: ⭐ CONSTRAINED-HOST-RAM boxes — ~78 GB here vs the moecache
# sibling's ~112 GB. Same weights, same engine, no expert pool.
# Caveats:
# • ⛔ TOOL CALLING IS BROKEN ON THIS ENGINE — template/minja, not the quant.
# Any request carrying `tools` returns HTTP 400 "Unable to generate parser for
# this template. Automatic parser generation failed: While executing
# CallExpression at line 163, column 57". llama.cpp builds a tool parser by
# statically walking the chat template; column 57 of line 163 is the `(` of
# `_args.items()`, which minja cannot evaluate during that pass. It fails at
# template-COMPILE time, so it is deterministic on every tools request, and the
# downstream symptom is `<tool_call>` left inline in content with `tool_calls[]`
# empty. Reported club-3090#1250 (@paulp83).
# • ✅ TOOL CALLING WORKS — via the vendored template override wired below.
# ⚠️ It does NOT work on the GGUF's own embedded template: every request carrying
# `tools` returns HTTP 400 "Unable to generate parser for this template …
# CallExpression at line 163, column 57", and /props reports supports_tools:false.
# Reported club-3090#1250 (@paulp83) and upstream ggml-org/llama.cpp#27754.
# ⛔⛔ THE LINE IN THAT ERROR IS THE CRASH SITE, NOT THE CAUSE. Two earlier readings
# of this bug were WRONG and are recorded so nobody re-derives them:
# ❌ "minja cannot evaluate .items()" — GLM-4.7-Flash.jinja has the identical
# construct, is exercised by llama.cpp's own autoparser tests, and passes.
# ❌ "the caps probe under-detects supports_object_arguments because the template
# iterates instead of indexing" — adding a named access changes nothing; the
# probe never gets that far.
# ⭐ ACTUAL CAUSE: minja does not implement NUMERIC DOTTED ATTRIBUTE ACCESS (`x.0`).
# Jinja2 defines `x.0` as `x[0]`; minja parses it as a non-computed member whose
# property is a number literal and throws. The GGUF template uses it in 4 places, so
# the caps probe throws, infers NOTHING, and leaves every capability false —
# including supports_object_arguments, which gates the JSON-string -> object
# conversion of tool arguments (chat.cpp). Arguments therefore stay a string, and
# the autoparser then dies on `.items()` at line 163. That is why supports_string_content
# is false too, which no tools-only explanation accounts for.
# ⇒ The override is the vendor template with `.0.` -> `[0].` in 4 places plus a
# null-tool guard. Wire format UNCHANGED. Verified with llama.cpp's own
# llama-template-analysis: all four capabilities flip false -> true.
# ⚠️ NOT club-3090#1195 — that is an INTERMITTENT 500 parsing the model's OUTPUT
# ("Failed to parse tool call arguments as JSON", 2/25 turns, payload-dependent).
# Same subsystem, opposite end. Do not merge the two.
# ⚠️ The vendor's CURRENT template does NOT fix it: zai-org/GLM-5.3-Flash
# chat_template.jinja (last modified 2026-09-07) still carries `_args.items()`
# at line 163, byte-identical. It DOES fix other things (null-safe content,
# `~` concat, break guards in the sort loops), so vendoring it is worth doing —
# just not as a fix for this.
# ⇒ Everything NOT passing `tools` is unaffected (completion, streaming,
# reasoning, long-context all pass). Use SKIP_TOOLS=1 with verify-full to get
# the rest of the run without the tool checks dominating it.
# ⚠️ NOT live-booted: verified by static analysis + the engine's own autoparser, not
# by serving 140 GiB end-to-end. Detail: patches/glm53-minja-numeric-index/README.md
# • ⚠️ ATTRIBUTION: the MoE expert cache is **leloch's** work
# (github.com/leloch/llama.cpp, RFC ggml-org#24528). This slug packages
# it; it does not originate it.
Expand Down Expand Up @@ -420,6 +428,9 @@ services:
- "${BIND_HOST:-0.0.0.0}:${ESTATE_PORT:-${PORT:-8136}}:8080"
volumes:
- "${MODEL_DIR:-../../../../../../models-cache}:/models:ro"
# minja-compatible chat template — WITHOUT this every tools request 400s and
# /props reports supports_tools:false. See patches/glm53-minja-numeric-index/README.md
- ../../../patches/glm53-minja-numeric-index/chat_template.jinja:/etc/glm53-chat-template.jinja:ro
environment:
# Drafter toggle - docker forwards only what is declared here.
- SPEC_N=${SPEC_N:-}
Expand Down Expand Up @@ -808,6 +819,14 @@ services:
- '--reasoning-effort'
- '${REASONING_EFFORT:-max}'
# Route thoughts to message.reasoning_content (never inline in content).
# ⭐ minja cannot parse `x.0` (numeric dotted attribute access); the GGUF's own
# template uses it in 4 places, which makes the jinja caps probe throw and report
# EVERY capability false — so tool arguments are never converted from JSON string
# to object and the autoparser dies. This override is the vendor template with
# `.0.` -> `[0].` (semantics-identical in Jinja2) + a null-tool guard. Wire format
# unchanged. club-3090#1250.
- '--chat-template-file'
- '/etc/glm53-chat-template.jinja'
- '--reasoning-format'
- '${REASONING_FORMAT:-deepseek}'
# ✅ What DOES work is not paying for it twice. This GGUF's template sets
Expand Down
53 changes: 36 additions & 17 deletions models/glm-5.3-flash/llamacpp-club3090/compose/dual/devquasar-q3km/moecache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,26 +87,34 @@
# Status: 🧪 Experimental
# Best for: long-context MoE serving experiments on the moe-cache engine
# Caveats:
# • ⛔ TOOL CALLING IS BROKEN ON THIS ENGINE — template/minja, not the quant.
# Any request carrying `tools` returns HTTP 400 "Unable to generate parser for
# this template. Automatic parser generation failed: While executing
# CallExpression at line 163, column 57". llama.cpp builds a tool parser by
# statically walking the chat template; column 57 of line 163 is the `(` of
# `_args.items()`, which minja cannot evaluate during that pass. It fails at
# template-COMPILE time, so it is deterministic on every tools request, and the
# downstream symptom is `<tool_call>` left inline in content with `tool_calls[]`
# empty. Reported club-3090#1250 (@paulp83).
# • ✅ TOOL CALLING WORKS — via the vendored template override wired below.
# ⚠️ It does NOT work on the GGUF's own embedded template: every request carrying
# `tools` returns HTTP 400 "Unable to generate parser for this template …
# CallExpression at line 163, column 57", and /props reports supports_tools:false.
# Reported club-3090#1250 (@paulp83) and upstream ggml-org/llama.cpp#27754.
# ⛔⛔ THE LINE IN THAT ERROR IS THE CRASH SITE, NOT THE CAUSE. Two earlier readings
# of this bug were WRONG and are recorded so nobody re-derives them:
# ❌ "minja cannot evaluate .items()" — GLM-4.7-Flash.jinja has the identical
# construct, is exercised by llama.cpp's own autoparser tests, and passes.
# ❌ "the caps probe under-detects supports_object_arguments because the template
# iterates instead of indexing" — adding a named access changes nothing; the
# probe never gets that far.
# ⭐ ACTUAL CAUSE: minja does not implement NUMERIC DOTTED ATTRIBUTE ACCESS (`x.0`).
# Jinja2 defines `x.0` as `x[0]`; minja parses it as a non-computed member whose
# property is a number literal and throws. The GGUF template uses it in 4 places, so
# the caps probe throws, infers NOTHING, and leaves every capability false —
# including supports_object_arguments, which gates the JSON-string -> object
# conversion of tool arguments (chat.cpp). Arguments therefore stay a string, and
# the autoparser then dies on `.items()` at line 163. That is why supports_string_content
# is false too, which no tools-only explanation accounts for.
# ⇒ The override is the vendor template with `.0.` -> `[0].` in 4 places plus a
# null-tool guard. Wire format UNCHANGED. Verified with llama.cpp's own
# llama-template-analysis: all four capabilities flip false -> true.
# ⚠️ NOT club-3090#1195 — that is an INTERMITTENT 500 parsing the model's OUTPUT
# ("Failed to parse tool call arguments as JSON", 2/25 turns, payload-dependent).
# Same subsystem, opposite end. Do not merge the two.
# ⚠️ The vendor's CURRENT template does NOT fix it: zai-org/GLM-5.3-Flash
# chat_template.jinja (last modified 2026-09-07) still carries `_args.items()`
# at line 163, byte-identical. It DOES fix other things (null-safe content,
# `~` concat, break guards in the sort loops), so vendoring it is worth doing —
# just not as a fix for this.
# ⇒ Everything NOT passing `tools` is unaffected (completion, streaming,
# reasoning, long-context all pass). Use SKIP_TOOLS=1 with verify-full to get
# the rest of the run without the tool checks dominating it.
# ⚠️ NOT live-booted: verified by static analysis + the engine's own autoparser, not
# by serving 140 GiB end-to-end. Detail: patches/glm53-minja-numeric-index/README.md
# • ⚠️ ATTRIBUTION: the MoE expert cache is **leloch's** work
# (github.com/leloch/llama.cpp, RFC ggml-org#24528). This slug packages
# it; it does not originate it.
Expand Down Expand Up @@ -237,6 +245,9 @@ services:
- "${BIND_HOST:-0.0.0.0}:${ESTATE_PORT:-${PORT:-8133}}:8080"
volumes:
- "${MODEL_DIR:-../../../../../../models-cache}:/models:ro"
# minja-compatible chat template — WITHOUT this every tools request 400s and
# /props reports supports_tools:false. See patches/glm53-minja-numeric-index/README.md
- ../../../patches/glm53-minja-numeric-index/chat_template.jinja:/etc/glm53-chat-template.jinja:ro
environment:
# Drafter toggle - docker forwards only what is declared here.
- SPEC_N=${SPEC_N:-}
Expand Down Expand Up @@ -625,6 +636,14 @@ services:
- '--reasoning-effort'
- '${REASONING_EFFORT:-max}'
# Route thoughts to message.reasoning_content (never inline in content).
# ⭐ minja cannot parse `x.0` (numeric dotted attribute access); the GGUF's own
# template uses it in 4 places, which makes the jinja caps probe throw and report
# EVERY capability false — so tool arguments are never converted from JSON string
# to object and the autoparser dies. This override is the vendor template with
# `.0.` -> `[0].` (semantics-identical in Jinja2) + a null-tool guard. Wire format
# unchanged. club-3090#1250.
- '--chat-template-file'
- '/etc/glm53-chat-template.jinja'
- '--reasoning-format'
- '${REASONING_FORMAT:-deepseek}'
# ✅ What DOES work is not paying for it twice. This GGUF's template sets
Expand Down
Loading