Skip to content

Commit 21cbce7

Browse files
ndemiancclaude
andcommitted
fix(spike): clear the probe timeout on failure; exact-42 verdict; doc ctx = 2^20
Addresses the PR #26 review (Copilot). - Stage 1's 60s timeout timer wasn't cleared if the initial fetch threw (DNS/auth) — the empty `finally` only "kept the timer for the stream". Wrapped fetch + the !ok return + the stream in a single try/finally that always calls `to.done()`, so the timer is cleared on a throw, an early return, or a normal end, and its abort can't fire spuriously. - The verdict used `/42/`, which false-positives on "142", "420", or prose like "…in 42 steps". Since the prompt asks for ONLY the number, tightened to `^\D*42\D*$` — 42 must be the sole number in the answer. Verified: accepts "42" / "42." / "= **42**"; rejects "142" / "420" / "1420" / mixed-digit prose. - docs/KIMI-K3.md said the context is "1,000,000 tokens" while the shipped caps (and their test) use 1,048,576 (2^20, what OpenRouter serves). Aligned the §2 table and the Path A caps example to 1,048,576 so a future edit can't "fix" the doc into breaking the meter alignment. Verified: node --check passes; the no-key path still exits 2 with guidance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 2015c79 commit 21cbce7

2 files changed

Lines changed: 28 additions & 23 deletions

File tree

‎docs/KIMI-K3.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ The gateway already ships a Moonshot flagship today — `GATEWAY_PRO_MODEL = 'mo
4141
| Model id | **`kimi-k3`** (canonical; "K3 Max" / "Swarm Max" are product names, not API strings) |
4242
| Streaming | Yes — emits **separate `reasoning_content` and `content`** deltas |
4343
| Tools | Yes — official tool-calling + dynamic-loading guides exist |
44-
| Context | 1,000,000 tokens |
44+
| Context | 1,048,576 tokens (2^20; Moonshot/OpenRouter market it as "1M") |
4545
| Pricing | ~$3 / M input, $15 / M output, $0.30 / M cached input *(secondary sources — confirm on the official pricing page before Path B billing)* |
4646

4747
**Two corrections to watch for:**
@@ -121,7 +121,7 @@ reasoning model — a poor ghost-text completer), no `agent: true` (appears clau
121121

122122
**A2 — per-model caps** · `providers/catalog.js` `CAPS` table (~`:27-56`):
123123
```js
124-
'kimi-k3': { context: 1000000, tools: true, reasoning: true },
124+
'kimi-k3': { context: 1048576, tools: true, reasoning: true }, // 2^20 — match the shipped caps exactly
125125
```
126126
`context` feeds the context meter; without it the agent falls back to 200 000 (`agent.js:618`) and would
127127
badly under-report K3's 1M window. Do **not** add `kimi-k3` to `FAST_COMPLETION` (`catalog.js:110-116`).

‎extensions/levelcode-ai/scripts/kimi-k3-spike.js‎

Lines changed: 26 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -71,30 +71,32 @@ async function stage1RawProbe() {
7171
};
7272
const tally = { content: 0, reasoning: 0, reasoning_content: 0, tool_calls: 0, thinkTag: false, sample: '' };
7373
const to = withTimeout(60000);
74-
let res;
7574
try {
76-
res = await fetch(BASE + '/chat/completions', {
75+
const res = await fetch(BASE + '/chat/completions', {
7776
method: 'POST', signal: to.signal,
7877
headers: Object.assign({ 'content-type': 'application/json', authorization: 'Bearer ' + KEY }, HEADERS),
7978
body: JSON.stringify(body)
8079
});
81-
} finally { /* keep timer until stream ends */ }
82-
if (!res.ok) { to.done(); return { ok: false, status: res.status, text: await res.text().catch(() => '') }; }
83-
await readLines(res, (line) => {
84-
const s = line.trim();
85-
if (!s.startsWith('data:')) { return; }
86-
const data = s.slice(5).trim();
87-
if (!data || data === '[DONE]') { return; }
88-
let ev; try { ev = JSON.parse(data); } catch { return; }
89-
const d = ev.choices && ev.choices[0] && ev.choices[0].delta;
90-
if (!d) { return; }
91-
if (typeof d.content === 'string' && d.content) { tally.content++; if (/<think\b/i.test(d.content)) { tally.thinkTag = true; } }
92-
if (d.reasoning != null && d.reasoning !== '') { tally.reasoning++; if (!tally.sample) { tally.sample = String(d.reasoning).slice(0, 60); } }
93-
if (d.reasoning_content != null && d.reasoning_content !== '') { tally.reasoning_content++; if (!tally.sample) { tally.sample = String(d.reasoning_content).slice(0, 60); } }
94-
if (Array.isArray(d.tool_calls) && d.tool_calls.length) { tally.tool_calls++; }
95-
});
96-
to.done();
97-
return { ok: true, tally };
80+
if (!res.ok) { return { ok: false, status: res.status, text: await res.text().catch(() => '') }; }
81+
await readLines(res, (line) => {
82+
const s = line.trim();
83+
if (!s.startsWith('data:')) { return; }
84+
const data = s.slice(5).trim();
85+
if (!data || data === '[DONE]') { return; }
86+
let ev; try { ev = JSON.parse(data); } catch { return; }
87+
const d = ev.choices && ev.choices[0] && ev.choices[0].delta;
88+
if (!d) { return; }
89+
if (typeof d.content === 'string' && d.content) { tally.content++; if (/<think\b/i.test(d.content)) { tally.thinkTag = true; } }
90+
if (d.reasoning != null && d.reasoning !== '') { tally.reasoning++; if (!tally.sample) { tally.sample = String(d.reasoning).slice(0, 60); } }
91+
if (d.reasoning_content != null && d.reasoning_content !== '') { tally.reasoning_content++; if (!tally.sample) { tally.sample = String(d.reasoning_content).slice(0, 60); } }
92+
if (Array.isArray(d.tool_calls) && d.tool_calls.length) { tally.tool_calls++; }
93+
});
94+
return { ok: true, tally };
95+
} finally {
96+
// Always clear the 60s timer — on a fetch throw (DNS/auth), an early !ok return, or a normal end.
97+
// Without this a pending timer could outlive the probe and its abort could fire spuriously.
98+
to.done();
99+
}
98100
}
99101

100102
/**
@@ -144,13 +146,16 @@ async function stage2RoundTrip() {
144146

145147
const finalText = t2.content.filter((b) => b.type === 'text').map((b) => b.text).join('').trim();
146148
console.log(' turn 2 stop_reason: ' + t2.stop_reason + ' · answer: ' + JSON.stringify((finalText || answer).slice(0, 80)));
147-
const looksRight = /42/.test(finalText || answer);
149+
// Exact numeric output, not a substring: the prompt asks for ONLY the number, so require 42 to be the
150+
// sole number in the answer. `/42/` would false-positive on "142", "420", or "…in 42 steps"; `^\D*42\D*$`
151+
// accepts "42", "42.", "= **42**" but rejects any answer containing a different digit.
152+
const looksRight = /^\D*42\D*$/.test((finalText || answer).trim());
148153
if (t2.stop_reason === 'tool_use') {
149154
return { verdict: 'PASS*', code: 0, why: 'Turn 2 succeeded but called a tool again instead of answering — the loop is NOT rejected by the missing reasoning (the key §3 risk is cleared). Multi-step chains work; just noting it did not stop here.' };
150155
}
151156
return looksRight
152157
? { verdict: 'PASS', code: 0, why: 'Turn 2 completed with the correct answer, with NO reasoning echoed back in the assistant turn. K3 does not require reasoning round-tripping → tools:true is safe.' }
153-
: { verdict: 'PASS?', code: 0, why: 'Turn 2 completed without error (the §3 rejection did NOT happen), but the answer did not clearly contain "42". Loop mechanics are fine; eyeball the answer above.' };
158+
: { verdict: 'PASS?', code: 0, why: 'Turn 2 completed without error (the §3 rejection did NOT happen), but the answer was not exactly "42" — likely right but verbose. Loop mechanics are fine; eyeball the answer above.' };
154159
}
155160

156161
/** Distinguish the §3 failure (400 needing reasoning) from capacity (429) and other errors. */

0 commit comments

Comments
 (0)