Skip to content

Commit e1be55e

Browse files
idoubiclaude
andcommitted
fix: warn before the tool-call budget runs out, and disclose when it did
The 20-iteration cap is what actually truncated the provisioning session analysed in 8318f7f, and it explains the ending better than the earlier read did: the model was mid-thought about running a real end-to-end test when the budget ran out, and the forced synthesis turned that into a confident sign-off with a tick beside a step that never executed. Three things were wrong, none of them the cap value itself. No warning before the cliff. maybeInjectSoftDeadlineWarning has told the model about the wall-clock budget since it shipped; the tool-call budget - the one that actually fires - told it nothing. A turn that plans five steps and is guillotined after four has no way to prioritise, and the step it loses is always the last one, which is verification. Adds the iteration analogue, firing with 30% of the budget left (6 calls of 20): enough to finish and verify, not enough to start exploring again. The forced-synthesis nudge optimised for the wrong thing. "Do not apologize without delivering content" plus "the most complete deliverable you can" reads as an instruction to sound finished. It now also requires saying the budget ran out, naming the steps that did not run, and reporting only what tool results actually showed - being cut off before a check is not the check passing. The truncation notice existed only as metadata. iterationCapReached has exactly one consumer, web/src/components/chat-screen.tsx. WeChat, Discord, Feishu, LINE and API callers drop it silently, so a truncated turn arrives on those channels looking like a finished answer with no indication otherwise. Disclosure that lives in UI chrome does not exist off that UI; it is now also in the message text for channels that cannot render the badge, on both the streaming and non-streaming paths (emitted as a trailing chunk too, since a streaming client renders what it received, not what was persisted). The default stays at 20. The session overran it partly on work that no longer happens - the anti-detect-browser detour, the filesystem-wide find, a re-read of an already-loaded skill - and provisioning now costs about five native tool calls instead of a dozen shelled-out CLI invocations. maxToolIterations remains per-agent configurable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 8318f7f commit e1be55e

2 files changed

Lines changed: 188 additions & 2 deletions

File tree

internal/agent/loop.go

Lines changed: 93 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2014,6 +2014,52 @@ func maybeInjectSoftDeadlineWarning(ctx context.Context, turnStart time.Time, me
20142014
return append(messages, warnMsg), true
20152015
}
20162016

2017+
// softIterationFraction is the share of the tool-call budget left when
2018+
// the wrap-up warning fires. 0.30 of 20 leaves 6 calls — enough to close
2019+
// out a multi-step task (finish the current step, verify, report) but
2020+
// not enough to start a new line of exploration.
2021+
const softIterationFraction = 0.30
2022+
2023+
// maybeInjectIterationBudgetWarning is the tool-call analogue of
2024+
// maybeInjectSoftDeadlineWarning.
2025+
//
2026+
// The wall-clock budget has warned the model since it shipped; the
2027+
// iteration budget never did, and it is the one that actually fires. A
2028+
// turn that plans five steps, spends its calls on the first four, and
2029+
// gets guillotined mid-plan produces the worst possible artifact: the
2030+
// forced synthesis reads as a confident completion report, because from
2031+
// the model's point of view it never learned it was about to be cut off.
2032+
// The verification step is what gets dropped — it is always last — so
2033+
// the turn ends by claiming success it never checked.
2034+
//
2035+
// Warning early enough to act is the whole point: this fires with
2036+
// softIterationFraction of the budget left, so the model can spend its
2037+
// remaining calls finishing rather than exploring.
2038+
func maybeInjectIterationBudgetWarning(ctx context.Context, used, max int, messages []provider.Message, alreadyFired bool) ([]provider.Message, bool) {
2039+
if alreadyFired || max <= 0 {
2040+
return messages, alreadyFired
2041+
}
2042+
remaining := max - used
2043+
if remaining <= 0 || float64(remaining) > float64(max)*softIterationFraction {
2044+
return messages, false
2045+
}
2046+
emitEvent(ctx, ChatEvent{Type: "status", Data: map[string]any{
2047+
"phase": "wrap_up",
2048+
"remaining_tool_calls": remaining,
2049+
}})
2050+
warnMsg := provider.Message{
2051+
Role: "system",
2052+
Content: fmt.Sprintf(
2053+
"Tool budget warning: %d of %d tool calls used — about %d remain before this turn is cut off and "+
2054+
"a final answer is forced from whatever you have. Stop exploring and start closing out. "+
2055+
"Spend the remaining calls on finishing the task and on any verification you told the user you would do; "+
2056+
"verification is the step that gets lost when the budget runs out, and an unverified claim reported as "+
2057+
"done is worse than an honest 'not yet checked'. If you cannot finish, say plainly which steps did not run.",
2058+
used, max, remaining),
2059+
}
2060+
return append(messages, warnMsg), true
2061+
}
2062+
20172063
// runToolsWithProgress wraps executeToolsConcurrently with a ticker that
20182064
// emits "tool_progress" events every toolProgressInterval while the
20192065
// call is in flight, so a slow tool (sandbox exec, subagent delegate)
@@ -2269,6 +2315,7 @@ func (a *Agent) HandleMessage(ctx context.Context, msg bus.InboundMessage) strin
22692315
// turn run until it's hard-killed mid-request.
22702316
turnStart := time.Now()
22712317
softDeadlineFired := false
2318+
iterBudgetWarned := false
22722319

22732320
// replyParts accumulates every non-empty assistant text segment
22742321
// emitted across iterations (preamble lines before tool calls + the
@@ -2289,6 +2336,7 @@ func (a *Agent) HandleMessage(ctx context.Context, msg bus.InboundMessage) strin
22892336
)
22902337

22912338
messages, softDeadlineFired = maybeInjectSoftDeadlineWarning(ctx, turnStart, messages, softDeadlineFired)
2339+
messages, iterBudgetWarned = maybeInjectIterationBudgetWarning(ctx, i, a.maxToolIterations, messages, iterBudgetWarned)
22922340

22932341
// Hook: BeforeModelCall
22942342
hcBefore := &HookContext{AgentName: a.name, Point: BeforeModelCall, Messages: messages, Channel: msg.Channel, AccountID: msg.AccountID, ChatID: msg.ChatID, UserID: a.ownerUserID}
@@ -2655,6 +2703,7 @@ func (a *Agent) HandleMessage(ctx context.Context, msg bus.InboundMessage) strin
26552703
// badge attached.
26562704
finalContent = fmt.Sprintf("I've reached the maximum number of tool iterations (%d) and couldn't synthesize a final response. The work above represents what I gathered before hitting the limit.", a.maxToolIterations)
26572705
}
2706+
finalContent += iterationCapNotice(msg.Channel, a.maxToolIterations)
26582707
capMeta := mergeMetadata(iterationCapMetadata(a.maxToolIterations), knowledgeMeta)
26592708
sess.Append(provider.Message{
26602709
Role: "assistant",
@@ -3024,10 +3073,12 @@ func (a *Agent) HandleMessageStream(ctx context.Context, msg bus.InboundMessage)
30243073
streakState := sameToolFailStreakState{}
30253074
turnStart := time.Now()
30263075
softDeadlineFired := false
3076+
iterBudgetWarned := false
30273077

30283078
// ReAct loop - use Chat for tool iterations
30293079
for i := 0; i < a.maxToolIterations; i++ {
30303080
messages, softDeadlineFired = maybeInjectSoftDeadlineWarning(ctx, turnStart, messages, softDeadlineFired)
3081+
messages, iterBudgetWarned = maybeInjectIterationBudgetWarning(ctx, i, a.maxToolIterations, messages, iterBudgetWarned)
30313082

30323083
hcBefore := &HookContext{AgentName: a.name, Point: BeforeModelCall, Messages: messages, Channel: msg.Channel, AccountID: msg.AccountID, ChatID: msg.ChatID, UserID: a.ownerUserID}
30333084
a.hooks.Run(ctx, hcBefore)
@@ -3239,7 +3290,8 @@ func (a *Agent) streamFinalDeliveryAfterCap(ctx context.Context, inboundMsg bus.
32393290
if err != nil {
32403291
// Streaming endpoint failed — persist+emit a fallback line
32413292
// with the badge so the user still gets the signal.
3242-
fallback := fmt.Sprintf("I've reached the maximum number of tool iterations (%d) and couldn't synthesize a final response. The work above represents what I gathered before hitting the limit.", a.maxToolIterations)
3293+
fallback := fmt.Sprintf("I've reached the maximum number of tool iterations (%d) and couldn't synthesize a final response. The work above represents what I gathered before hitting the limit.", a.maxToolIterations) +
3294+
iterationCapNotice(inboundMsg.Channel, a.maxToolIterations)
32433295
fallbackMsg := provider.Message{Role: "assistant", Content: fallback, Metadata: capMeta, Timestamp: time.Now().UnixMilli()}
32443296
sess.Append(fallbackMsg)
32453297
emitEvent(ctx, ChatEvent{Type: "content", Data: map[string]any{"content": fallback, "metadata": capMeta}})
@@ -3287,6 +3339,18 @@ func (a *Agent) streamFinalDeliveryAfterCap(ctx context.Context, inboundMsg bus.
32873339
if content == "" {
32883340
content = fmt.Sprintf("I've reached the maximum number of tool iterations (%d) and couldn't synthesize a final response. The work above represents what I gathered before hitting the limit.", a.maxToolIterations)
32893341
}
3342+
// Emit the truncation notice as a trailing chunk too, not just in
3343+
// the persisted message: a streaming consumer renders what it
3344+
// received, so a notice added only to the stored copy would never
3345+
// reach the person actually reading the reply.
3346+
if notice := iterationCapNotice(inboundMsg.Channel, a.maxToolIterations); notice != "" {
3347+
content += notice
3348+
select {
3349+
case outCh <- provider.StreamChunk{Content: notice}:
3350+
case <-ctx.Done():
3351+
return
3352+
}
3353+
}
32903354
finalMsg := provider.Message{
32913355
Role: "assistant",
32923356
Content: content,
@@ -3345,12 +3409,39 @@ func capReachedNudge(maxIterations int) provider.Message {
33453409
return provider.Message{
33463410
Role: "system",
33473411
Content: fmt.Sprintf(
3348-
"You've used all %d tool-call iterations available for this turn. Tools are now disabled for this final response — do not attempt to call any. Synthesize what you've already gathered into the most complete deliverable you can: if the user asked for a structured artifact (table, list, ICP summary, email drafts, etc.), produce it now from the existing tool results. For any fields you couldn't resolve, mark them as 'unknown' / 'not found' / 'partial' rather than dropping rows or skipping the structure — give the user something usable plus an honest note about what's missing. Do not apologize without delivering content.",
3412+
"You've used all %d tool-call iterations available for this turn. Tools are now disabled for this final response — do not attempt to call any. "+
3413+
"Synthesize what you've already gathered into the most complete deliverable you can: if the user asked for a structured artifact (table, list, ICP summary, email drafts, etc.), produce it now from the existing tool results. "+
3414+
"For any fields you couldn't resolve, mark them as 'unknown' / 'not found' / 'partial' rather than dropping rows or skipping the structure — give the user something usable plus an honest note about what's missing. Do not apologize without delivering content.\n\n"+
3415+
"Two things you MUST get right, because this turn ended early and the user cannot see that unless you say it:\n"+
3416+
"- State up front that you ran out of tool budget and the work is incomplete. Name the steps you planned but did not run.\n"+
3417+
"- Report ONLY what tool results actually showed. Do not mark a step done, tick it off, or call something verified unless a tool result in this conversation confirms it — being cut off before a check is not the same as the check passing. "+
3418+
"If your last plan step was verification and it never ran, the honest report is 'configured but not verified', never a completion claim.",
33493419
maxIterations,
33503420
),
33513421
}
33523422
}
33533423

3424+
// iterationCapNotice is the in-band truncation marker for channels that
3425+
// cannot render the iterationCapReached badge.
3426+
//
3427+
// That badge is read by exactly one consumer — the web chat UI. Every
3428+
// other surface (WeChat, Discord, Feishu, LINE, plain API consumers)
3429+
// drops the metadata silently, so a turn that was guillotined mid-plan
3430+
// arrives looking like a finished, confident answer. The disclosure that
3431+
// the work is partial cannot live only in UI chrome; on those channels
3432+
// it has to be in the text or it does not exist.
3433+
//
3434+
// Returns "" for channels that do render the badge, so the web UI
3435+
// doesn't show the same warning twice.
3436+
func iterationCapNotice(channel string, maxIterations int) string {
3437+
if channel == "web" {
3438+
return ""
3439+
}
3440+
return fmt.Sprintf(
3441+
"\n\n---\n⚠️ This turn hit its %d tool-call limit and was cut short — the answer above is built from partial results, and any step described as done was not necessarily verified. Reply to continue.",
3442+
maxIterations)
3443+
}
3444+
33543445
// iterationCapMetadata is the assistant-side metadata stamped on the
33553446
// forced final-delivery message so the UI can badge the bubble. Kept
33563447
// as a constructor so the key name stays canonical across the streaming

internal/agent/loop_stall_test.go

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,3 +230,98 @@ func TestUpdateSameToolFailStreak(t *testing.T) {
230230
})
231231
}
232232

233+
234+
// The tool-call budget is the limit that actually fires in practice, yet
235+
// it warned nobody until now — only the wall-clock budget did. A turn
236+
// that plans five steps and gets cut off after four ends by reporting
237+
// the fifth as done, because the model never learned it was running out.
238+
func TestMaybeInjectIterationBudgetWarning_FiresOnceNearTheCap(t *testing.T) {
239+
ctx := context.Background()
240+
base := []provider.Message{{Role: "user", Content: "go"}}
241+
242+
// 13 of 20 used: 7 left, above the 30% threshold — stay quiet.
243+
got, fired := maybeInjectIterationBudgetWarning(ctx, 13, 20, base, false)
244+
if fired || len(got) != len(base) {
245+
t.Fatalf("must not fire while budget is comfortable (fired=%v, msgs=%d)", fired, len(got))
246+
}
247+
248+
// 14 of 20: 6 left, exactly at 30% — warn.
249+
got, fired = maybeInjectIterationBudgetWarning(ctx, 14, 20, base, false)
250+
if !fired {
251+
t.Fatal("expected the warning to fire with 30% of the budget left")
252+
}
253+
if len(got) != len(base)+1 {
254+
t.Fatalf("expected one appended system message, got %d", len(got))
255+
}
256+
warn := got[len(got)-1]
257+
if warn.Role != "system" {
258+
t.Errorf("warning should be a system message, got role %q", warn.Role)
259+
}
260+
for _, want := range []string{"Tool budget warning", "verification"} {
261+
if !strings.Contains(warn.Content, want) {
262+
t.Errorf("warning missing %q: %s", want, warn.Content)
263+
}
264+
}
265+
266+
// Already fired — never append twice in one turn.
267+
got2, fired2 := maybeInjectIterationBudgetWarning(ctx, 18, 20, got, true)
268+
if !fired2 || len(got2) != len(got) {
269+
t.Errorf("warning must fire at most once per turn")
270+
}
271+
}
272+
273+
// A budget already spent has nothing to warn about — the cap nudge takes
274+
// over at that point.
275+
func TestMaybeInjectIterationBudgetWarning_NoBudgetNoFire(t *testing.T) {
276+
ctx := context.Background()
277+
base := []provider.Message{{Role: "user", Content: "go"}}
278+
if _, fired := maybeInjectIterationBudgetWarning(ctx, 20, 20, base, false); fired {
279+
t.Error("must not fire when the budget is already exhausted")
280+
}
281+
if _, fired := maybeInjectIterationBudgetWarning(ctx, 0, 0, base, false); fired {
282+
t.Error("must not fire when no cap is configured")
283+
}
284+
}
285+
286+
// The cap-reached banner is rendered by exactly one consumer — the web
287+
// chat UI. On every other channel the metadata is dropped, so without an
288+
// in-band notice a guillotined turn arrives looking like a finished,
289+
// confident answer.
290+
func TestIterationCapNoticeReachesNonWebChannels(t *testing.T) {
291+
if notice := iterationCapNotice("web", 20); notice != "" {
292+
t.Errorf("web renders its own badge; text notice would duplicate it: %q", notice)
293+
}
294+
for _, ch := range []string{"wechat", "discord", "feishu", "line", "api", ""} {
295+
notice := iterationCapNotice(ch, 20)
296+
if notice == "" {
297+
t.Errorf("channel %q silently drops the cap metadata and needs an in-band notice", ch)
298+
continue
299+
}
300+
if !strings.Contains(notice, "20") {
301+
t.Errorf("notice for %q should name the limit: %q", ch, notice)
302+
}
303+
if !strings.Contains(notice, "not necessarily verified") {
304+
t.Errorf("notice for %q should undercut completion claims: %q", ch, notice)
305+
}
306+
}
307+
}
308+
309+
// The forced-synthesis nudge used to push purely toward "deliver
310+
// content", which is how a truncated turn produced a confident
311+
// completion report with a tick next to a step that never ran.
312+
func TestCapReachedNudgeDemandsHonestyAboutTruncation(t *testing.T) {
313+
msg := capReachedNudge(20)
314+
if msg.Role != "system" {
315+
t.Fatalf("nudge role = %q", msg.Role)
316+
}
317+
for _, want := range []string{
318+
"ran out of tool budget",
319+
"did not run",
320+
"unless a tool result",
321+
"configured but not verified",
322+
} {
323+
if !strings.Contains(msg.Content, want) {
324+
t.Errorf("nudge should require honesty about %q:\n%s", want, msg.Content)
325+
}
326+
}
327+
}

0 commit comments

Comments
 (0)