Skip to content

fix(activity): apply activity_max_response_size on the write path - #1174

Merged
Dumbris merged 2 commits into
smart-mcp-proxy:mainfrom
antongulin:fix/activity-response-truncation
Sep 4, 2026
Merged

Dumbris merged 2 commits into
smart-mcp-proxy:mainfrom
antongulin:fix/activity-response-truncation

Conversation

@antongulin

@antongulin antongulin commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request

Related #1173

Description

activity_max_response_size is declared in config with a 64KB default and never read. truncateResponse and its exported wrapper TruncateActivityResponse have no production caller either — the only caller in the tree is internal/storage/activity_test.go:317.

So nothing bounds a single activity record. Retention prunes the log by age, count and total size, but the full response text is persisted for every record type, and one proxied upstream response can be many megabytes.

On the deployment where I found this, activity_records held 82 records over 200KB, the largest 10.75MB — about 160x the configured cap — with nothing logged to suggest anything was wrong. config.db had reached ~940MB.

What this changes

It wires up the truncation that already exists, rather than adding a new mechanism:

  • maxResponseSize on ActivityService, defaulting to the documented 64KB
  • SetMaxResponseSize, taking the same "ignore non-positive" shape as the other retention setters, so an unset config value cannot silently disable the cap
  • applied in the three places that build a record: handleToolCallCompleted, handleInternalToolCall, handlePromptGet
  • fed cfg.ActivityMaxResponseSize from runtime.New, next to the existing retention wiring, and added to the "Activity retention config applied" log line

Two things worth a reviewer's attention

1. ResponseTruncated is OR-ed, not assigned. That field already carries the Spec 103 meaning "the recorded response is larger than the one the agent received" (activity_service.go, the comment above responseTruncated := getBoolPayload(...)). Storage truncation is the opposite direction. I OR-ed the two so neither can clear the other, on the basis that the field documents itself as "true if response was truncated" and a consumer wanting the untruncated text is out of luck in both cases. If you would rather have a separate field for the storage-side meaning, say so and I will split it.

2. handlePromptGet did not set ResponseTruncated at all and now does.

There is also a small alias, DefaultActivityMaxResponseSize = storage.DefaultMaxResponseSize, because NewActivityService's first parameter is named storage and shadows the package inside that function body.

Testing

  • I have tested these changes locally
  • I have added/updated tests that prove my fix is effective or my feature works
  • All existing tests pass

New file internal/runtime/activity_response_truncation_test.go, five cases. They drive the real event path through handleEvent and assert on what actually reached storage, rather than on a mock:

  • an upstream tool-call response is capped and flagged
  • an internal_tool_call response takes the same cap (this is call_tool_read, where the large records came from)
  • an emitter-set response_truncated survives a record storage does not shorten
  • the 64KB default applies with no config set
  • SetMaxResponseSize ignores non-positive values

Negative control: with the body of truncateForStorage replaced by return response, false, three of the five fail; restoring it makes them pass. So they fail on pre-fix code, not just on a mock.

--- FAIL: TestToolCallResponseTruncatedForStorage
--- FAIL: TestInternalToolCallResponseTruncatedForStorage
--- FAIL: TestDefaultResponseCapAppliesWithoutConfig

Ran green:

ok  internal/runtime               122.985s
ok  internal/runtime/configsvc       0.139s
ok  internal/runtime/stateview       0.151s
ok  internal/runtime/supervisor      1.875s
ok  internal/runtime/supervisor/actor 0.839s
ok  internal/storage                24.752s
ok  internal/config                  6.242s
ok  internal/httpapi                 1.025s

Plus -race on the new tests, go build ./..., go vet ./internal/runtime/..., gofmt, and staticcheck ./internal/runtime/ — all clean. internal/server was still running locally when I opened this; I will follow up in a comment with the result, and CI covers it regardless.

A note for anyone writing similar tests

storage.DefaultActivityFilter() sets ExcludeCallToolSuccess: true, so a successful call_tool_* record is invisible to it. My first draft used that filter, saw zero rows and looked like a storage bug. The tests pass an explicit storage.ActivityFilter{Limit: 100} instead.

🤖 Generated with Claude Code

activity_max_response_size was declared in config with a 64KB default and
never read. truncateResponse and its exported wrapper TruncateActivityResponse
had no production caller either, so nothing bounded a single activity record:
the full response text was persisted for every record type.

Retention prunes the log by age, count and total size, but a single proxied
upstream response could still be many megabytes. On one real deployment this
produced 82 records over 200KB in activity_records, the largest 10.75MB, or
160x the configured cap, with nothing logged to suggest a problem.

Wire the existing truncation into ActivityService:

- carry maxResponseSize on the service, defaulting to the documented 64KB
- SetMaxResponseSize, matching the "ignore non-positive" shape of the other
  retention setters so an unset config cannot silently disable the cap
- apply it in handleToolCallCompleted, handleInternalToolCall and
  handlePromptGet, which are the three sites that build a record
- feed it cfg.ActivityMaxResponseSize from runtime.New

The truncation result is OR-ed into ResponseTruncated rather than assigned.
That field already carries the Spec 103 meaning "the recorded response is
larger than the one the agent received", which is the opposite direction, so
neither meaning may clear the other. handlePromptGet did not set the field at
all and now does.

Tests drive the real event path and assert on what reaches storage. Verified
they fail with the truncation call neutered and pass with it in place.

Fixes smart-mcp-proxy#1173

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@antongulin

Copy link
Copy Markdown
Contributor Author

Following up on the internal/server run I said was still going when I opened this.

It passes. Zero failures, zero skips:

ok  github.com/smart-mcp-proxy/mcpproxy-go/internal/server         593.273s
ok  github.com/smart-mcp-proxy/mcpproxy-go/internal/server/tokens    1.156s
exit=0

Worth recording that my first attempt at this package was invalid, in case anyone reads the earlier state. I ran it without building the binary first, and 12 E2E tests failed instantly:

--- FAIL: TestBinaryStartupAndShutdown (0.00s)
    binary.go:154: mcpproxy binary not found at .../internal/server/mcpproxy.
    Set MCPPROXY_BINARY_PATH to the built binary or run: go build -o mcpproxy ./cmd/mcpproxy

That was me skipping the build step CONTRIBUTING calls for, not a regression from this change. After go build -o mcpproxy ./cmd/mcpproxy and re-running with MCPPROXY_BINARY_PATH set, the result above is what I get.

Full local verification for this branch:

ok  internal/runtime                  122.985s
ok  internal/runtime/configsvc          0.139s
ok  internal/runtime/stateview          0.151s
ok  internal/runtime/supervisor         1.875s
ok  internal/runtime/supervisor/actor   0.839s
ok  internal/storage                   24.752s
ok  internal/config                     6.242s
ok  internal/httpapi                    1.025s
ok  internal/server                   593.273s
ok  internal/server/tokens              1.156s

Plus -race on the new tests, go build ./..., go vet ./internal/runtime/..., gofmt, and staticcheck ./internal/runtime/ — all clean.

So the third checkbox in the description is now backed by an actual full run rather than a partial one.

Dumbris added a commit that referenced this pull request Sep 2, 2026
… token-bench truth-sync (#1200)

* chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench

- mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped
  2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532).
- action-log-transparency: new task activity-storage-bounds tracking
  #1173/#1174 (response truncation never wired), #1176 (unbounded
  per-server tool_calls buckets) and #1175 (omitempty zero erasure).
- token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160
  are all merged. Clears both --check-github errors.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

@Dumbris Dumbris left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the diagnosis is right, the wiring is the correct shape, and the negative control in the description is exactly what I want to see. Two things need to change before merge, and a few smaller ones.

Blocking

1. Sensitive-data detection now scans the truncated text.
In handleToolCallCompleted the new line reassigns response before the Spec 026 block reads it (scanText := response at ~L706, when no detection_text is supplied — the default). handlePromptGet does the same with responseStr (~L1062). Before this PR the detector saw the full response and applied its own cap (sensitive_data_detection.max_payload_size_kb, default 1024KB); after it, anything past byte 65536 is never scanned, and nothing records that the scan window shrank. Please capture the untruncated string before truncating and pass that to runAsyncDetection in both handlers, and add a test with a secret placed after the 64KB boundary under the default cap.

2. OR-ing storage truncation into ResponseTruncated breaks the cost predicate for internal_tool_call records.
Two consumers read that flag on internal records with the specific meaning "the agent received less than ResponseBytes, so exclude the row": internal/runtime/usage_aggregate.go:491 (truncatedBuiltinOverstatesDelivery, feeding the Usage timeline, rebuilt from persisted records on cold start) and bench/replaycorpus/flags.go:373 (ReasonTruncatedRetrieveOverstates). A storage-cut, delivered-whole built-in now carries the flag with an honest ResponseBytes, so it is dropped from delivered traffic — reachable in default config for describe_tool (no agent-side cut) and for retrieve_tools/code_execution whenever activity_max_response_size < tool_response_limit. Reproduced: an 80,000-byte describe_tool record gives RespBytesSum=0 on this branch and 80000 on main.

For tool_call and prompt_get the OR is fine — the comment above that predicate already defines the flag on upstream records as "cut on the way into the log, bytes honest". The narrow fix is to not OR for ActivityTypeInternalToolCall (the stored text still carries the ...[truncated] suffix), plus a test mirroring TestUsageAggregate_TruncatedBuiltinDoesNotInflateDeliveredBytes in the other direction. A separate storage_truncated field is the cleaner answer but drags in contracts, swagger, CSV export and three UI badges; I would leave that for a follow-up.

Non-blocking

  • Please change Fixes #1173 to Related #1173 — this repo never auto-closes issues from PRs (see the commit conventions in .specify/memory/constitution.md), and in this case the reporter's table shows the other 432MB is server_*_tool_calls, which this PR does not touch (tracked as #1176). Arguments is also still stored whole on every record type; that can be a follow-on issue too.
  • handlePromptGet truncation and its new flag have no test; a sixth case mirroring TestToolCallResponseTruncatedForStorage with EventTypeActivityPromptGet would close that (the whole package still passes with only the prompt path neutered).
  • docs/features/activity-log.md row for activity_max_response_size could now say what it enforces: applies to tool_call/internal_tool_call/prompt_get, 0/absent = 65536, cannot be disabled, applied at startup only (same as its retention siblings), already-stored records are not rewritten and the BBolt file does not shrink.

Everything else checked out: UTF-8 splitting at the cut is tolerated by every consumer (encoding/json coerces), the suffix can push a record a few bytes over the cap (fine), -race is clean, and I confirmed the negative control independently.

…g honest

Two defects in the review of smart-mcp-proxy#1174, both in the wiring rather than the
truncation itself.

1. Storage truncation ran before the Spec 026 sensitive-data scan.

handleToolCallCompleted reassigned response, and handlePromptGet reassigned
responseStr, before the detector read them. The detector applies its own cap
(sensitive_data_detection.max_payload_size_kb, 1024KB by default), so the
effect was to shrink the scan window to activity_max_response_size — 64KB in
the default configuration, a 16x reduction — and nothing recorded that it had
moved. A secret past the storage cut simply stopped being detected.

Both handlers now capture the response as received and pass that to
runAsyncDetection, while still storing the cut copy. detection_text continues
to win where the TOON seam supplies it (spec 084 FR-007b).

2. OR-ing storage truncation into ResponseTruncated broke delivered bytes.

On an internal_tool_call that flag means "the agent received LESS than
ResponseBytes", and two consumers read it as "exclude this row from delivered
traffic": truncatedBuiltinOverstatesDelivery, which feeds the usage timeline
and is rebuilt from persisted records on cold start, and
bench/replaycorpus/flags.go. A built-in cut only on the way into the log was
delivered whole with an honest ResponseBytes, so the flag erased it — reachable
in default config for describe_tool, and for retrieve_tools/code_execution
whenever activity_max_response_size < tool_response_limit.

handleInternalToolCall now truncates the stored text without touching the flag.
The "...[truncated]" suffix still marks the text. tool_call and prompt_get keep
the OR, where the meaning already matches. A dedicated storage_truncated field
would be cleaner but reaches contracts, swagger, CSV export and three UI badges;
left for a follow-up.

Tests:

- TestDetectionScansUntruncatedResponse places an AWS key past the 64KB
  boundary under the default cap and asserts it is still detected, on both the
  tool and prompt paths, while absent from the stored response
- TestStorageTruncatedBuiltinStillCountsDeliveredBytes drives an 80,000-byte
  describe_tool through the event path and asserts 80,000 delivered bytes,
  mirroring TestUsageAggregate_TruncatedBuiltinDoesNotInflateDeliveredBytes
  from the other side
- TestInternalToolCallResponseTruncatedForStorage now asserts the flag stays
  false, which is the corrected behaviour
- TestPromptGetResponseTruncatedForStorage covers the prompt path, which had
  no test

Both fixes negative-controlled: restoring either defect fails the new tests and
nothing else. docs/features/activity-log.md now states what the setting
enforces, which record types it covers, and that detection is unaffected.

Related smart-mcp-proxy#1173

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@antongulin

Copy link
Copy Markdown
Contributor Author

Both blockers fixed in 6901487.

1. Scan before truncate. handleToolCallCompleted and handlePromptGet now capture the response as received and pass that to runAsyncDetection, while still storing the cut copy. detection_text still wins where the TOON seam supplies it. TestDetectionScansUntruncatedResponse puts an AWS key past the 64KB boundary under the default cap and asserts it is still detected on both paths — and absent from the stored response.

2. No OR for internal_tool_call. handleInternalToolCall truncates the stored text without touching ResponseTruncated; tool_call and prompt_get keep the OR, where the meaning already matches. TestStorageTruncatedBuiltinStillCountsDeliveredBytes drives an 80,000-byte describe_tool through the event path and asserts 80,000 delivered bytes; TestInternalToolCallResponseTruncatedForStorage now asserts the flag stays false. Left storage_truncated as a follow-up, as you suggested.

Also done: TestPromptGetResponseTruncatedForStorage for the untested prompt path, PR body changed to Related #1173, and the activity_max_response_size doc row now states what it enforces, which record types it covers, startup-only, no rewrite of stored records, and that detection is unaffected.

Both fixes negative-controlled — restoring either defect fails the new tests and nothing else. ./internal/runtime/... and ./internal/storage/... pass.

One thing I left alone: the first commit's message still reads Fixes #1173. Say the word and I will amend and force-push.

@Dumbris
Dumbris merged commit 7a27851 into smart-mcp-proxy:main Sep 4, 2026
36 of 38 checks passed
@Dumbris

Dumbris commented Sep 4, 2026

Copy link
Copy Markdown
Member

Thanks Anton — merged.

Good catch. A config field declared and never read, plus an exported helper with no production caller, is exactly the kind of thing that sits there for a year. The 10.75MB record and the 940MB config.db made the case better than any argument would have.

Both review fixes are right, and I checked the negative controls rather than taking them on trust — reverting either one does fail the new tests.

One thing: TestDetectionScansUntruncatedResponse times out under -race. Its ~75KB payload takes ~9.5s to scan (the race detector costs ~24x on that path) against waitForDetectionMetadata's 5s deadline — which is why it went red in CI and green for you locally. Nothing wrong with the logic; just the payload sizing. Fixed in #1207 with an explicit 1KB cap instead of the 64KB default: same seam, 0.5s.

Thanks also for the DefaultActivityFilter note in the description — that one would have cost the next person an hour.

github-actions Bot pushed a commit that referenced this pull request Sep 4, 2026
…tector (#1207)

The test added in #1174 failed in both Unit Tests and End-to-End Tests, and the
defect is its payload sizing rather than anything it guards.

To place a secret past the storage cut it padded the response to
storage.DefaultMaxResponseSize+10_000 (~75KB), relying on the 64KB default cap.
security.Detector.Scan costs ~5.5us/byte, and the race detector multiplies that
by ~24x (measured: 114-137us/byte at 1K-75K). So the scan took ~9.5s against the
5s poll deadline in waitForDetectionMetadata. Because runAsyncDetection only
writes metadata on a hit, an over-budget scan surfaces as "timed out waiting for
detection metadata" rather than a failed assertion, which is why it read as a
logic failure. CI runs -race in both workflows; a plain local run does not.

Set the cap explicitly with SetMaxResponseSize(1024) and pad to 4KB instead. The
seam is identical - the AWS key sits at bytes [4113, 4133), 3089 bytes past the
1024-byte cut - and truncateResponse honours any positive maxSize with no
minimum floor. Runtime drops from ~9.5s to ~0.5s per subtest.

Verified:

- 5/5 runs under -race, 0.45-0.50s per subtest against the 5s deadline
- negative control: moving the detectionSource capture back to after
  truncateForStorage in both handlers still fails the test at the smaller
  payload, so it is not passing vacuously
- second control: stubbing truncateForStorage to a no-op fails the
  NotContains assertion, proving the key is genuinely past the cut
- whole internal/runtime package green under -race, under the unit-tests.yml
  invocation and under the e2e-tests.yml -short invocation
- headroom measured under CPU oversubscription: still passes at 8x slowdown
  (2.96-3.76s), breaks only around 10.8x

TestDefaultResponseCapAppliesWithoutConfig continues to cover the 64KB default
separately, so no coverage is lost.
Dumbris added a commit that referenced this pull request Sep 5, 2026
…ros, add db compact (#1214)

* fix(storage): bound the tool-call history and stop erasing configured zeros

Closes the other half of the 940MB config.db in #1173.

#1176 — the per-server server_<id>_tool_calls buckets held ~432MB of that
940MB and were unbounded in every direction: RecordToolCall did
CreateBucketIfNotExists + Put with no size cap, no count cap, no age prune and
no eviction, and ToolCallRecord.Response is a bare interface{} holding the
whole upstream CallToolResult. Nothing ever deleted a bucket either —
CleanupStaleServerData exists but has no caller — so a server removed from the
config left its entire call history behind forever.

Bound it at the single write seam, so every producer is covered rather than the
ones a caller remembered to cap:

- a per-record cap on the marshalled response AND arguments. Over the cap the
  field becomes {truncated, original_bytes, preview, note} and the record
  carries response_truncated + response_bytes. It stays a JSON OBJECT on
  purpose: contracts.ToolCallRecord.Response is a documented payload on four
  OAS endpoints and httpapi.maskToolCallRecord walks it as structured JSON, so
  retyping it to a string would be a breaking API change.
- eviction of the oldest records past a per-server count, in the same
  transaction as the write, so the bucket can never exceed the cap even
  briefly. The walk goes backwards from the newest key and stops at the cap, so
  its cost is bounded by the cap rather than the bucket size; Stats().KeyN is
  deliberately unused because inside the write transaction it does not count
  the Put that just happened.
- bucket deletion on server removal, and an orphan sweep beside the existing
  PruneOrphanToolApprovals.

The sweep takes server NAMES, as its sibling does, but these buckets are keyed
by server IDENTITY — the SHA-256 GenerateServerID computes from a server's
attributes — so it resolves one to the other through server_identities. A live
run proved this is not theoretical: the first version compared bucket keys
against names, which marks every live history an orphan, and it deleted a real
bucket on restart. The unit tests missed it because they passed the name AS the
ServerID; they now register real identities and assert the id is not the name.
An empty identity store aborts the sweep — that state means "not populated
yet", not "every server is gone".

#1175 — three config fields carried omitempty on a numeric type while 0 was a
documented meaningful value, so SaveConfig's json.MarshalIndent deleted an
operator's explicit 0 and the field reverted to its default on the next write:
max_result_size_chars ("Set to 0 to disable"), activity_max_size_mb
("0=disabled") and observability.tracing.sample_rate. All three become
tri-state pointers with Effective…() readers. Dropping omitempty from the int
is not equivalent — that materialises an absent key as an explicit 0 and turns
the feature off for everyone who never set it.

The defaults live in the readers, not only in DefaultConfig(), because two REST
write paths decode a client document into a ZERO-valued config.Config
(handlePatchConfig's `var merged config.Config`, UnmaskLiveConfigDocument's
`&config.Config{}`) and never see the constructor.

One trap the conversion had to clear: runtime.New gated SetMaxResponseSize
behind `if … || cfg.ActivityMaxSizeMB >= 0`, a clause that is trivially true
for a plain int and was the only reason the 64KB activity cap from #1174 ran at
all. A naive pointer rewrite would have silently disabled it for every config
that omits the key, and the existing default-path test would not have caught it
because it constructs ActivityService directly. The per-record caps are now
applied unconditionally, outside the retention guard where they never belonged.

A reflection guard test walks the whole Config tree and fails on any numeric
field with omitempty that is not on an allowlist with a stated reason. It found
14 more; each was checked against its doc comment or read site and every one
genuinely treats 0 and absent as the same value.

mcpproxy db compact / db stats — the caps bound future growth, but BBolt never
returns freed pages to the OS, so an already-large config.db stays large.
Offline compaction via bbolt.Compact (already pinned, no new dependency) with
the atomic temp-file+rename recipe from config/loader.go, and db stats to say
whether it is worth running. A locked database fails fast with exit 3 and a
remediation line rather than retrying.

The output is explicit that the default run frees no disk: the retained
config.db.bak holds the original inode, so "reclaimed" describes the file and
the filesystem gets nothing back until the backup is deleted. That matters most
to the user who ran out of disk in the first place.

Verified on a real instance (isolated port/data-dir/config, stdio upstream): a
300KB response stored as a 2.2KB truncated object with response_bytes=300065; a
300KB argument likewise; 8 calls against a cap of 5 leaving exactly 5 records;
history surviving a restart (3 before, 3 after — 5 before / 0 after with the
identity bug); and compaction taking 4.0MB to 2.0MB with every record, the
upstream server and its tools intact afterwards.

Closes #1175, closes #1176
Related #1173

* fix(storage): close the defects two cross-model reviews found

Independent reviews by opencode gpt-5.6-sol and codex gpt-5.6-sol over the
three areas of the previous commit. Every finding is fixed, triaged with a
reason, or pinned by a test that fails without the fix.

REPLAY (found by codex; the worst of the set). Capping stored arguments made
POST /api/v1/tool-calls/{id}/replay unsafe: with no argument override it
re-dispatches the STORED arguments, which for a capped record are the
{truncated, original_bytes, preview} placeholder. A replayed call_tool_write or
call_tool_destructive would have reached a real system with nonsense. Records
now carry arguments_truncated and replay refuses them with a message telling
the caller to supply arguments explicitly.

BACKUP COULD DESTROY THE LIVE DATABASE (codex). copyFile — the fallback when
os.Link fails — did os.WriteFile straight onto the destination. That path can
BE a hard link to config.db (an earlier run's .bak, or a concurrent
compaction's), so the "backup" step could truncate the database it was backing
up. It now writes a temp file and renames, which replaces the directory entry
and leaves any other link to the old inode intact.

THE SWAP WAS UNGUARDED (both). bbolt's lock is released before the rename — the
file has to be closed for a rename to work on Windows — leaving the destructive
half of a minutes-long operation unprotected. A core starting in that window
ends up holding the old, unlinked inode and every write it makes disappears.
Two fixes: a whole-command lock file (O_CREATE|O_EXCL, atomic everywhere, no
build-tagged syscall) so two compactions cannot overlap at all, and a
pre-replace check that the source is unchanged and unheld. The check opens
READ-ONLY — a read-write open makes bbolt commit a freelist flush on a database
whose freelist was never persisted, modifying the very file the check exists to
prove untouched.

ORPHAN SWEEP DELETED LIVE HISTORY (opencode). An identity whose JSON would not
unmarshal was skipped while still counting the identity store as populated, so
a live server's hashed bucket became unresolvable and was swept. The bucket key
IS the identity id, so an unreadable record can be spared without reading it.
Such a bucket can then never be reclaimed, which is the right way round —
losing a live server's history is worse than leaking a dead one's — and it is
now logged rather than silent.

TRUNCATION MARKER IGNORED THE CAP (codex). The placeholder carried a fixed 2KB
preview whatever the configured cap was, so tool_call_max_response_size: 1024
produced a "truncated" record of ~2.3KB. The preview now shrinks to fit,
leaving room for the envelope.

TWO ALLOWLIST ENTRIES WERE FALSE (codex). The guard test claimed 0 and absent
agree for activity_retention_days and activity_max_records. They did not: an
omitted key took DefaultConfig's 90 days / 100000 records, while an explicit 0
was ignored by SetRetentionConfig and fell through to that service's own 7 days
/ 10000 — a tenth of the documented retention, which a save then silently
reverted. EffectiveActivityRetentionDays / EffectiveActivityMaxRecords make the
two agree on the documented default. docker_isolation.timeout also differs, but
its only reader renders it in a doctor payload, so it stays allowlisted with
that as the stated reason rather than an untrue equivalence.

GUARD TEST WAS BLIND UNDER -tags server (opencode). Four server-edition fields
are invisible to an untagged reflection walk; all four are normalised by
ServerEditionConfig.Validate and are now allowlisted with that citation. Run
this test under both tags.

Smaller: chmod moved before the destination fsync so the mode is durable with
the data; a failed directory fsync warns instead of being dropped; a .bak that
cannot be removed after a failed rename is reported rather than swallowed; a
partial fallback copy is cleaned up.

Analysed and left alone, with the reasoning recorded in the code: eviction
relies on bytewise key order matching chronological order, which holds because
UnixNano is a fixed 19 digits from 2001 to 2262 — documented so a future key
format change cannot break it silently.

Process notes. opencode's first run on the db command exited 0 with NO verdict
after being refused reads outside its launch directory; bbolt's source was
staged in-repo and it was re-run. Codex could not write anywhere, so its
mutation testing was static; those neuters were run here instead — including
one that exposed a vacuous test of my own, which locked the database too early
and passed on the injected error rather than on the guard.

Verified again on a real instance after every fix: a 200KB response and a 200KB
argument both stored as bounded placeholders with response_truncated,
response_bytes and arguments_truncated set; both records surviving a restart
with no orphan sweep firing.

Related #1175, #1176, #1173

* docs(release): notice for the config.db bloat fix and its manual compaction step

The release body is generated from commit messages and never reads CHANGELOG.md,
so anything a user must ACT on has to go through .github/RELEASE_NOTICE.md.

Two things qualify here. The #1173 storage fixes bound future growth but cannot
shrink an existing file — BBolt does not return freed pages to the OS — so a
user sitting on a 940MB config.db has to run 'mcpproxy db compact' themselves,
with mcpproxy stopped, and then delete the backup that the default run keeps.
Without that step they upgrade and see no improvement.

And two config keys quietly changed meaning: activity_retention_days: 0 and
activity_max_records: 0 used to fall through to internal defaults far smaller
than the documented ones (7 days / 10000 records instead of 90 / 100000). That
is a fix, but anyone who leaned on 0 to keep the log small needs to say what
they meant.

Related #1173, #1175, #1176

* fix(storage): size the truncation marker by measuring it, and fix lock ownership

Second opencode review round, on the fixes from the first.

THE TRUNCATION MARKER COULD EXCEED ITS OWN CAP. previewBytes budgeted the
preview by RAW length against a fixed 512-byte envelope reserve, but the
preview is JSON embedded in JSON: every quote and backslash is escaped a second
time on the way in, so a quote-heavy payload roughly doubles. Measured: a
1024-byte cap produced a 1243-byte record, a 4096-byte cap produced 4308. The
placeholder now halves its preview until the MARSHALLED object fits, and falls
back to a bare {truncated, original_bytes} — about 45 bytes — for caps too
small to hold the note. Estimating the encoded size from the raw size was the
mistake; it now measures.

The existing test passed through all of this because it used strings.Repeat("q")
as filler, which escapes to nothing. It now uses quotes and backslashes and
covers caps from 64 bytes up, and it reproduces the exact 1243/4308 figures
when the fix is reverted.

RELEASING THE COMPACTION LOCK COULD REMOVE SOMEONE ELSE'S. The release closure
deleted the pathname unconditionally. Sequence: someone clears A's
stale-looking lock by hand, B takes a fresh one, A finishes and deletes the
pathname — which is now B's — and a third compaction starts alongside B. The
lock file now carries a pid+timestamp token and release removes it only while
it still contains that token.

THE PRE-REPLACE CHECK NOW RUNS AS LATE AS POSSIBLE, after the backup rather
than before it, so the window between "the source is unheld" and the rename is
microseconds instead of spanning backup creation. It cannot be closed
completely — the file must be closed for a rename to work on Windows, so no
lock can span the rename — and that residual is now stated in the code rather
than implied. Aborting there also cleans up the backup it had just made.

Also recorded rather than fixed: the staleness check compares size and mtime,
not a content hash, because hashing a multi-gigabyte file to guard a
microsecond window would cost more than the operation it protects. An in-place
write preserving both would slip past it — but making one requires an open
handle, which is what the exclusive-open check refuses.

Verified on a real instance: db stats reporting 1.3MB reclaimable, a second
compaction refused by the lock with the remediation message, a real compaction
taking 4.0MB to 2.0MB, the lock file cleaned up afterwards, and every record
still readable through the API.

Related #1176

* fix(test): stop TestPreReplaceCheckAbortsWhenTheDatabaseIsHeld failing on Windows

The test stat'ed the database BEFORE opening the holder. Opening a bbolt
database can write to it — a freelist flush, which is what happens on Windows —
so by the time the pre-replace check ran, the file had legitimately changed and
the check aborted on its staleness half rather than its lock half. Both halves
abort the swap, which is the property that matters, but this case exists to pin
the LOCK detection specifically, and it asserted the error type to prove it.

Stat after the holder is open, so the file looks unchanged and only the lock
check can fire. The Windows failure also cancelled both macOS jobs through the
matrix, which is why three builds went red for one cause.
Dumbris added a commit that referenced this pull request Sep 9, 2026
Specs 104 (auto routing mode) and 105 (agent-token scope hardening) were
written and cross-reviewed on 2026-09-07 but never committed; five merged
PRs already cite 105 by name as their acceptance contract. This lands both
and records them in roadmap.yaml so the DAG reflects what shipped and what
is left.

Spec 105 — agent-token scope hardening. The Spec 104 review verified the
scoped-token promise against the code one surface at a time and found eight
places where a legitimately narrow token could learn about or act on servers
outside its grant. Five fix sessions ran from that review and merged on
2026-09-08: #1223 target tier, #1224 tail_log, #1225 set_profile, #1226
read_cache provenance, #1227 prompt owner and deleted-pin enumeration. Each
was live-verified against a baseline binary and reviewed to a clean verdict.
The epic records those five as done tasks and carries eight todo tasks, one
per remaining FR group, taken from the "Follow-ups / Spec 105 gaps"
checklists in those PR bodies: FR-005 retrieve_tools scoping, FR-008 direct
publication identity, FR-010 refusal shapes, the FR-003/FR-004 selectable-
profile predicate for unpinned tokens, FR-002 legacy cache invalidation,
FR-007 log-record attribution, FR-009 producer-side identity, and the
FR-011/013/014 regression suite.

Spec 104 — auto routing mode. Measures the three candidate surfaces per
session on the catalog that session will actually see and serves the richest
rung under a token budget. It depends on 105: its FR-016 states the
invariant those corrections make true. P1 is US1-4, P2 is US5-6.

PR evidence sits on the child tasks rather than the epic, because
gen-roadmap.py --check-github reads an all-merged epic pr: list as "this
epic is done" and the scope-hardening epic is not.

Also flips activity-storage-bounds to done: #1174 wired the 64KB write-path
cap and #1214 bounded the per-server tool_calls buckets, stopped omitempty
erasing configured zeros and added db compact, closing #1175 and #1176.

The specs/README index rows are appended for both, though that table has
been stale since 043 and does not list 044-103.

Validation: gen-roadmap.py --check-github reports 0 errors. The one warning
(mcp-2026-upgrade has no pr: link or child tasks) is pre-existing.
@antongulin

Copy link
Copy Markdown
Contributor Author

Thanks for merging! Glad the configured response-size limit is now enforced when storing activity records.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants