fix: keep the last raw-byte truncations on a rune boundary (#1726)

# Pull Request

## Description

Round 3 of the UTF-8-safe truncation sweep started in #1647 and
continued in #1664 / #1665. Three production sites still cut text with a
raw byte slice (`s[:n]`), so a non-ASCII value whose cut offset lands
inside a multi-byte rune is published as invalid UTF-8, and JSON
encoding rewrites the tail to U+FFFD.

### 1. `internal/storage` — the activity-log per-record cap

`truncateResponse` is the only thing bounding a single activity record.
It now goes through `truncateRuneSafe`, the helper the per-server
tool-call records in the *same package*
(`internal/storage/tool_call_bounds.go`) already use for exactly this
reason.

**Reachability.** An upstream tool response arrives as the `response`
payload of an activity event, and `ActivityService.handleEvent` →
`truncateForStorage` → `storage.TruncateActivityResponse` →
`truncateResponse` applies `activity_max_response_size` (default 64 KB)
on the write path (`internal/runtime/activity_service.go:267`). Any
response over the cap whose byte at the cut offset is a UTF-8
continuation byte — ordinary CJK, Cyrillic, or emoji text — lands in
`config.db` malformed, and from there in `GET /api/v1/activities`, `GET
/api/v1/activities/{id}` and the CSV export. With a 3-byte rune, 2 of
every 3 cut offsets are mid-rune.

### 2. `internal/outputvalidation` — the 500-byte violation `Reason`

A `jsonschema` message quotes the offending instance location, so a
non-ASCII property name or value sits astride the cut.

**Reachability.** `Validator.Validate` calls `truncate(err.Error(),
500)` (`validator.go:157`) on any structured tool output that fails its
`outputSchema`; the `Reason` goes back to the agent and into the audit
line. Measured on the pre-fix code with a 200-rune non-ASCII property
name, shifting the cut by one byte at a time: offsets 0 and 2 produce
invalid UTF-8, offset 1 is valid — i.e. 2 of 3.

### 3. `internal/registries` — the 300-byte Pulse description cap

Registry descriptions are upstream-authored free text.

**Reachability.** `parsePulseWithoutGuesser` fills
`ServerEntry.Description` from `short_description` /
`EXPERIMENTAL_ai_generated_description` for every result of a Pulse
registry search, which the REST API and the Web UI catalog then
JSON-encode. Same 2-of-3 pattern in the test.

The 300 literal is now a named constant (`pulseDescriptionMaxBytes`)
since it appeared twice.

ASCII behaviour is unchanged at all three sites, and the retained prefix
never exceeds the cap.

**Out of scope,** deliberately, so this stays reviewable:
- The CLI table cells in `cmd/mcpproxy` that #1664 also left alone —
sent separately.
- `cmd/mcpproxy/security_cmd.go` and `cmd/mcpproxy/main.go` have the
same shape but are under open PRs (#1714, #1720, #1725), so I have not
touched them.
- `internal/configimport.SanitizeServerName` and
`internal/dockernaming.SanitizeServerName` also cut at a byte offset,
but both strip everything outside `[A-Za-z0-9._-]` first, so no
multi-byte rune can reach the cut. Left as they are.

## Testing

- [x] I have tested these changes locally
- [x] I have added/updated tests that prove my fix is effective or my
feature works
- [x] All existing tests pass

New tests, each failing by name at the base SHA (`14cb288`) and passing
after:

| Test | Package |
|---|---|
| `TestTruncateResponseKeepsRuneBoundary`,
`TestTruncateResponseASCIIUnchanged` | `internal/storage` |
| `TestViolationReasonIsValidUTF8`, `TestTruncateASCIIUnchanged` |
`internal/outputvalidation` |
| `TestPulseDescriptionTruncationIsValidUTF8` | `internal/registries` |

Each boundary test sweeps every offset inside a 3-byte rune rather than
asserting one fixed cap, so it cannot pass by luck.

Commands run (exit status in brackets):

```
git diff --check                                                            [0]
gofmt -l internal/storage internal/outputvalidation internal/registries     [0, no output]
go vet ./internal/storage ./internal/outputvalidation ./internal/registries [0]
go test ./internal/storage ./internal/outputvalidation ./internal/registries [0]
go test ./internal/runtime -run 'TestToolCallResponseTruncatedForStorage|TestInternalToolCallResponseTruncatedForStorage|TestStorageTruncatedBuiltinStillCountsDeliveredBytes|TestPromptGetResponseTruncatedForStorage|TestDefaultResponseCapAppliesWithoutConfig|TestEmitterTruncationFlagSurvivesUntruncatedRecord'  [0, 6/6 pass]
```

The `internal/runtime` selection is the end-to-end activity-truncation
suite that drives the storage cut through the real event path, included
to show the write-path behaviour and the `ResponseTruncated` flag
semantics are unchanged.

Not run: `./scripts/run-all-tests.sh` and `-race` (local disk
constraints on this machine). Nothing here is concurrent.

Author: rudycelekli

internal/outputvalidation/reason_utf8_test.go

@@ -0,0 +1,45 @@
+package outputvalidation
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+	"unicode/utf8"
+)
+
+// A schema-violation Reason is capped at 500 bytes, and jsonschema's message
+// quotes the offending instance location — so a non-ASCII property name or
+// value puts multi-byte runes astride the cut. The raw byte slice left the
+// Reason invalid UTF-8, which JSON encoding rewrites to U+FFFD on the way to
+// the agent and the audit line (round 3 of #1647/#1664/#1665).
+//
+// The 1-byte padding shifts every later byte, so the loop covers all three
+// offsets inside a 3-byte rune; two of the three are mid-rune.
+func TestViolationReasonIsValidUTF8(t *testing.T) {
+	v := New(0, 0, nil)
+	for pad := 0; pad < 3; pad++ {
+		name := strings.Repeat("a", pad) + strings.Repeat("語", 200)
+		schema := fmt.Sprintf(`{"type":"object","properties":{%q:{"type":"integer"}}}`, name)
+
+		verdict := v.Validate(fmt.Sprintf("srv:tool%d", pad), schema, map[string]any{name: "not-an-integer"})
+		if !verdict.IsViolation() {
+			t.Fatalf("pad=%d: expected a violation", pad)
+		}
+		if len(verdict.Reason) > 500 {
+			t.Errorf("pad=%d: reason is %d bytes, over the 500-byte cap", pad, len(verdict.Reason))
+		}
+		if !utf8.ValidString(verdict.Reason) {
+			t.Errorf("pad=%d: reason is not valid UTF-8 (cut inside a rune)", pad)
+		}
+	}
+}
+
+// ASCII behaviour must not move.
+func TestTruncateASCIIUnchanged(t *testing.T) {
+	if got, want := truncate(strings.Repeat("x", 10), 20), strings.Repeat("x", 10); got != want {
+		t.Errorf("under the cap: got %q, want %q", got, want)
+	}
+	if got, want := truncate(strings.Repeat("x", 30), 20), strings.Repeat("x", 17)+"..."; got != want {
+		t.Errorf("over the cap: got %q, want %q", got, want)
+	}
+}

internal/outputvalidation/validator.go

@@ -11,6 +11,8 @@ import (
 
 	"github.com/santhosh-tekuri/jsonschema/v6"
 	"go.uber.org/zap"
+
+	"github.com/smart-mcp-proxy/mcpproxy-go/internal/stringutil"
 )
 
 // compileCount counts how many times a schema is actually compiled (vs served
@@ -225,10 +227,15 @@ func hashSchema(schemaJSON string) uint64 {
 	return h.Sum64()
 }
 
-// truncate shortens s to at most maxLen characters, appending "..." if truncated.
+// truncate shortens s to at most maxLen bytes, appending "..." if truncated.
+//
+// The cut goes through stringutil.UTF8Prefix: a jsonschema message quotes the
+// offending instance location, so a non-ASCII property name or value puts
+// multi-byte runes astride the cut, and a raw byte slice leaves the Reason
+// invalid UTF-8 (U+FFFD once JSON-encoded for the agent and the audit line).
 func truncate(s string, maxLen int) string {
 	if len(s) <= maxLen {
 		return s
 	}
-	return s[:maxLen-3] + "..."
+	return stringutil.UTF8Prefix(s, maxLen-3) + "..."
 }

internal/registries/description_utf8_test.go

@@ -0,0 +1,41 @@
+package registries
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+	"testing"
+	"unicode/utf8"
+)
+
+// PulseMCP descriptions are upstream-authored free text, so a non-Latin
+// description longer than the 300-byte display cap put multi-byte runes
+// astride the cut. The raw byte slice left ServerEntry.Description invalid
+// UTF-8, which the REST/Web UI JSON encoder rewrites to U+FFFD (round 3 of
+// #1647/#1664/#1665).
+//
+// The 1-byte padding covers all three offsets inside a 3-byte rune.
+func TestPulseDescriptionTruncationIsValidUTF8(t *testing.T) {
+	for _, field := range []string{"short_description", "EXPERIMENTAL_ai_generated_description"} {
+		for pad := 0; pad < 3; pad++ {
+			desc := strings.Repeat("a", pad) + strings.Repeat("語", 200)
+			payload := fmt.Sprintf(`{"servers":[{"name":"srv",%q:%q}]}`, field, desc)
+
+			var raw map[string]interface{}
+			if err := json.Unmarshal([]byte(payload), &raw); err != nil {
+				t.Fatalf("fixture is not JSON: %v", err)
+			}
+			entries := parsePulseWithoutGuesser(raw)
+			if len(entries) != 1 {
+				t.Fatalf("%s pad=%d: parsed %d entries, want 1", field, pad, len(entries))
+			}
+			got := entries[0].Description
+			if len(got) > 300 {
+				t.Errorf("%s pad=%d: description is %d bytes, over the 300-byte cap", field, pad, len(got))
+			}
+			if !utf8.ValidString(got) {
+				t.Errorf("%s pad=%d: description is not valid UTF-8 (cut inside a rune)", field, pad)
+			}
+		}
+	}
+}

internal/registries/search.go

@@ -9,8 +9,13 @@ import (
 	"strings"
 
 	"github.com/smart-mcp-proxy/mcpproxy-go/internal/experiments"
+	"github.com/smart-mcp-proxy/mcpproxy-go/internal/stringutil"
 )
 
+// pulseDescriptionMaxBytes caps the description carried on a Pulse registry
+// search result.
+const pulseDescriptionMaxBytes = 300
+
 // Sentinel errors for single-server lookup. Surfaces map these to stable error
 // codes (registry_not_found / server_not_found) for cross-surface consistency
 // (CN-004).
@@ -702,19 +707,15 @@ func parsePulseWithoutGuesser(rawData interface{}) []ServerEntry {
 			Name: name,
 		}
 
-		// Try to get description from multiple fields
+		// Try to get description from multiple fields. The display cap goes
+		// through stringutil.UTF8Prefix: a registry description is
+		// upstream-authored free text, and a raw byte cut at 300 can land
+		// inside a multi-byte rune, leaving the entry invalid UTF-8 (U+FFFD
+		// once the REST/Web UI encoder sees it).
 		if shortDesc, ok := itemMap["short_description"].(string); ok && shortDesc != "" {
-			if len(shortDesc) > 300 {
-				server.Description = shortDesc[:300]
-			} else {
-				server.Description = shortDesc
-			}
+			server.Description = stringutil.UTF8Prefix(shortDesc, pulseDescriptionMaxBytes)
 		} else if aiDesc, ok := itemMap["EXPERIMENTAL_ai_generated_description"].(string); ok && aiDesc != "" {
-			if len(aiDesc) > 300 {
-				server.Description = aiDesc[:300]
-			} else {
-				server.Description = aiDesc
-			}
+			server.Description = stringutil.UTF8Prefix(aiDesc, pulseDescriptionMaxBytes)
 		} else {
 			server.Description = noDescAvailable
 		}

internal/storage/activity.go

@@ -32,14 +32,20 @@ func parseActivityKey(key []byte) string {
 
 // truncateResponse truncates a response string if it exceeds maxSize.
 // Returns the (potentially truncated) string and whether truncation occurred.
+//
+// The cut goes through truncateRuneSafe, the same in-package helper the
+// per-server tool-call records use: an upstream response is arbitrary UTF-8,
+// and a raw byte cut at maxSize can land inside a multi-byte rune. The record
+// is then persisted and served over /api/v1/activities with invalid UTF-8,
+// which JSON encoding rewrites to U+FFFD.
 func truncateResponse(response string, maxSize int) (string, bool) {
 	if maxSize <= 0 {
 		maxSize = DefaultMaxResponseSize
 	}
 	if len(response) <= maxSize {
 		return response, false
 	}
-	return response[:maxSize] + "...[truncated]", true
+	return truncateRuneSafe(response, maxSize) + truncationMarker, true
 }
 
 // SaveActivity stores an activity record in BBolt.

internal/storage/activity_utf8_truncation_test.go

@@ -0,0 +1,45 @@
+package storage
+
+import (
+	"strings"
+	"testing"
+	"unicode/utf8"
+)
+
+// Round 3 of the UTF-8 truncation sweep (#1647, #1664, #1665): the activity
+// log's own per-record cut was still a raw byte slice, so a response whose
+// cut offset lands inside a multi-byte rune was persisted — and served over
+// /api/v1/activities and the CSV export — as invalid UTF-8, which JSON
+// encoding rewrites to U+FFFD.
+//
+// Every offset inside a 3-byte rune is exercised, because only two of the
+// three are mid-rune: a single fixed cap would pass by luck.
+func TestTruncateResponseKeepsRuneBoundary(t *testing.T) {
+	body := strings.Repeat("語", 200) // 3 bytes per rune
+
+	for _, maxSize := range []int{100, 101, 102, 103, 104, 105} {
+		got, truncated := truncateResponse(body, maxSize)
+		if !truncated {
+			t.Fatalf("maxSize=%d: expected truncation of a %d-byte body", maxSize, len(body))
+		}
+		kept := strings.TrimSuffix(got, "...[truncated]")
+		if !utf8.ValidString(kept) {
+			t.Errorf("maxSize=%d: stored response ends in a partial rune: % x",
+				maxSize, []byte(kept)[max(0, len(kept)-4):])
+		}
+		if len(kept) > maxSize {
+			t.Errorf("maxSize=%d: kept %d bytes, which is over the cap", maxSize, len(kept))
+		}
+	}
+}
+
+// ASCII behaviour must not move: the cap is still applied exactly.
+func TestTruncateResponseASCIIUnchanged(t *testing.T) {
+	got, truncated := truncateResponse(strings.Repeat("x", 500), 100)
+	if !truncated {
+		t.Fatal("expected truncation")
+	}
+	if want := strings.Repeat("x", 100) + "...[truncated]"; got != want {
+		t.Errorf("ASCII cut moved:\n got %q\nwant %q", got, want)
+	}
+}