Skip to content

Commit 607a80e

Browse files
authored
fix(server): record redacted mutation payloads and target server in the activity log (#1149)
* fix(server): record redacted mutation payloads and the target server in activity (#1146) The activity records for the two mutating built-in tools carried only {operation, name}. Every payload field — env_json, headers_json, oauth_json, isolation_json, args_json, command, url, protocol, enabled, trust_mode, init_timeout, registry, id — was dropped, so the audit trail could say that a server was patched but never what changed. The read-only read_cache tool logged more than either mutator. On top of that all 18 emit sites passed targetServer="" and EmitActivityInternalToolCall omits an empty target_server, so the Activity Log Server column rendered "-" and `activity list --server <name>` missed every mutation. Record the FULL argument set instead of a hand-maintained allowlist (the allowlist is what rotted into this bug), redacted through the existing internal/oauth helpers, and thread the resolved server name through every emit site. update/patch additionally record the resolved config diff: patch semantics (deep merge, `null` means remove) make what was requested genuinely different from what changed. Redaction policy: redacted-but-PRESENT. oauth.RedactEnvValues / RedactStringHeaders / RedactURLQueryParams mask values with `••••<last2> (<N> chars)`, so the operator sees THAT API_KEY changed and which credential it was, while ${keyring:…}/${env:…} references and ordinary config (protocol, LOG_LEVEL, image python:3.12) stay readable. The walk is structure-aware: marshalling to JSON and running the regex scrubber over it rewrites across the closing quote and yields invalid JSON. Each string leaf is capped at 512 bytes so one mutation carrying a large env map cannot bloat the store. config.RevealSecretHeaders is deliberately NOT honoured here — the activity store is persisted, CLI-exported and SSE-broadcast, a different durability class from the live `list` projection it governs. Also closes two pre-existing leaks found while wiring this up: config.FieldChange carries raw before/after VALUES, and both handleUpdateUpstream and handlePatchUpstream put configDiff.Modified straight into responseMap["changes"] and into a zap.Any audit line — so a patch echoed another server's env values, Authorization header and oauth.client_secret back to the calling agent, into main.log, into BBolt and over SSE. Both now render through the shared redactedConfigDiff. Field paths and the key set are preserved, which is what agents branch on (TestE2E_PatchPreservesOAuthConfig stays green); only values are masked. handleUpdateUpstream / handlePatchUpstream now return the resolved *config.ConfigDiff; both are unexported with a single caller each. * fix(server): mask argv secrets and stop caller-forged attribution in activity Cross-model review of #1146 found four ways the "record the full mutation payload" change could leak or be spoofed, plus one unannounced contract change. All five are addressed here. 1. Command-line ARGUMENT VALUES were recorded verbatim. argv has no key to judge by — the slice branch recursed with the parent key ("args"), so every element fell through redactActivityString unmasked. `uvx mcp-foo --api-key sk-live-…` therefore persisted the credential in BBolt, in the SSE payload and in `mcpproxy activity list`, on the exact path #1146 set out to seal. redactActivityArgv now masks by two rules: the flag names the credential (`--api-key VALUE` / `--api-key=VALUE`, delegated to the same env-var rule via the new oauth.IsSensitiveKeyName), and the value is recognisable on its own (internal/security's detector, reused rather than re-invented). Package names, subcommands, paths and ports still round-trip verbatim. 2. targetServer was taken from the unvalidated agent-supplied `name` and hoisted above the dispatch, so operations that never receive the request (list, list_quarantined) attributed their row to a server they never touched. activityAttributedOps is now an allowlist — unknown and unparsed operations default to unattributed — pinned to both operation switches in BOTH directions by an AST test, so a new mutating operation cannot silently lose its Server column either. 3. Populating targetServer let management chatter into /api/v1/activity/summary's top_servers/top_tools, which previously skipped those rows only by the accident of an empty ServerName. Those lists answer "which upstreams is this proxy talking to", so the management built-ins are now excluded explicitly (storage.IsManagementBuiltin — the same population CountsAsCall already names). The rows stay in the totals. 4. Capturing the whole request captured caller-supplied `_auth_*` keys, which internal/runtime copies onto ActivityRecord.UserID/UserEmail. Neither handler calls injectAuthMetadata, so those can only be client-supplied: a caller-controlled identity stamp on the audit row for a privileged mutation. Stripped with security.StripInternalArgs. 5. The masked changes.modified[*].from/.to is agent-observable. Non-secret values already round-trip truthfully (now pinned by a test); the upstream_servers description says so. Goldens regenerated and the delta enumerated on all three surfaces. * fix(server): close the remaining activity-redaction leaks (round 3) Key-name matching was still the only rule on several activity paths, so a credential whose NAME gave nothing away reached BBolt, the SSE payload and `mcpproxy activity list` in the clear. - argv: the inline `--flag=VALUE` form consulted only the flag-name rule, so `--endpoint=ghp_…` leaked while the identical `--endpoint ghp_…` masked. redactActivityString now ends in the value-shaped detector for EVERY leaf, which keeps the two spellings in step by construction. - headers: oauth's header matcher only recognised whole delimiter-separated segments, so `X-AuthToken` / `X-SecretValue` matched nothing. It now also splits CamelCase (X-Author-ID / X-Monkey-ID stay readable), and header values additionally go through the detector — neither rule alone is sufficient. - `upstream_servers add` emitted a second, unredacted config_change row built from the resolved values; it goes through the same redaction now. - audit surfaces mask with the new oauth.AuditMaskValue (a fixed marker) rather than MaskValue's `••••<last2> (<N> chars)`: persisted, streamed and exported rows must not carry a credential's exact length and trailing bytes. The interactive/read surfaces keep MaskValue — the write path's unmask contract depends on its exact rendering. - a `*_json` argument that does not parse now fails CLOSED. Heuristic string scrubbing is regex-shaped and missed `{"API_KEY" "sk-live-…"}` entirely; the row records that the parameter was sent and its size, not its bytes. oauth gains *With variants of the three redactors so the mask can vary without forking the "which name is sensitive" rules.
1 parent 84ef98c commit 607a80e

14 files changed

Lines changed: 1926 additions & 91 deletions

‎internal/httpapi/activity.go‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -754,12 +754,25 @@ func (s *Server) handleActivitySummary(w http.ResponseWriter, r *http.Request) {
754754
otherCount++
755755
}
756756

757-
// Count by server
757+
// Count by server / by tool — UPSTREAM traffic only.
758+
//
759+
// Issue #1146 gave the management built-ins a target_server so the
760+
// Activity Log could render a Server column and --server could filter
761+
// on it. These two lists answer a different question ("which upstreams
762+
// is this proxy talking to"), and they used to skip those rows only by
763+
// the accident of an empty ServerName. Excluding them explicitly keeps
764+
// a burst of config edits from reading as traffic to the server being
765+
// configured, and keeps "github:upstream_servers" — a built-in no
766+
// upstream owns — out of the top-tools list. The rows stay in
767+
// totalCount: they are real activity, just not upstream traffic.
768+
if storage.IsManagementBuiltin(a) {
769+
continue
770+
}
771+
758772
if a.ServerName != "" {
759773
serverCounts[a.ServerName]++
760774
}
761775

762-
// Count by tool (server:tool)
763776
if a.ServerName != "" && a.ToolName != "" {
764777
key := a.ServerName + ":" + a.ToolName
765778
toolCounts[key]++
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
package httpapi
2+
3+
import (
4+
"encoding/json"
5+
"fmt"
6+
"net/http"
7+
"net/http/httptest"
8+
"testing"
9+
"time"
10+
11+
"github.com/stretchr/testify/assert"
12+
"github.com/stretchr/testify/require"
13+
"go.uber.org/zap"
14+
15+
"github.com/smart-mcp-proxy/mcpproxy-go/internal/contracts"
16+
"github.com/smart-mcp-proxy/mcpproxy-go/internal/storage"
17+
)
18+
19+
// Cross-model review of #1146, finding 3. Giving the management built-ins a
20+
// target_server fixed the Activity Log's Server column — and, as a side effect,
21+
// let their rows into /api/v1/activity/summary's per-server aggregation, which
22+
// previously skipped them only by the accident of an empty ServerName.
23+
//
24+
// top_servers / top_tools answer "which upstreams is this proxy talking to".
25+
// upstream_servers and quarantine_security are mcpproxy's own housekeeping —
26+
// the population storage.CountsAsCall already excludes by name — so a burst of
27+
// config edits must not read as traffic to the server being configured, and
28+
// "github:upstream_servers" is not a tool github owns.
29+
func TestActivitySummary_ManagementChatterIsNotUpstreamTraffic(t *testing.T) {
30+
const (
31+
realCalls = 3
32+
managementRows = 10
33+
managementTool = "upstream_servers"
34+
quarantineTool = "quarantine_security"
35+
quarantineCount = 4
36+
)
37+
now := time.Now().UTC()
38+
39+
var activities []*storage.ActivityRecord
40+
for i := 0; i < realCalls; i++ {
41+
activities = append(activities, &storage.ActivityRecord{
42+
ID: fmt.Sprintf("call-%d", i), Type: storage.ActivityTypeToolCall,
43+
ServerName: "github", ToolName: "search",
44+
Status: storage.ActivityStatusSuccess, Timestamp: now.Add(-time.Duration(i) * time.Minute),
45+
})
46+
}
47+
for i := 0; i < managementRows; i++ {
48+
activities = append(activities, &storage.ActivityRecord{
49+
ID: fmt.Sprintf("mgmt-%d", i), Type: storage.ActivityTypeInternalToolCall,
50+
ServerName: "github", ToolName: managementTool,
51+
Status: storage.ActivityStatusSuccess, Timestamp: now.Add(-time.Duration(i) * time.Minute),
52+
})
53+
}
54+
for i := 0; i < quarantineCount; i++ {
55+
activities = append(activities, &storage.ActivityRecord{
56+
ID: fmt.Sprintf("quar-%d", i), Type: storage.ActivityTypeInternalToolCall,
57+
ServerName: "github", ToolName: quarantineTool,
58+
Status: storage.ActivityStatusSuccess, Timestamp: now.Add(-time.Duration(i) * time.Minute),
59+
})
60+
}
61+
62+
srv := NewServer(&summaryScopeController{activities: activities}, zap.NewNop().Sugar(), nil)
63+
64+
req := httptest.NewRequest(http.MethodGet, "/api/v1/activity/summary?period=24h", nil)
65+
req.Header.Set("X-API-Key", "test-key")
66+
w := httptest.NewRecorder()
67+
srv.ServeHTTP(w, req)
68+
require.Equal(t, http.StatusOK, w.Code)
69+
70+
var resp struct {
71+
Success bool `json:"success"`
72+
Data contracts.ActivitySummaryResponse `json:"data"`
73+
}
74+
require.NoError(t, json.NewDecoder(w.Body).Decode(&resp))
75+
require.True(t, resp.Success)
76+
77+
// Every row is still in the log and still in the denominator.
78+
assert.Equal(t, realCalls+managementRows+quarantineCount, resp.Data.TotalCount,
79+
"the management rows are real activity and must stay in the total")
80+
81+
require.Len(t, resp.Data.TopServers, 1)
82+
assert.Equal(t, "github", resp.Data.TopServers[0].Name)
83+
assert.Equal(t, realCalls, resp.Data.TopServers[0].Count,
84+
"config edits to a server are not traffic to it")
85+
86+
for _, tool := range resp.Data.TopTools {
87+
assert.NotEqual(t, managementTool, tool.Tool,
88+
"upstream_servers is mcpproxy's built-in, not a tool the upstream owns")
89+
assert.NotEqual(t, quarantineTool, tool.Tool,
90+
"quarantine_security is mcpproxy's built-in, not a tool the upstream owns")
91+
}
92+
require.Len(t, resp.Data.TopTools, 1)
93+
assert.Equal(t, "github", resp.Data.TopTools[0].Server)
94+
assert.Equal(t, "search", resp.Data.TopTools[0].Tool)
95+
assert.Equal(t, realCalls, resp.Data.TopTools[0].Count)
96+
}

‎internal/oauth/logging.go‎

Lines changed: 108 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,28 @@ var sensitiveHeaderSegments = map[string]bool{
4747

4848
var headerNameSegmentPattern = regexp.MustCompile(`[a-z0-9]+`)
4949

50+
// headerNameCamelSegmentPattern splits a CamelCase header name on its case
51+
// transitions. Issue #1146 (review round 3): the delimiter-based pass above
52+
// only ever sees whole `-`/`_`-separated segments, so `X-AuthToken` collapsed
53+
// to the single segment "authtoken", matched nothing, and a credential under a
54+
// custom header name was recorded and logged in the clear. Splitting on case
55+
// transitions as well recovers "Auth" and "Token" without reopening the
56+
// substring false positives the segment rule exists to prevent: "Author" and
57+
// "Monkey" are still single segments that equal no marker.
58+
//
59+
// Only mixed-case words are extracted here — an all-caps run like
60+
// `X-CUSTOM-TOKEN` has no case transition to split on and is already covered by
61+
// the delimiter pass.
62+
var headerNameCamelSegmentPattern = regexp.MustCompile(`[A-Z][a-z0-9]*`)
63+
64+
// IsSensitiveHeaderName reports whether an HTTP header NAME looks like it
65+
// carries a credential. Exported for callers that must decide masking from a
66+
// name they hold apart from the value (the activity/audit redactor, issue
67+
// #1146) so the rule stays defined in exactly one place.
68+
func IsSensitiveHeaderName(name string) bool {
69+
return isSensitiveHeaderKey(name)
70+
}
71+
5072
func isSensitiveHeaderKey(name string) bool {
5173
if sensitiveHeaders[strings.ToLower(name)] {
5274
return true
@@ -58,6 +80,12 @@ func isSensitiveHeaderKey(name string) bool {
5880
}
5981
}
6082

83+
for _, segment := range headerNameCamelSegmentPattern.FindAllString(name, -1) {
84+
if sensitiveHeaderSegments[strings.ToLower(segment)] {
85+
return true
86+
}
87+
}
88+
6189
return false
6290
}
6391

@@ -159,13 +187,28 @@ func RedactHeaders(headers http.Header) map[string]string {
159187
// "Convert to secret" affordance work on the UI side because the user
160188
// can confirm a recognisable suffix before approving.
161189
func RedactStringHeaders(headers map[string]string) map[string]string {
190+
return RedactStringHeadersWith(headers, MaskValue)
191+
}
192+
193+
// RedactStringHeadersWith is RedactStringHeaders with a caller-chosen mask.
194+
//
195+
// Issue #1146 (review round 3): MaskValue's `••••<last2> (<N> chars)` rendering
196+
// is a deliberate trade for the INTERACTIVE surfaces — it lets an operator
197+
// recognise which token is configured, and the write path (UnmaskHeaders)
198+
// depends on being able to recognise it when a client echoes it back. On the
199+
// AUDIT surfaces that trade is wrong: those rows are persisted, streamed and
200+
// exported, so the length and trailing bytes become a durable fingerprint of
201+
// every credential. Those callers pass AuditMaskValue instead. Splitting the
202+
// mask out — rather than forking the "which name is sensitive" rules — keeps
203+
// one definition of sensitivity behind both renderings.
204+
func RedactStringHeadersWith(headers map[string]string, mask func(string) string) map[string]string {
162205
if headers == nil {
163206
return nil
164207
}
165208
redacted := make(map[string]string, len(headers))
166209
for key, value := range headers {
167210
if isSensitiveHeaderKey(key) {
168-
redacted[key] = MaskValue(value)
211+
redacted[key] = mask(value)
169212
} else {
170213
redacted[key] = RedactSensitiveData(value)
171214
}
@@ -191,6 +234,16 @@ var sensitiveEnvMarkers = []string{
191234
"DSN", "CONNECTION_STRING", "CONN_STR",
192235
}
193236

237+
// IsSensitiveKeyName reports whether a NAME looks like it holds a secret, using
238+
// the same case-insensitive marker match RedactEnvValues applies to env-var
239+
// keys. Exported for callers that have to decide masking from a name they hold
240+
// separately from the value — e.g. a command-line flag naming the argv token
241+
// that follows it (issue #1146), where there is no map to hand to
242+
// RedactEnvValues.
243+
func IsSensitiveKeyName(name string) bool {
244+
return isSensitiveEnvKey(name)
245+
}
246+
194247
// isSensitiveEnvKey reports whether an env var name looks like it holds a
195248
// secret, based on a case-insensitive substring match against
196249
// sensitiveEnvMarkers.
@@ -218,12 +271,18 @@ func isSensitiveEnvKey(name string) bool {
218271
// with a RedactSensitiveData pass over the value as a defence-in-depth fallback
219272
// for embedded secrets (it leaves ordinary values like `debug` untouched).
220273
func RedactEnvValues(env map[string]string) map[string]string {
274+
return RedactEnvValuesWith(env, MaskValue)
275+
}
276+
277+
// RedactEnvValuesWith is RedactEnvValues with a caller-chosen mask. See
278+
// RedactStringHeadersWith for why the audit surfaces need a different one.
279+
func RedactEnvValuesWith(env map[string]string, mask func(string) string) map[string]string {
221280
if env == nil {
222281
return nil
223282
}
224283
redacted := make(map[string]string, len(env))
225284
for key, value := range env {
226-
redacted[key] = maskedEnvValue(key, value)
285+
redacted[key] = maskedEnvValueWith(key, value, mask)
227286
}
228287
return redacted
229288
}
@@ -239,11 +298,15 @@ func RedactEnvValues(env map[string]string) map[string]string {
239298
// embedded userinfo password and any credential query params are masked while
240299
// scheme/host/db stay readable.
241300
func maskedEnvValue(key, value string) string {
301+
return maskedEnvValueWith(key, value, MaskValue)
302+
}
303+
304+
func maskedEnvValueWith(key, value string, mask func(string) string) string {
242305
if isSensitiveEnvKey(key) {
243-
return MaskValue(value)
306+
return mask(value)
244307
}
245308
if strings.Contains(value, "://") {
246-
return RedactURLQueryParams(value)
309+
return RedactURLQueryParamsWith(value, mask)
247310
}
248311
return RedactSensitiveData(value)
249312
}
@@ -302,6 +365,12 @@ func isSensitiveQueryParam(name string) bool {
302365
// are labels, not secrets. A URL with no query, or no sensitive params, is
303366
// returned unchanged. On parse failure it falls back to the regex RedactURL.
304367
func RedactURLQueryParams(rawURL string) string {
368+
return RedactURLQueryParamsWith(rawURL, MaskValue)
369+
}
370+
371+
// RedactURLQueryParamsWith is RedactURLQueryParams with a caller-chosen mask.
372+
// See RedactStringHeadersWith for why the audit surfaces need a different one.
373+
func RedactURLQueryParamsWith(rawURL string, mask func(string) string) string {
305374
if rawURL == "" {
306375
return rawURL
307376
}
@@ -318,7 +387,7 @@ func RedactURLQueryParams(rawURL string) string {
318387
// below.
319388
if u.User != nil {
320389
if pw, hasPW := u.User.Password(); hasPW && !isConfigReference(pw) {
321-
u.User = url.UserPassword(u.User.Username(), MaskValue(pw))
390+
u.User = url.UserPassword(u.User.Username(), mask(pw))
322391
changed = true
323392
}
324393
}
@@ -351,7 +420,7 @@ func RedactURLQueryParams(rawURL string) string {
351420
if isConfigReference(decVal) {
352421
continue
353422
}
354-
parts[i] = key + "=" + url.QueryEscape(MaskValue(decVal))
423+
parts[i] = key + "=" + url.QueryEscape(mask(decVal))
355424
queryChanged = true
356425
}
357426
if queryChanged {
@@ -382,6 +451,39 @@ func isConfigReference(v string) bool {
382451
return configRefPattern.MatchString(v)
383452
}
384453

454+
// IsConfigReference is the exported form of isConfigReference, for callers
455+
// outside this package that must decide whether a value is a secret-store
456+
// LABEL rather than a secret (issue #1146: the activity redactor runs a
457+
// value-shaped detector over everything the name rules pass through, and a
458+
// reference must not be mangled by it).
459+
func IsConfigReference(v string) bool {
460+
return isConfigReference(v)
461+
}
462+
463+
// AuditMaskValue is the mask for surfaces that PERSIST and EXPORT what they
464+
// redact — the activity store, its SSE payloads and `mcpproxy activity list`
465+
// (issue #1146, review round 3).
466+
//
467+
// Unlike MaskValue it carries no length and no trailing bytes. On an
468+
// interactive surface those help an operator recognise which token is
469+
// configured and are re-read within seconds; written into an audit row they
470+
// become a durable fingerprint of the credential — a correlation handle across
471+
// records and a materially smaller search space for a low-entropy secret. The
472+
// audit surfaces have no need for the affordance, so they do not pay for it.
473+
//
474+
// ${keyring:…} / ${env:…} references still pass through unchanged: they are
475+
// labels pointing at the secret store, and masking them would erase exactly the
476+
// information the audit row exists to carry.
477+
func AuditMaskValue(v string) string {
478+
if v == "" {
479+
return "(empty)"
480+
}
481+
if isConfigReference(v) {
482+
return v
483+
}
484+
return "••••"
485+
}
486+
385487
// MaskValue renders a string secret as `••••<last2> (<N> chars)` for
386488
// human display. Returns "(empty)" for empty input, a 4-bullet preview
387489
// for values up to 4 characters (where revealing the last two would

0 commit comments

Comments
 (0)