This document describes the internal architecture of MCPProxy, including the management service, runtime lifecycle, and event system.
The management service (internal/management/) provides a centralized business logic layer for upstream server management operations, eliminating code duplication across CLI, REST API, and MCP interfaces.
┌─────────────────────────────────────────────────────────────┐
│ Client Interfaces │
├───────────────┬─────────────────┬───────────────────────────┤
│ CLI Commands │ REST API │ MCP Protocol │
│ (upstream) │ (/api/v1/*) │ (upstream_servers tool) │
└───────┬───────┴────────┬────────┴───────────┬───────────────┘
│ │ │
└────────────────┼────────────────────┘
│
▼
┌─────────────────────┐
│ Management Service │
│ (internal/mgmt/) │
└──────────┬──────────┘
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Runtime │ │ Config │ │ Events │
│Operations│ │ Gates │ │ Emitter │
└──────────┘ └──────────┘ └──────────┘
-
Service Interface (
service.go:16-102): Defines all management operations- Single-server:
RestartServer(),EnableServer(),DisableServer() - Bulk operations:
RestartAll(),EnableAll(),DisableAll() - Diagnostics:
GetServerHealth(),RunDiagnostics() - Server CRUD:
AddServer(),RemoveServer(),QuarantineServer() - Tool operations:
GetServerTools(),TriggerOAuthLogin()
- Single-server:
-
Configuration Gates: All operations respect centralized configuration guards
disable_management: Blocks all write operations when trueread_only_mode: Blocks all configuration modifications
-
Bulk Operations (
service.go:243-388): Efficient multi-server management- Sequential execution with partial failure handling
- Returns
BulkOperationResultwith success/failure counts - Collects per-server errors in results map
- Continues on individual failures, reports aggregate results
-
Event Integration: All operations emit events through event bus
servers.changed: Notifies UI of server state changes- Triggers SSE updates to web UI and tray application
- Enables real-time synchronization across interfaces
- Code Deduplication: 40%+ reduction in duplicate code across interfaces
- Consistent Behavior: All interfaces use identical business logic
- Centralized Validation: Configuration gates enforced in one place
- Easier Testing: Unit tests cover all interfaces through service layer
- Future Extensibility: New interfaces can reuse existing service methods
// CLI usage (cmd/mcpproxy/upstream_cmd.go:547-636)
result, err := client.RestartAll(ctx)
fmt.Printf(" Total servers: %d\n", result.Total)
fmt.Printf(" ✅ Successful: %d\n", result.Successful)
fmt.Printf(" ❌ Failed: %d\n", result.Failed)
// REST API usage (internal/httpapi/server.go:772-866)
mgmtSvc := s.controller.GetManagementService().(ManagementService)
result, err := mgmtSvc.RestartAll(r.Context())
s.writeSuccess(w, result)
// MCP protocol usage (future integration)
result, err := mgmtService.RestartAll(ctx)
return mcpResponse(result)The runtime package (internal/runtime/) provides core non-HTTP lifecycle management, separating concerns from the HTTP server layer.
- Configuration Management: Centralized config loading, validation, and hot-reload
- Background Services: Connection management, tool indexing, and health monitoring
- State Management: Thread-safe status tracking and upstream server state
- Event System: Real-time event broadcasting for UI and SSE consumers
The event bus enables real-time communication between runtime and UI components:
Event Types:
servers.changed- Server configuration or state changesconfig.reloaded- Configuration file reloaded from disk
Event Flow:
- Runtime operations trigger events via
emitServersChanged()andemitConfigReloaded() - Events are broadcast to subscribers through buffered channels
- Server forwards events to tray UI and SSE endpoints
- Tray menus refresh automatically without file watching
- Web UI receives live updates via
/eventsSSE endpoint
SSE Integration:
/eventsendpoint streams both status updates and runtime events- Automatic connection management with proper cleanup
- JSON-formatted event payloads for easy consumption
Every change to the server set, or to any server's fields, is one config
commit (Runtime.Commit, internal/runtime/commit.go). The live
configuration is the only source of server membership and fields; the
upstreams rows in config.db are its projection, written by the commit.
SaveConfiguration never rebuilds the server list from storage, so a removed
server cannot be resurrected by a later save.
A commit holds configCommitMu from reading the live config until the new
config is published and runs one pipeline:
- Build: clone the live config; the caller mutates the copy through a
CommitTx(CreateServer,UpdateServer,SetEnabled,SetQuarantined,RemoveServer,ReplaceConfig,ReconcileLive,MutateGlobal). - Gate: the #937 admission gate, against the stored view.
- Prepare:
configsvc.Prepareassigns the commit generationGand per-server generations (ServerGen{Gen, Incarnation}; an unchanged entry keeps its generation, a re-created name gets a new incarnation). - Locks: the approval lock of every server step 6 writes, sorted:
changed servers, servers with an
UnderApprovalLockhook, removals, replacements, OAuth-state clears and projection-repair saves of a server the commit did not change (a repair can rewrite the stored trust and quarantine policy a discovery pass reads under that lock). A global approval-policy change (quarantine_enabled) takes every configured server's lock. Dropping the lock for an unchanged server whose row is repaired reopens the discovery-versus-repair race (#1571). - Disk (only when a path is set and the commit writes the file): the raw
file is captured, then the new file is written. A file reload
(
ReloadConfiguration) never writes the file, so it skips this step. - Storage: one bbolt transaction (rows with their stamps, deletions,
meta.config_commit_generation), then the index purge for removed, quarantined and disabled servers, then the in-memory purge (last-good tool snapshot, capture stamp). A removed or replaced server loses its whole purge set in that transaction (internal/storage/purge.go): tool and prompt approvals, OAuth tokens and DCR client (exactnameorname_<16 hex>key), OAuth completion events, scan reports and jobs, the integrity baseline, tool-call history and identities, and tool hashes. The in-memory purge also bumps the server's last-good purge sequence, so a local reindex that copied the snapshot before the removal drops its copy (it re-checks the sequence under the approval lock). The approval locks are released, then — still under the commit lock — the purged servers' OAuth refresh schedules are cancelled and their inspection exemptions revoked (leaf locks only), so no later commit's same-name replacement can lose a fresh schedule or exemption to a name-keyed clear. On a storage failure nothing is published; when step 5 wrote the file, the captured file is restored byte for byte (or removed if it did not exist). A failed reload wrote nothing, so the file keeps the operator's edit unapplied: the next reload applies it, but the next disk-writing commit (SaveConfiguration, a server mutation,ApplyConfig) writes the live config over it. - Advance:
Manager.AdvanceGenerationsrecords every configured server's committed generation in the upstream manager (and a tombstone for a removed one), unregisters the instance of every removed or replaced server and of every server whose connection settings,enabledorquarantinedchanged, and restamps the retained instances with their new config and generation (Spec 116 PR3). - Publish:
configsvc.Publishand the legacyr.cfgswap.
After the lock is released the commit disconnects the instances step 7 unregistered, reconciles the supervisor when an exemption was revoked, drops cached scan verdicts (purge observers; the scanner's verdict cache is also keyed by incarnation, so a replacement is never served the old entry before the observer runs), schedules connect workers, requests discovery for restamped instances and sends its events once.
Generation-bound workers and supervisor actions (Spec 116 PR3, #1563).
Everything that changes the manager's registry or connects an instance
asynchronously is bound to the commit generation it was scheduled for: the
commit's connect/disconnect workers (AddServerAt, RemoveServerAt), the
supervisor's actions (generation from the snapshot they were planned from),
RestartServer, ForceReconnectAll and Docker recovery. The manager
re-checks the generation against its desired view under captureMu +
Manager.mu, in the same critical section as the change, and a stale call
returns ErrStaleGeneration and does nothing (logged at debug level). An
action on an existing instance (a supervisor disconnect, an OAuth logout's
disconnect, a restart or reconnect replacement through ReplaceInstanceAt)
is also bound to that instance's InstanceID, because a restart installs a
new instance at the same generation. Every instance leaves the registry
through one unregisterLocked, which marks it unregistered so it never
connects again and retires it (the retirement barrier below).
Discovery and review captures carry the generation of the instance they
listed and are dropped when it is not the stored row's; a last-good tool
snapshot carries it too, so an approval-driven reindex never reapplies an
old endpoint's tools after an endpoint change (it re-lists instead).
scripts/check-upstream-write-sites.sh also fails on an unbound manager
mutation in internal/runtime or internal/server, and on any change to
Manager.clients outside unregisterLocked/installLocked.
Retirement barrier (Spec 116 PR4, #1564). Each upstream instance owns a
dial gate (internal/upstream/dialgate). Everything the instance sends goes
through it: every TCP dial of its HTTP, SSE and OAuth traffic and every write
on a connection it dialed (a per-instance gated *http.Transport), every
process it launches (stdio, Docker run and the local launcher start in an
admitted section that also registers the process tree with the gate, so
Retire kills it; the OAuth browser opener is only admitted, since the
browser it hands the URL to is not the instance's process), every MCP request and notification its mcp-go transport sends (a
capability-preserving dialgate.WrapTransport), and every OAuth preflight,
discovery, registration, code exchange and refresh (oauth.Env carries the
instance transport). unregisterLocked retires the instance before it
returns, so a commit retires a removed, replaced or quarantined instance
before it publishes, and a restart, ForceReconnectAll or Docker recovery
retires the old instance before it installs the new one. Retire sets the
latch, cancels the gate context, closes tracked connections, kills tracked
process groups, waits for admitted sections that began before the latch (a
write-lock barrier; each section is one leaf operation whose connection it
has already closed, so the wait is bounded), and sweeps again. After it
returns nothing new is launched, dialed or written: a dial, a launch or a
write fails with ErrRetired without touching the network, and a tool call
on the retired instance fails as "server unavailable" without marking the
replacement unhealthy. Teardown (the MCP close, the process-group kill,
docker stop) still runs; the manual OAuth flow and token refresh refuse a
retired instance and report it as "server not found". A launched process
whose tree cannot be contained (a Windows Job Object that cannot be created
or assigned) is killed and its start fails. On Windows the child is
created CREATE_SUSPENDED, assigned to its Job Object and only then resumed
(NtResumeProcess), so it cannot fork a descendant outside the Job. On Unix the launcher keeps an
exited leader unreaped while a descendant it left in its process group runs,
so the group ID stays pinned and the retirement kill (or Stop) reaches that
descendant; a group whose leader was reaped is never signalled, since its ID
may have been reused. scripts/check-upstream-write-sites.sh also fails on
ungated HTTP in internal/oauth, internal/upstream/core and
internal/transport (allowlist: scripts/ungated-http-allowlist.txt).
Lock order: the gate's lock is innermost (captureMu → Manager.mu →
Gate.mu → Gate.connsMu) and is held only around one dial, one write or
one process start.
OAuth credential binding (Spec 116 PR2b, #1607). Each committed server
view carries an OAuth epoch next to its incarnation: the generation of the
commit that created the name or last cleared its OAuth state because its OAuth
configuration changed. {Incarnation, OAuthEpoch} is the server's credential
binding. The commit stamps it on the snapshot, the upstreams row and the
manager's desired view in one pass, and the manager binds every instance it
builds to the binding of the generation it was built for (never read from the
row). Every OAuth write and clear an instance makes goes through a storage
method that checks the row's binding in the same bbolt transaction: token
saves (a login's code exchange with its paired client, a refresh), DCR client
saves, completion events, the 5xx token clear, the config-build DCR hygiene
clears and the clears after a rejected refresh or login. The removal purge
deletes the row and the OAuth-config clear raises the row's epoch in the
transaction that deletes the credentials, so an old instance's write that was
already past its network round-trip either committed first (and was deleted)
or finds the row gone or re-bound and writes nothing; a same-name re-add never
sees the old server's tokens. Reads are epoch-filtered only: a bound read
treats a record of another epoch as absent, but does not check the row, so a
removed name stays unreadable because its removal purges the records. Every
write to the oauth_tokens bucket takes a new Revision from the bucket
sequence, and a binding-conditional clear acts only on the revision its
caller inspected, so a parked clear never erases what a same-binding
replacement (a restart) rewrote. The one exception is the login flow's
stale-DCR clear (ClearOAuthClientIfBoundAndClientID): it holds the client
id it sent, not a record it read, so it checks the binding and that client
id; a parked login-path clear can still erase a same-binding rewrite that
kept the same client id. Refresh flights, the terminal latch, the cooldown
and the re-registration state are keyed by (server key, binding), so an
old instance never joins the flight of a replacement with a different
binding (a removal and re-add, or an OAuth-config change) or receives its
token. A same-binding replacement (a restart) shares the key and can join
the old instance's flight; a joiner whose flight failed because its
initiator was retired retries on its own transport rather than taking that
answer. A refresh schedule built from a token save records the binding the
token was saved under, and a reactive flight's terminal outcome is applied
only to a schedule of the flight's binding, checked when that schedule is
selected. A late save notification from an older binding (bindings are
commit generations, so older is known) does not replace the schedule of a
newer one. The exception is a schedule loaded from storage at startup: it
carries the zero binding, so no binding comparison is made for it, and a
terminal outcome of any flight for the server applies to it (the resolver
check that the flight ran under the server's current binding still
applies). Its first token save replaces it with a bound schedule. A token
store guards its 5xx-recovery clear with the revision the refresh flight
itself wrote or read, never a later re-read, so a same-binding rewrite with
the same access token survives the clear. Before it builds an instance the manager also checks that the
stored row still carries exactly the committed view it holds (one bbolt
View), so a stale worker builds no OAuth config at all. Logout and the
standalone CLI logout (ClearOAuthTokenForStandaloneLogout, one exact key)
stay current-state clears. scripts/check-upstream-write-sites.sh fails on an
unconditional OAuth mutator in internal/upstream or internal/oauth. No
lock edge is added: the checks are bbolt leaves taken with no runtime lock
held, and the coordinator's mutex stays a leaf.
Incarnation-bound approvals (Spec 116 PR2). Each server has an
incarnation that changes only when its name is removed and created again.
Every approval writer (tool approve/block/toggle, the unquarantine baseline
promotion, prompt approvals, the scanner's baseline and blocks) captures the
incarnation when it starts — or takes the client's expected_incarnation —
and re-checks it against the stored row under the server's approval lock,
the lock the removal purges under. A mismatch writes nothing and returns
ErrServerReplaced (REST 409 server_replaced). Scanner approval
unquarantines through a commit with ExpectIncarnation. Asynchronous
writers that can outlive a removal (scan job and report saves, identity
registration, tool-call history) use storage methods that check the row's
incarnation in the same bbolt transaction as the write. A discovery pass makes its final eligibility check (stored row
present, enabled, not quarantined, generation unchanged) and its index writes
in one hold of the server's approval lock, and approval policy is read from
storage, so a pass can never re-index a server a commit has just quarantined
or removed.
Lock order (outermost first): configCommitMu → server approval locks
(sorted; released before step 7) → storage.Manager.mu/bbolt; configCommitMu
→ upstream.Manager.captureMu (W) → Manager.mu in step 7; configCommitMu
→ configsvc update lock → Runtime.mu. Code holding any of the inner locks
must never call Commit. scripts/check-upstream-write-sites.sh (CI) fails
when an upstreams row is written anywhere but the storage package and
internal/runtime/commit*.go.
Initialization:
- Runtime created with config, logger, and manager dependencies
- Background initialization starts server connections and tool indexing
- Status updates broadcast through event system
Background Services:
- Connection Management: Periodic reconnection attempts with exponential backoff
- Tool Indexing: Automatic discovery and search index updates every 15 minutes
- Configuration Sync: File-based config changes trigger runtime resync
Shutdown:
- Graceful context cancellation cascades to all background services
- Upstream servers disconnected with proper subprocess and Docker container cleanup
- Resources closed in dependency order (upstream → cache → index → storage)
Subprocess Shutdown Flow:
- Graceful Close (10s timeout): Close MCP client connection, wait for subprocess to exit cleanly
- Force Kill (9s timeout): If graceful close fails, send SIGTERM to process group, poll for exit, then SIGKILL
| Timeout | Value | Purpose |
|---|---|---|
| MCP Client Close | 10s | Wait for graceful stdin/stdout close |
| SIGTERM → SIGKILL | 9s | Time between graceful and force kill |
| Docker Cleanup | 30s | Container stop/kill timeout |
See Shutdown Behavior for detailed documentation.
The tray application uses a robust state machine architecture for reliable core management.
- Normal flow:
StateInitializing→StateLaunchingCore→StateWaitingForCore→StateConnectingAPI→StateConnected - Error states:
StateCoreErrorPortConflict,StateCoreErrorDBLocked,StateCoreErrorGeneral,StateCoreErrorConfig - Recovery states:
StateReconnecting,StateFailed,StateShuttingDown
- Process Monitor (
cmd/mcpproxy-tray/internal/monitor/process.go): Monitors core subprocess lifecycle - Health Monitor (
cmd/mcpproxy-tray/internal/monitor/health.go): Performs socket-aware HTTP health checks on core API (/healthz,/readyz) - State Machine (
cmd/mcpproxy-tray/internal/state/machine.go): Manages state transitions and automatic retry logic
Core process exit codes are mapped to specific state machine events:
- Exit code 2 (port conflict) →
EventPortConflict - Exit code 3 (database locked) →
EventDBLocked - Exit code 4 (config error) →
EventConfigError - Exit code 5 (permission error) →
EventPermissionError - Exit code 6 (shutdown timeout: graceful shutdown exceeded its 60s hard deadline after SIGINT/SIGTERM) →
EventGeneralError(no dedicated event; a restart is the right response) - Other errors →
EventGeneralError
Error states automatically retry core launch with exponential backoff:
StateCoreErrorGeneral: 2 retries with 3s delay (3 total attempts)StateCoreErrorPortConflict: 2 retries with 10s delayStateCoreErrorDBLocked: 3 retries with 5s delay- After max retries exceeded → transitions to
StateFailed
MCPPROXY_TRAY_SKIP_CORE=1- Skip core launch (for development)MCPPROXY_CORE_URL=http://localhost:8085- Custom core URLMCPPROXY_TRAY_PORT=8090- Custom tray port
- Socket Communication - Tray-core IPC details
- CLI Management Commands - CLI reference
- Configuration - Full configuration reference