Skip to content

Commit 0fc92a2

Browse files
MrScriptyMrScripty
authored
Merge pull request #4 from MrScripty/feat/issue3-mcp-observability
Implement opt-in persistent MCP interaction observability
2 parents 188beda + 418dd8d commit 0fc92a2

14 files changed

Lines changed: 2217 additions & 18 deletions

File tree

‎docs/plans/mcp-observability/execution-ledger.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Execution ledger
2+
3+
2026-10-14: Read issue #3 and all comments (none) via authorized GitHub access.
4+
Cloned Coding-Standards separately; baseline 188beda1fa477d21d576c233dd8c7f4c4c267d23.
5+
Core/Router, selected owners, MCP/catalog/runtime identity, contract discovery,
6+
input feedback and observer-v3 qualification/reconciliation inspected.
7+
Python 3.12.14 is inside the declared >=3.11,<3.13 range.
8+
User subsequently explicitly authorized push + PR after testing and review;
9+
merge and deployment remain excluded. No other repository modified.
10+
11+
2026-10-14: Implemented one recorder boundary and operator report;22 focused
12+
synthetic tests passed on Python 3.12.14 in 27.628 s. Independent GPT-6.1 Sol High
13+
review accepted exact source hashes after resolving seven findings, including
14+
rejection of prior private SQLite remnants. Observer-v3 suites passed 39 tests
15+
in 19.044 s. Contract projection check passed. Verifier initially exposed stale
16+
README/source input digests; refreshed through public verify_repository debug
17+
transport, then 73 suites/121 checks passed. Generated suite-inputs manifest added
18+
to allowed adjacent write set. Full Engine run remains active; baseline fixture
19+
failure I1 reproduced on a clean independent baseline clone, unchanged by this
20+
feature. Four-mode cold baseline/candidate comparison preserved counts, outputs
21+
and exact serialized byte totals. No raw traces or databases are deliverables.
22+
23+
2026-10-14: Final focused recorder+MCP+runtime-identity run passed 40 tests in 42.362 s
24+
(23 recorder,13 MCP,4 runtime identity). Final constants use one owner with the
25+
same defaults; reviewer accepted and rebound exact source hashes. Repeated
26+
four-configuration baseline/candidate response comparison passed on final source.
27+
The broad Engine run completed 93 tests (one known baseline failure), then was
28+
interrupted during the next owner module to parallelize remaining independent
29+
modules in four isolated Python processes. The interrupted case is re-run in full
30+
with its module; only completed prior modules count toward aggregate coverage.
31+
No interrupted case is counted as passed. All test stores are private temporary
32+
resources. Generated inputs refreshed after staging new module membership;
33+
verify_repository returned 73 suites/121 checks passed. Source, docs, synthetic
34+
tests, staged scope and sensitive-file risk reviewed before the implementation
35+
commit; no database, raw interaction log, secret or personal configuration staged.
36+
37+
2026-10-14: Accepted local implementation boundary at e2334869. Broad aggregate
38+
coverage completed 636 cases (93 completed serial prefix +543 remaining),635
39+
qualified passes and the unchanged baseline I1 failure. Initial efficiency child
40+
launch without CI's PYTHONPATH=. failed equally on baseline/candidate; both passed
41+
with that declared environment. No additional feature failure remained. Final
42+
contract projection and73-suite/121-check verifier passed. Exact independent
43+
review bindings still match committed sources. CLI GitHub authentication reports
44+
invalid GH_TOKEN (I2); no push/PR or fallback credential/connection attempted.
45+
Local scope is complete, publication externally blocked. Final evidence commit
46+
retains clean source/branch and sanitized reports for parent; bundle outside repo
47+
preserves exact committed ref. Parent owns subsequent publication/merge.
48+
49+
2026-10-14: Parent approved bounded I1 repair for PR #4 after prior service failures.
50+
Verified real checkout read/write/fsync, clean branch/head and remote PR head
51+
4887b7af; no workspace recovery or duplicate push needed. Revisited I1 disposition
52+
under explicit follow-up authority. Fixture now resolves POLICY from its immutable
53+
compiled snapshot, asserts existence and selects semantic_revision and its single
54+
successor before the existing counting mocks. Production validator, normative
55+
content and both count assertions are unchanged. Failing case, analysis module,
56+
full Engine suite and independent review are required before the next commit/push.
57+
58+
2026-10-15: Fixture repair qualified. Owner targeted case passed 1 test in 3.748 s;
59+
independent targeted case passed in 3.676 s. Refreshed the registered test input
60+
through public Engine verify_repository; staged the new review report membership
61+
and refreshed again before freezing inputs. The initial analysis/full attempts
62+
with stale generated input were interrupted and are not acceptance evidence.
63+
Final analysis module passed 23 tests in 113.275 s. One uninterrupted full serial
64+
Engine invocation passed all 636 tests in 2161.675 s under PYTHONPATH=.; the formerly
65+
failing case and all observer-v3 tests passed in that run. Independent review bound
66+
test_analysis.py SHA256 7f51e403f03f5c57be8b1ca5736a79bcf139c87dd9eb4703ce3f7e59f1ba3d01.
67+
No production, validator or normative edits; both call-count assertions unchanged.
68+
Final evidence retains the historical I1 failure and resolves prior I2 transport
69+
confusion. Reviewed bounded staged scope before committing for approved PR #4
70+
branch push; publication identity is the Git ref, merge remains parent-owned.

‎docs/plans/mcp-observability/issues.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Findings
2+
3+
## I1 — baseline proposal-analysis fixture (medium, resolved)
4+
5+
`test_analysis.AnalysisWorkflowTest.test_proposal_analysis_uses_one_projection_and_one_verified_base`
6+
originally hardcoded accepted semantic revision 1/proposed 2. Both the candidate and a clean
7+
Git clone of baseline `188beda1fa477d21d576c233dd8c7f4c4c267d23` return
8+
`AUTHORING.INVALID_SEMANTIC_REVISION` before the test reaches its projection-count
9+
assertions. Independent one-test baseline run:1 test, 1 failure, 1.397 s; candidate:
10+
1 test, 1 failure, 1.606 s. At initial handoff head 4887b7af this source test was unchanged by issue3.
11+
12+
Owner: analysis fixture maintainers. Parent explicitly authorized this bounded
13+
follow-up for PR #4. Disposition: resolved by deriving the accepted semantic
14+
revision and its successor from the same immutable snapshot used for the proposal,
15+
before the counting mocks. Policy existence, real proposal/analysis results and
16+
both count assertions remain required. Validator and normative content unchanged.
17+
The targeted case passed (3.748 s), all 23 analysis tests passed (113.275 s), and
18+
the full serial Engine suite passed all 636 tests (2161.675 s). Independent fixture
19+
review passed; see reports/fixture-review.md. Historical baseline failure evidence
20+
above remains retained rather than relabeled as a baseline pass.
21+
22+
## Resolved review findings
23+
24+
Independent reviewer found and verified resolution of R1–R7: diagnostic field
25+
binding, malformed-envelope coverage, measurement failure, normal retention loss,
26+
report scope/bounds, pre-migration capability checks and residual private SQLite
27+
content. See reports/independent-review.md for source-bound evidence.
28+
29+
Actual model/client traces are outside this task: synthetic process evidence does
30+
not certify model intent or acquisition. No new model qualification is claimed.
31+
32+
## I2 — publication authentication (resolved)
33+
34+
The GitHub CLI reported an invalid configured `GH_TOKEN`, initially treated as a
35+
publication blocker. A later explicitly authorized attempt through the existing
36+
Git remote succeeded without inspecting/copying tokens or changing authentication.
37+
Remote head 4887b7af was verified and the parent opened draft PR #4. The CLI failure
38+
did not establish failure of the separate Git transport. Existing bundle and
39+
checkout were preserved; subsequent publication uses the approved Git branch.

‎docs/plans/mcp-observability/plan.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# Persistent MCP interaction observability (issue 3)
2+
3+
Status: Accepted (local fixture qualification). Acceptance: satisfied. Operation: verify. Owner: implementation agent;
4+
publication and merge authority remain with the parent/operator. Baseline:
5+
`188beda1fa477d21d576c233dd8c7f4c4c267d23` (interface 46).
6+
Canonical plan: `docs/plans/mcp-observability/plan.md`.
7+
8+
## Objective and authority
9+
10+
Implement opt-in, local, agent-invisible SQLite observations at the existing MCP
11+
boundary. Keep domain transactions, tools, outcomes, protocol, retries, generated
12+
contracts, normative standards and observer-v3 qualification unchanged. No model
13+
calls, Tuldok changes, installation configuration or live telemetry collection.
14+
The subsequent user approval permits pushing the verified feature branch and
15+
opening a PR containing only code, synthetic tests and sanitized documentation.
16+
Parent owns merge. No deployment or enabling on existing installations.
17+
18+
Private feature branch `feat/issue3-mcp-observability` in the separate clone
19+
`/workspace/Coding-Standards`; target `main`, integration owner parent. Retain
20+
branch/checkout for publication decision and review; no cleanup or history rewrite.
21+
No writes to Nearwork or other mounted repositories.
22+
23+
## Routed owners and decisions
24+
25+
Core and Router select implementation, verification/oracles, planning/discovery,
26+
proportionality, commit, documentation; library application, IPC and persistence; diagnostics,
27+
architecture, code-design, contracts/evolution/protocols, concurrency, security,
28+
resilience, performance and cross-platform. Python has no language profile.
29+
No normative or generated contract changes: no generated-contract profile,
30+
standards-authoring workflow, release or framework profile is needed.
31+
No AGENTS.md exists in this clone or its workspace ancestors. Runtime identity
32+
uses process UUID and exact catalog/schema/implementation digests, independent
33+
of Git/content authority. Observer v3 remains qualification evidence and retains
34+
its independent exact-source replay. No production dependency on test code.
35+
36+
## Binding design
37+
38+
- MCP owns request execution and response delivery. Recorder owns only sanitized
39+
observations, SQLite transactions, retention and health. Operator CLI reads
40+
that separate database; Engine stores and authorization remain untouched.
41+
- Explicit operator CLI configuration supplies a database in an existing trusted
42+
directory and a deployment label. Omission creates no recorder/file. POSIX local
43+
regular files only; reject symlinks, nonprivate files, unrelated databases and
44+
unsupported schemas. The trusted directory must exclude concurrent hostile
45+
path replacement. Same-user agents can read files: filesystem modes are damage
46+
reduction, never an agent authorization boundary. Store no secrets regardless.
47+
- Schema v1: immutable interaction starts plus independent terminal events, joined
48+
by generated UUID; process/session UUID and ordinal establish local chronology.
49+
Sessions bind runtime metadata, deployment label and per-operation schema
50+
digests. No client-supplied request IDs, names or paths are stored verbatim.
51+
- Pair end/error with begin only after response write/flush in stdio. Direct
52+
dispatch records a constructed response (distinct delivery evidence). Parsing,
53+
validation, typed rejection, tool exceptions and interruption are classified
54+
without raw messages. Unmatched starts remain incomplete/unknown, including
55+
process kill and response delivery failure; never infer completion on restart.
56+
- Bounded input shape: fixed schema-derived property names, scalar type markers,
57+
depth/node/array caps; redact sensitive subtrees and unknown member names.
58+
Never retain scalar values, arbitrary map keys, prompts, bodies, tokens,
59+
credentials, outputs, exception messages or client metadata. Static schema
60+
selector/operation names are admitted only by exact installed-catalog membership.
61+
Feedback retains sanitized field pointers and fixed constraint keywords only.
62+
- No raw input digests (dictionary attacks). Shape digests identify structural
63+
variants only. Catalog and implementation digests identify exact source material.
64+
MCP has no legitimate cross-process session/task continuation in this server;
65+
do not invent one from prompts or domain handles. Reports state that limitation.
66+
- Serial synchronous recorder, no queue/thread/retry. SQLite `BEGIN IMMEDIATE`
67+
owns each mutation and rollback. Finite busy timeout and page quota; rollback
68+
journal rather than unbounded WAL. Maximum retained interactions and age are
69+
operator policies with bounded defaults; prune starts and events atomically.
70+
Retention may evict unmatched observations and is explicitly reported.
71+
- Version 0 is accepted only for a zero-length new database and migrates transactionally
72+
to v1. Existing v1 reopens intact; future/corrupt/unrelated databases fail open
73+
without replacement. No inferred migration or automatic destructive recovery.
74+
- Recorder failure degrades once, disables further recording for that process,
75+
and reports a fixed nonsensitive reason plus counters on operator stderr.
76+
It never retries domain work or changes its return. Operator fixes storage and
77+
restarts to recover recording. Health is independent of the failing database.
78+
- Read-only bounded JSON report: operation/error/shape counts, observed latency
79+
and bytes, incomplete counts, discovery/failed-attempt sequences before a
80+
later success in the same process/operation, catalog comparisons. Denominators
81+
and delivery scope explicit. Correlation cannot prove intent, actual repair,
82+
client acquisition, token usage or billing. Retained traces cannot reconstruct
83+
redacted requests; observer v3 remains the source-backed replay authority.
84+
85+
## Composed-design admission: applicable
86+
87+
1. Independent concerns: MCP owns execution/delivery; recorder owns privacy,
88+
observations and local retention; operator report owns read-only interpretation;
89+
RuntimeIdentity/catalog own source/contract attribution. No observer-v3 coupling.
90+
2. Necessary interleaving: begin before dispatch, terminal after delivery, metadata
91+
captured at recorder construction. Accidental coupling avoided: no Engine SQL,
92+
tool registration, hidden model work or telemetry-driven domain decisions.
93+
3. Composition root knows configuration, immutable runtime/catalog and close;
94+
dispatch/stdio know begin/finish lifecycle. Callers never own SQL or redaction.
95+
4. Privacy changes touch recorder/tests/docs; schema revisions flow through existing
96+
catalog identity; delivery changes touch MCP integration; storage revisions
97+
touch recorder migration and CLI together. Domain changes need no recorder edits.
98+
5. Stable runtime metadata crosses the seam; no store handles or validator internals.
99+
Schema property vocabulary is diagnostic disclosure authority, not validation.
100+
6. Recorder can fail and be disabled independently; CLI reads across restart.
101+
MCP remains independently usable. Tests exercise both enabled and omitted paths.
102+
7. Deleting recorder moves necessary privacy/storage/retention/lifecycle complexity
103+
back into MCP, so retain it. No generic adapter registry, background worker,
104+
second protocol, exported observer tool or migration framework is admitted.
105+
8. Inherent complexity is bounded evidence, privacy and SQLite lifecycle; keep it
106+
in one production module plus its read-only reporting entrypoint, not harnesses.
107+
108+
## Milestones and evidence
109+
110+
A Accepted: design/source/privacy review; write set this plan directory only.
111+
B Accepted: recorder and MCP integration; write set
112+
`tools/standards_engine/standards_engine/{interaction_observer.py,mcp.py}`,
113+
`tools/standards_engine/tests/test_interaction_observer.py`. Gate: focused
114+
privacy, failure and outcome-preservation tests.
115+
C Accepted: operator report, configuration and operational documentation; same
116+
module plus `tools/standards_engine/MCP-OBSERVABILITY.md` and README link.
117+
Gate: bounded read-only report, catalog changes, accurate denominators.
118+
D Accepted: cold stdio/process replacement, exceptions/interruption, real unavailable,
119+
locked/full SQLite, concurrent processes, migration/retention/privacy. Existing MCP,
120+
runtime identity, discovery and observer-v3 tests plus full Engine suite and
121+
contract projection check. Synthetic baseline/candidate call and byte comparison.
122+
E Accepted: independent GPT-6.1 Sol High design/source/privacy review accepted with all seven findings resolved; staged and full branch review, local commits, authorized push and PR.
123+
No real logs or confidential artifacts in Git. Parent owns merge. The generated
124+
`evaluation/standards-effectiveness/generated/suite-inputs.json` is an allowed
125+
adjacent write: refresh through the Engine after staging declared inputs; it
126+
tracks source/README digests and is not a normative standards change.
127+
128+
F Accepted: repair I1 using the actual policy semantic revision from the immutable
129+
snapshot, before entering the existing counting mocks. Write set:
130+
`tools/standards_engine/tests/test_analysis.py`, this plan/ledger/issues, a bounded
131+
fixture-review report and current verification evidence; generated verifier input
132+
manifest only if its compiler requires a refresh. Gate: failing case, analysis
133+
module and full Engine suite pass; independent review; reviewed commit/push to
134+
existing approved PR #4 branch. Validator, normative content and both count
135+
assertions remain unchanged. No telemetry activation or raw log publication.
136+
137+
Current phase: complete local qualification. Next slice: none. Blockers: none.
138+
F evidence: targeted case passed; analysis module 23/23; full serial Engine suite
139+
636/636; independent fixture review satisfied. I1 is resolved without changing
140+
validator, normative content or either count assertion; historical baseline failure
141+
remains in issues/evidence. I2 resolved through the existing Git transport; parent
142+
opened PR #4. The verified follow-up is packaged for the explicitly approved branch
143+
push; Git/PR refs own publication identity and parent owns hosted CI/merge.
144+
See [verification](reports/verification.md), [fixture review](reports/fixture-review.md), [independent source/privacy review](reports/independent-review.md), [ledger](execution-ledger.md) and [issues](issues.md).

‎docs/plans/mcp-observability/reports/baseline-comparison.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Synthetic baseline/candidate comparison
2+
3+
Baseline: `188beda1fa477d21d576c233dd8c7f4c4c267d23`, extracted with `git archive` into an isolated temporary directory. Candidate: reviewed source hashes in [independent review](independent-review.md). Python 3.12.14; SQLite 3.53.1; locked dependency versions matched. No model/client session or real interaction log was collected.
4+
5+
Each cold process received initialize, initialized notification, tools/list, describe_input(runtime_info), an invalid runtime_info request, a valid runtime_info request and ping. Six requests, one notification, three tool calls, six replies ; 688 sent stdio bytes. The operator recorder created only temporary synthetic databases, removed after the comparison.
6+
7+
| Purpose | Output schemas | Baseline bytes | Candidate off | Candidate on | Replies equal |
8+
| --- | --- | ---: | ---: | ---: | --- |
9+
| authoring | eager | 753498 | 753498 | 753498 | yes |
10+
| authoring | on-demand | 115030 | 115030 | 115030 | yes |
11+
| application | eager | 89041 | 89041 | 89041 | yes |
12+
| application | on-demand | 22115 | 22115 | 22115 | yes |
13+
14+
Comparison parsed every reply, including schema catalog, typed rejection, text content and runtime observations. Only process UUID and implementation digest were normalized: these must differ for distinct loaded sources/processes. Full JSON response byte totals matched without normalization. No tools, protocol messages, operation retries or domain results were added.
15+
16+
One sample per configuration is outcome/byte preservation evidence, not a latency benchmark. Observed cold process times ranged from 852.56 ms to 1193.81 ms with other tests active. No throughput/overhead, token or billing claim is made.
17+
18+
Reproduction: extract the named baseline with git archive; start the baseline MCP and candidate MCP with --purpose authoring/application and --output-schemas eager/on-demand. Send the seven listed JSON-RPC messages, compare parsed replies after replacing instance_id/implementation_digest (including JSON text blocks), and count UTF-8 bytes of the actual stdio streams. Repeat the candidate with --observe-database pointing at a fresh private temporary path and --observe-deployment synthetic-comparison.

0 commit comments

Comments
 (0)