[Nayana-R-Gowda]

Signed-off-by: NAYANA.R nayana.r7813@gmail.com

Issue: MCP tool invocations (e.g. mcp_903e1051_dw-mk-ask-question) fail with InstanaTracer.start_as_current_span() got an unexpected keyword argument 'context' when ContextForge runs under Instana AutoTrace with instana < 3.12.
Root cause: ContextForge's OpenTelemetryRequestMiddleware.call (mcpgateway/observability.py) passes context= to _TRACER.start_as_current_span(...) whenever an inbound request carries a traceparent header. Older Instana tracers (< 3.12) predate OTel's context kwarg and raise TypeError, aborting the request — even though tracing itself is nonessential.
Fix:

• Wrapped the span start in a try/except TypeError that retries without the context kwarg and logs a debug message, so tracer incompatibilities never break inbound requests.
• Added regression test test_middleware_tolerates_tracer_without_context_kwarg with a fake context-rejecting tracer.
• All test_observability.py tests pass; ruff-clean.
  Files changed (fix/instana-tracer-context-kwarg):
• mcpgateway/observability.py:980
• tests/unit/mcpgateway/test_observability.py

Closes #7208

[madhav165]

Thanks for addressing the Instana compatibility issue. The original change was well targeted: it identified the inbound request-root span as the failure point, preserved request execution for older Instana tracers, and included a focused regression test.

What was good

• The fix is localized to the request-root tracing middleware.
• It preserves support for normal OpenTelemetry tracers.
• The regression test reproduces the reported Instana behavior.
• I traced the middleware through MCP, RPC, virtual-server, internal MCP, and A2A invocation paths, as well as downstream context propagation.

What I fixed

The broad except TypeError also retried unrelated tracer failures, including cases where no context argument had been supplied. This could hide the original error and invoke a stateful tracer twice.

I pushed commit 5d7879731, which limits the fallback to the known compatibility case:

• context must have been supplied;
• the error must report an unexpected keyword argument;
• the rejected argument must specifically be context.

All other TypeError exceptions now propagate unchanged.

Tests added

• An unrelated TypeError is not retried when no context was extracted.
• An internal/unrelated tracer TypeError is not swallowed when context was supplied.

The focused tests and the complete test_observability.py suite pass. Ruff passes for the modified source, and git diff --check is clean.

With those guardrails in place, the compatibility approach looks sound.

[msureshkumar88]

The fix stays in the correct backend layer. The narrow retry filter preserves unrelated exceptions. No schema or Alembic change is necessary.

blocking | mcpgateway/observability.py:992 | Removing context loses the extracted parent and can split active and forwarded traces | Attach the extracted context during the fallback span lifetime and detach it in finally.

With a valid inbound `traceparent`, `has_remote_parent` remains true. The retry removes `context` without attaching the extracted parent. A tracer that uses the current context can therefore start a separate root. The header-publication guard retains the original inbound header, so active spans and raw-header forwarding can use different traces. I reproduced successful request processing with different active and forwarded trace IDs using the real OpenTelemetry API and a context-rejecting tracer adapter. Actual legacy Instana AutoTrace remains untested. Preserve parenting through the standard context API and verify compatibility with the legacy provider and propagator.

functionally-impacting | tests/unit/mcpgateway/test_observability.py:836 | The success test cannot detect lost parenting, and the PR adds no live-gateway regression | Add a parent-preservation test and the required black-box MCP test, or document a live-test waiver.

The success test returns `nullcontext(None)`, supplies no inbound `traceparent`, and forces an invalid remote parent. Add a valid remote parent and a context-rejecting tracer that creates a span. Assert parent identity, active and forwarded trace consistency, response completion, and context restoration after success and app failure. Exercise authenticated MCP `tools/call` against a running gateway with the incompatible provider. Repository instructions require this black-box test for behavior that a live gateway can exercise.

Validation: The focused module has 55 passing tests and four SDK-dependent skips. All added executable production lines ran. git diff --check passes. Production Ruff checks pass; seven unused test imports already exist in the base version.

Please fix the trace-parent defect before merge. No UI or database changes are required.

[Nayana-R-Gowda]

mcpgateway/observability.py

1. Imports opentelemetry.context as otel_context (with otel_context = None fallback when OTEL is absent).
2. In OpenTelemetryRequestMiddleware, when the tracer rejects the context kwarg (Instana < 3.12 fallback path):

• Attaches the extracted inbound parent_context via otel_context.attach(parent_context) before retrying the span (:998-1003).
• Detaches it in a finally block after the span lifetime (:1060-1065).

3. Refactored the span body into try/with/finally so detach always runs on success and on app exception.

tests/unit/mcpgateway/test_observability.py (+4 tests, ~190 lines)

• test_fallback_preserves_inbound_parent_with_context_rejecting_tracer — valid inbound traceparent + context-rejecting tracer; asserts parent span id, active trace == forwarded trace, response completion, context restored.
• test_fallback_restores_context_when_app_raises_with_context_rejecting_tracer — same, with the app raising; asserts detach still runs and span status is ERROR.
• test_middleware_does_not_retry_type_error_without_context — unrelated TypeError propagates, tracer called once.
• test_middleware_does_not_swallow_unrelated_type_error_with_context — same when context was supplied.

[msureshkumar88]

The update preserves the inbound parent during successful fallback spans and restores context after application failures. The narrow TypeError filter remains intact.

blocking | mcpgateway/observability.py:1005 | A failure in the fallback tracer call leaves the inbound context attached because cleanup starts at line 1012 | Move span creation and retry inside the cleanup try/finally; add a retry-failure regression test.

I reproduced this with the real OpenTelemetry context API and W3C propagator. A tracer rejects `context`, then raises `RuntimeError` during retry. The exception propagates, but the outer context remains replaced by the inbound trace. Later spans in that task can inherit the failed request's parent.

functionally-impacting | tests/unit/mcpgateway/test_observability.py:933 | The requested live-gateway regression remains absent, and the PR documents no waiver | Add an authenticated MCP tools/call test against a running gateway with the incompatible provider, or document the requested waiver.

The PR changes only the middleware and its unit tests. The parent-preservation unit tests address the earlier unit-test request. They do not exercise authenticated MCP tools/call against a running gateway.

Validation: The focused unit module has 58 passing tests and six SDK-dependent skips. Production Ruff and fix-commit whitespace checks pass. GitHub CI passes, including pytest with the observability extra. An independent retry-failure reproduction confirms the context leak.

Not checked: actual legacy Instana AutoTrace, a live MCP request, or the full local validation gate.

Please resolve both findings before merge.

[Nayana-R-Gowda]

functionally-impacting — missing live-gateway regression — Fixed in 69d4ec4.
Added tests/live_gateway/mcp/test_mcp_trace_context_e2e.py: an authenticated MCP tools/call against a running gateway subprocess whose tracer emulates instana < 3.12 (rejects context). It asserts request completion, request-root span continuation of the inbound trace, parent identity, and active/forwarded trace consistency. Wired into the test-observability CI job at .github/workflows/pytest.yml:228.
The narrow TypeError filter is unchanged: only a supplied context keyword reported as an unexpected keyword argument activates the fallback; unrelated TypeErrors propagate.
Verification: focused module passes; the live MCP test passes locally; git diff --check clean. The remaining local failure (test_observability_disabled_without_otlp_endpoint) is a pre-existing .env artifact (endpoint set locally, unset in CI). The seven unused test imports pre-exist in the base.

[madhav165]

Review follow-up: compatibility consolidated and live-tested

Thanks for the additional fixes and validation work. After tracing the failure through every caller, the important finding was that this was not limited to ContextForge's request middleware.

FastAPI telemetry, MCP SDK 2.x telemetry, and ContextForge all call the same legacy Instana tracer using the standard OpenTelemetry context= argument. Instana before 3.12 instead expects its older span_context= representation.

What changed

I pushed d693f935f, which consolidates compatibility at the shared tracer boundary:

• adapts legacy InstanaTracer.start_span();
• adapts legacy InstanaTracer.start_as_current_span();
• converts an OTel parent context into the Instana SpanContext expected by Instana 3.11;
• patches the tracer class once, covering instances already cached by FastAPI and MCP SDK;
• leaves modern Instana and ordinary OTel tracers unchanged;
• removes the duplicate Instana-specific retry from OpenTelemetryRequestMiddleware.

This also removes the fallback lifecycle that caused the context-leak finding when retry span creation failed.

Test consolidation

The overlapping fallback and synthetic-tracer tests were reduced to focused adapter coverage. The synthetic live test was replaced with a self-contained test using the real instana==3.11.1 AutoTrace provider.

The live regression verifies:

• gateway startup under legacy Instana;
• GET /health returns 200;
• authenticated streamable-HTTP MCP tools/call succeeds;
• unauthenticated MCP access remains denied;
• no context-keyword TypeError escapes;
• both legacy tracer methods are adapted.

CI now runs this test explicitly with LEGACY_INSTANA_E2E=1, so unavailable prerequisites fail rather than silently skip.

Validation

• Complete observability unit module: 69 passed
• Real Instana 3.11.1 live-gateway regression: 4 passed
• Ruff on changed production/live-test files: passed
• Python compilation: passed
• git diff --check: passed

This now covers the actual process-wide failure while keeping a single compatibility mechanism.