Default opentelemetry_exclude_spans to ["receive", "send"] and support it on FastMCP (#364)

Author: lesnik512

benchmarks/README.md

@@ -101,21 +101,35 @@ Sentry +57.1, OpenTelemetry +122.4 and all four +228.2. The table holds.
 
 | scenario | RPS | µs/req | gain |
 |---|---:|---:|---|
-| `otel` as lite-bootstrap configures it | 7267 | 137.6 | - |
+| `otel` as lite-bootstrap 1.x configured it | 7267 | 137.6 | - |
 | `+ opentelemetry_exclude_spans=["receive", "send"]` | 9651 | 103.6 | −34.0 µs |
 | `+ opentelemetry_sampler=ParentBased(TraceIdRatioBased(0.01))` | 12154 | 82.3 | −55.3 µs |
 | both | 15518 | 64.4 | **2.14x** |
 
 1. `FastAPIInstrumentor.instrument_app` accepts `exclude_spans: list[Literal["receive","send"]]`,
-   which `FastAPIConfig.opentelemetry_exclude_spans` passes through. It is empty by default, so
-   **every request still produces three spans** - the server span plus one each for the ASGI
-   `receive` and `send` events. Two thirds of the spans, one quarter of the cost, and almost
-   nobody looks at them. Set it to `["receive", "send"]` to drop the two event spans.
+   which `FastAPIConfig.opentelemetry_exclude_spans` passes through. In 1.x it was empty by
+   default, so every request produced three spans: the server span plus one each for the ASGI
+   `receive` and `send` events. Those two were two thirds of the spans and a quarter of the cost.
+   Since 2.0 it defaults to `["receive", "send"]` on FastAPI, Litestar and FastMCP.
 2. `OpenTelemetryConfig.opentelemetry_sampler` is passed to the `TracerProvider`. Left unset, the
    SDK default `parentbased_always_on` applies and every request is recorded, serialized and
    shipped. A 1% ratio sampler is worth 55 µs/req here. The default stays always-on deliberately:
    sampling rate is a user decision, not something to pick on a service's behalf.
 
+Rerun for 2.0 in-process on the M2 host (macOS, CPython 3.14.7, opentelemetry-sdk 1.45.1,
+opentelemetry-instrumentation-asgi 0.66b1, fastapi 0.162.0, litestar 2.25.0, fastmcp 4.1.4),
+100% sampling, with `bench.py <suite> bare,otel,otel_all_spans --repeat 3` (FastMCP with
+`--ops 1500 --warmup 300`). `otel` is the 2.0 default and `otel_all_spans` sets
+`opentelemetry_exclude_spans=[]`, the 1.x default:
+
+| suite | `otel` | `otel_all_spans` | saved |
+|---|---:|---:|---:|
+| FastAPI, per request | +91.8 | +127.8 | 36.0 |
+| Litestar, per request | +79.3 | +116.1 | 36.8 |
+| FastMCP, per `tools/call` | +175.4 | +273.8 | 98.4 |
+
+FastMCP's rounds differed by up to 9%, FastAPI's `otel` row by 17%.
+
 ### 4b. Sentry: the cost is one thing, and it is not the one people tune
 
 In-process ablation, Sentry only, baseline 16.5 µs/req:
@@ -233,7 +247,8 @@ Since 2.0, lite-bootstrap reports FastAPI request errors itself while Sentry tra
 | `sentry_integration_sampled_trace` | +59.2 | +113.2 |
 
 The last row on 3.0.0b2 varied by 40% between rounds. The table above was measured before 2.0. Its
-`sentry_nosess` row now has the same config as `sentry`, and `sentry_noauto` no longer stops request
+OpenTelemetry rows record the `receive` and `send` spans, which 2.0 leaves out by default (section
+4a). Its `sentry_nosess` row now has the same config as `sentry`, and `sentry_noauto` no longer stops request
 capture, because lean capture needs no auto-enabling integration.
 
 FastAPI real-server runs under uvicorn in the container were unstable (bare repeat medians of 8235
@@ -281,13 +296,13 @@ FastAPIConfig(
     ],
     sentry_logging_breadcrumb_level=None,
     # OpenTelemetry: sampling is the single biggest saving here
-    opentelemetry_exclude_spans=["receive", "send"],
     opentelemetry_sampler=ParentBased(TraceIdRatioBased(0.01)),
 )
 ```
 <!-- --8<-- [end:tuned] -->
 
-The tuned runs also set `sentry_auto_session_tracking=False`, which is the default since 2.0.
+The tuned runs also set `sentry_auto_session_tracking=False` and
+`opentelemetry_exclude_spans=["receive", "send"]`, which are the defaults since 2.0.
 
 Trade-offs, in order of what you give up: log breadcrumbs on Sentry errors, Sentry release health,
 Sentry-side trace correlation, 99% of OTel traces, ASGI event spans.
@@ -418,7 +433,8 @@ On 3.0 an incoming sampled `sentry-trace` makes `LitestarIntegration` record a r
 tracing off. `sentry_nointegration`, which disables `LitestarIntegration` and turns lean capture off,
 so nothing captures request errors, measured +0.3 and +0.0 in a later run that put `sentry` at
 +6.7 and +6.8. The table above was measured before 2.0. Its `sentry_nosess` scenario now has the same
-config as `sentry`, because sessions are off by default since 2.0.
+config as `sentry`, because sessions are off by default since 2.0. Its OpenTelemetry rows record the
+`receive` and `send` spans, which 2.0 leaves out by default (section 4a).
 
 lite-bootstrap's Litestar defaults with no endpoint configured (Prometheus, health, logging and
 Swagger on) cost +11.7 µs on `/ping`, all of it Prometheus. The access-log middleware writes each record
@@ -555,6 +571,9 @@ over a baseline of about 620 µs per tool call:
 
 Rounds differed by up to 15%, so the lean rows are not resolved from each other.
 
+The OpenTelemetry rows in the first table record the ASGI `receive` and `send` spans, which 2.0
+leaves out by default: section 4a has the rerun.
+
 ### 8f. Logging, per call
 
 The per-call chain is the same on every framework: structlog's stdlib `BoundLogger`, lite-bootstrap's

benchmarks/frameworks/common.py

@@ -163,6 +163,7 @@ def sentry(**extra_params: object) -> dict[str, typing.Any]:
 OTEL: typing.Final = {"opentelemetry_endpoint": OTLP_TRACES, "opentelemetry_exporter_protocol": "http"}
 OTEL_METRICS: typing.Final = {"opentelemetry_metrics_endpoint": OTLP_METRICS}
 EXCL: typing.Final = {"opentelemetry_exclude_spans": ["receive", "send"]}
+ALL_SPANS: typing.Final = {"opentelemetry_exclude_spans": []}
 PROM: typing.Final = {"prometheus_metrics_path": "/metrics"}
 LOG: typing.Final = {"logging_enabled": True}
 CORS: typing.Final = {"cors_allowed_origins": ["https://example.com"], "cors_allowed_methods": ["GET"]}

benchmarks/frameworks/fastapi_suite.py

@@ -3,6 +3,7 @@
 import typing
 
 from common import (
+    ALL_SPANS,
     CORS,
     EXCL,
     LOG,
@@ -37,6 +38,7 @@
     "log_mw": Scenario(config=base_config(**LOG, logging_access_log_enabled=True)),
     "prom": Scenario(config=base_config(**PROM)),
     "otel": Scenario(config=base_config(**OTEL)),
+    "otel_all_spans": Scenario(config=base_config(**OTEL, **ALL_SPANS)),
     "otel_excl": Scenario(config=base_config(**OTEL, **EXCL)),
     "otel_10": Scenario(config=base_config(**OTEL, **ratio(0.1))),
     "otel_10_excl": Scenario(config=base_config(**OTEL, **ratio(0.1), **EXCL)),

benchmarks/frameworks/fastmcp_suite.py

@@ -9,6 +9,7 @@
 
 import fastmcp.server.telemetry
 from common import (
+    ALL_SPANS,
     LOG,
     OTEL,
     PROM,
@@ -46,6 +47,7 @@ def no_fastmcp_spans() -> None:
     "log_mw": Scenario(config=base_config(**LOG, logging_access_log_enabled=True)),
     "prom": Scenario(config=base_config(**PROM)),
     "otel": Scenario(config=base_config(**OTEL)),
+    "otel_all_spans": Scenario(config=base_config(**OTEL, **ALL_SPANS)),
     "otel_nofastmcp_spans": Scenario(config=base_config(**OTEL), patch=no_fastmcp_spans),
     "otel_10": Scenario(config=base_config(**OTEL, **ratio(0.1))),
     "sentry": Scenario(config=base_config(**sentry())),

benchmarks/frameworks/litestar_suite.py

@@ -9,6 +9,7 @@
 import litestar
 import structlog
 from common import (
+    ALL_SPANS,
     CORS,
     EXCL,
     LOG,
@@ -50,6 +51,7 @@
     "prom": Scenario(config=base_config(**PROM)),
     "prom_nogroup": Scenario(config=base_config(**PROM, prometheus_group_path=False)),
     "otel": Scenario(config=base_config(**OTEL)),
+    "otel_all_spans": Scenario(config=base_config(**OTEL, **ALL_SPANS)),
     "otel_excl": Scenario(config=base_config(**OTEL, **EXCL)),
     "otel_10": Scenario(config=base_config(**OTEL, **ratio(0.1))),
     "otel_10_excl": Scenario(config=base_config(**OTEL, **ratio(0.1), **EXCL)),

docs/introduction/configuration.md

@@ -233,12 +233,13 @@ config = FastAPIConfig(
 )
 ```
 
-For FastAPI and Litestar there is additionally:
+For FastAPI, Litestar and FastMCP there is additionally:
 
-- `opentelemetry_exclude_spans` - drops the ASGI `receive` and/or `send` spans, leaving only the
-  server span. Empty by default, which records all three. `["receive", "send"]` is worth ~34 µs per
-  request on the benchmark endpoint, at the cost of two thirds of the spans disappearing from your
-  trace view.
+- `opentelemetry_exclude_spans` - the ASGI events that get no span of their own. Defaults to
+  `["receive", "send"]` since 2.0, so each request records only its server span. At 100% sampling
+  the two event spans cost about 36 to 66 µs per request on FastAPI and Litestar, and about 98 µs
+  per FastMCP tool call. Set it to `[]` to record them again, or to `["receive"]` or `["send"]` to
+  keep one kind.
 
 For FastStream there is additionally:
 

docs/introduction/performance.md

@@ -52,7 +52,7 @@ In rough order of what they return:
 |---|---|---|
 | [`opentelemetry_sampler`](configuration.md#opentelemetry), 1% ratio | ~55 µs/req | 99% of your traces |
 | `http_methods_to_capture=()` on the Sentry ASGI integrations | ~35 µs/req | Sentry-side trace correlation |
-| [`opentelemetry_exclude_spans`](configuration.md#opentelemetry), FastAPI and Litestar | ~34 µs/req | the two ASGI event spans |
+| [`opentelemetry_exclude_spans=["receive", "send"]`](configuration.md#opentelemetry), the default since 2.0 | ~34 µs/req | the two ASGI event spans |
 | [`sentry_auto_session_tracking=False`](configuration.md#sentry), the default since 2.0 | ~7.7 µs/req | Sentry release health |
 | [`sentry_logging_breadcrumb_level=None`](configuration.md#sentry-logging-integration) | ~7 µs per log record | log breadcrumbs on errors |
 
@@ -68,7 +68,7 @@ request, every request then pays the per-record cost measured here:
 
 --8<-- "benchmarks/README.md:logging"
 
-lite-bootstrap already applies five savings for you. It turns Sentry Logs off in the logging
+lite-bootstrap already applies six savings for you. It turns Sentry Logs off in the logging
 integration, because it never enables them and on 2.x the handler formats every record before
 checking whether they are enabled; that one costs nothing. Since 2.0 it also leaves release-health
 sessions off and starts breadcrumbs at `WARNING`, so an `INFO` record skips the breadcrumb handler.
@@ -78,8 +78,10 @@ back. And while Sentry tracing is off it reports request errors without sentry-s
 integration, which costs you request transactions and bodies the handler never read; set `sentry_lean_error_capture=False` to keep the integration. With tracing off it also turns
 sentry-sdk's [auto-enabling integrations](configuration.md#sentry-auto-enabling-integrations) off,
 which costs you their SQL, Redis and outgoing-HTTP breadcrumbs; set
-`sentry_auto_enabling_integrations=True` to keep them. The numbers on this page were measured
-before 2.0, against the old defaults.
+`sentry_auto_enabling_integrations=True` to keep them. On FastAPI, Litestar and FastMCP it also
+records no span for the ASGI `receive` and `send` events, which costs you those spans; set
+`opentelemetry_exclude_spans=[]` to get them back. The numbers on this page were measured before
+2.0, against the old defaults.
 
 ## What each saving costs you
 
@@ -148,7 +150,8 @@ about +21 µs per tool call ([FastMCP: Sentry](../integrations/fastmcp.md#sentry
 --8<-- "benchmarks/README.md:otelfloor"
 
 If you sample down to save CPU, this floor is what you keep paying. Excluding the `receive` and `send`
-spans lowers it: Litestar's 1% row with them excluded sits below its 0% row with them kept.
+spans lowers it: Litestar's 1% row with them excluded sits below its 0% row with them kept. Since 2.0
+they are excluded by default.
 
 ## Reproducing on macOS
 

lite_bootstrap/bootstrappers/fastapi_bootstrapper.py

@@ -16,7 +16,7 @@
     HealthCheckTypedDict,
 )
 from lite_bootstrap.instruments.logging_instrument import LoggingConfig, LoggingInstrument
-from lite_bootstrap.instruments.opentelemetry_instrument import OpenTelemetryConfig, OpenTelemetryInstrument
+from lite_bootstrap.instruments.opentelemetry_instrument import OpenTelemetryAsgiConfig, OpenTelemetryInstrument
 from lite_bootstrap.instruments.prometheus_instrument import PrometheusConfig, PrometheusInstrument
 from lite_bootstrap.instruments.pyroscope_instrument import PyroscopeConfig, PyroscopeInstrument
 from lite_bootstrap.instruments.sentry_instrument import (
@@ -64,16 +64,14 @@ class FastAPIConfig(
     CorsConfig,
     HealthChecksConfig,
     LoggingConfig,
-    OpenTelemetryConfig,
+    OpenTelemetryAsgiConfig,
     PrometheusConfig,
     PyroscopeConfig,
     SentryLeanCaptureConfig,
     SwaggerConfig,
 ):
     application: "fastapi.FastAPI | UnsetType" = UNSET
     application_kwargs: dict[str, typing.Any] = dataclasses.field(default_factory=dict)
-    # Not on OpenTelemetryConfig: only the FastAPI and Litestar instrumentors take `exclude_spans`.
-    opentelemetry_exclude_spans: list[typing.Literal["receive", "send"]] = dataclasses.field(default_factory=list)
     prometheus_instrumentator_additional_params: dict[str, typing.Any] = dataclasses.field(default_factory=dict)
     prometheus_instrument_additional_params: dict[str, typing.Any] = dataclasses.field(default_factory=dict)
     prometheus_expose_additional_params: dict[str, typing.Any] = dataclasses.field(default_factory=dict)

lite_bootstrap/bootstrappers/fastmcp_bootstrapper.py

@@ -11,7 +11,7 @@
 from lite_bootstrap.bootstrappers.base import BaseBootstrapper
 from lite_bootstrap.instruments.healthchecks_instrument import HealthChecksConfig, HealthChecksInstrument
 from lite_bootstrap.instruments.logging_instrument import LoggingConfig, LoggingInstrument
-from lite_bootstrap.instruments.opentelemetry_instrument import OpenTelemetryConfig, OpenTelemetryInstrument
+from lite_bootstrap.instruments.opentelemetry_instrument import OpenTelemetryAsgiConfig, OpenTelemetryInstrument
 from lite_bootstrap.instruments.prometheus_instrument import PrometheusConfig, PrometheusInstrument
 from lite_bootstrap.instruments.pyroscope_instrument import PyroscopeConfig, PyroscopeInstrument
 from lite_bootstrap.instruments.sentry_instrument import (
@@ -201,7 +201,12 @@ async def on_message(
 
 @dataclasses.dataclass(kw_only=True, slots=True, frozen=True)
 class FastMcpConfig(
-    HealthChecksConfig, LoggingConfig, OpenTelemetryConfig, PrometheusConfig, PyroscopeConfig, SentryLeanCaptureConfig
+    HealthChecksConfig,
+    LoggingConfig,
+    OpenTelemetryAsgiConfig,
+    PrometheusConfig,
+    PyroscopeConfig,
+    SentryLeanCaptureConfig,
 ):
     application: "FastMCP[typing.Any]" = dataclasses.field(default_factory=_make_fastmcp)
     logging_access_log_enabled: bool = False
@@ -264,6 +269,7 @@ def _instrument_http_app(self, http_application: "StarletteWithLifespan") -> "St
             ),
             # OpenTelemetryMiddleware only parses a raw string from 0.56b0; the floor is 0.49b0.
             excluded_urls=parse_excluded_urls(",".join(self._build_excluded_url_patterns("STARLETTE"))),
+            exclude_spans=self.bootstrap_config.opentelemetry_exclude_spans,
             tracer_provider=get_tracer_provider(),
             meter_provider=get_meter_provider(),
         )

lite_bootstrap/bootstrappers/litestar_bootstrapper.py

@@ -15,7 +15,7 @@
     HealthCheckTypedDict,
 )
 from lite_bootstrap.instruments.logging_instrument import LoggingConfig, LoggingInstrument
-from lite_bootstrap.instruments.opentelemetry_instrument import OpenTelemetryConfig, OpenTelemetryInstrument
+from lite_bootstrap.instruments.opentelemetry_instrument import OpenTelemetryAsgiConfig, OpenTelemetryInstrument
 from lite_bootstrap.instruments.prometheus_instrument import (
     PrometheusConfig as PrometheusBootstrapperConfig,
 )
@@ -254,7 +254,7 @@ class LitestarConfig(
     CorsConfig,
     HealthChecksConfig,
     LoggingConfig,
-    OpenTelemetryConfig,
+    OpenTelemetryAsgiConfig,
     PrometheusBootstrapperConfig,
     PyroscopeConfig,
     SentryLeanCaptureConfig,
@@ -263,7 +263,6 @@ class LitestarConfig(
     application_config: "AppConfig" = dataclasses.field(default_factory=lambda: AppConfig())  # noqa: PLW0108
     litestar_logging_middleware_config: "LoggingMiddlewareConfig | None" = None
     logging_access_log_enabled: bool = False
-    opentelemetry_exclude_spans: list[typing.Literal["receive", "send"]] = dataclasses.field(default_factory=list)
     prometheus_additional_params: dict[str, typing.Any] = dataclasses.field(default_factory=dict)
     # Bounds path-label cardinality (Litestar defaults False -> raw URLs leak memory). See litestar#4891.
     prometheus_group_path: bool = True