Skip to content

Default opentelemetry_generate_health_check_spans to False - #366

Merged
lesnik512 merged 1 commit into
mainfrom
health-check-spans-off
Oct 29, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
health-check-spans-off

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Closes #311.

opentelemetry_generate_health_check_spans now defaults to False, so a Kubernetes probe sends no trace. The access log has always left the health path out, so tracing and logging now agree. The change is the default at opentelemetry_instrument.py:86. _build_infrastructure_excluded_paths already adds the health path while the flag is off, so FastAPI, Litestar and FastMCP need nothing else. FastStream's health handler skips its own span wrapper on the same flag.

Tests

Written first. These failed on main:

  • test_faststream_health_check_records_no_span_by_default: one health request with the default config produced a GET /custom-health/ span.
  • test_fastapi_otel_leaves_the_health_check_untraced_by_default: a default FastAPIConfig traced GET /custom-health/. This covers the issue's check, a default FastAPI app with opentelemetry_log_traces=True printing no span for the health path. That test uses its own config because the shared fixture sets the flag explicitly.

These kept the old default and now set opentelemetry_generate_health_check_spans=True explicitly:

test_faststream_health_check_span_is_recorded_when_turned_on is new, and it passes on main too. No FastStream test had checked that the span is actually recorded. Its helper patches the module's tracer with a fresh proxy tracer. The module-level trace.get_tracer(__name__) binds to the first real provider it sees, and the suite installs a new provider for every test. In production there is one provider per process, so this only affects tests.

The full suite passes, 577 tests at 100% coverage (just test-ci). just lint (including ty), just adr-check and just docs-build pass too. No floor changes.

Not in the issue

While the flag is off, OpenTelemetry's HTTP metrics also skip health-check requests, because both instrumentations skip an excluded URL completely. I checked this on FastAPI and Litestar with an in-memory metric reader: with the flag off, a health request records no http.server.* metric, and with it on, it records one. Since #365, a service with opentelemetry_metrics_endpoint set has no Prometheus HTTP metrics either, so its probe requests are now counted nowhere. The configuration.md bullet says so. Probes usually stay out of request metrics anyway, so I kept the issue's behaviour and didn't add a separate switch.

Measurement

In-process on the M2 host (macOS, CPython 3.14.7, fastapi 0.162.0, litestar 2.25.0, opentelemetry-sdk 1.45.1), at 100% sampling. Figures are µs per /health/ request over the health check without OpenTelemetry:

scenario FastAPI Litestar
health_otel: the 2.0 default +5.1 +1.9
health_otel_spans: the 1.x default +99.2 +82.4

Litestar's rounds differed by up to 24%. As the issue says, the gain is trace volume, one trace per probe per pod, not CPU. health_otel_spans is a new scenario in the FastAPI and Litestar suites. health_otel_nospan now has the same config as health_otel, and I kept it so the existing README command still runs.

Docs

  • configuration.md: the flag's bullet gives the new default, says it now matches the access log, explains the metrics side effect and says how to turn spans back on. The FastStream paragraph says the span is recorded when the flag is True.
  • fastmcp.md: "unless ... is True".
  • performance.md: the "savings applied for you" paragraph now has eight savings.
  • benchmarks/README.md: the two health rows are relabelled "1.x default", §4d gets the rerun, and §9 gets the FastAPI command.

Breaking changes

Health-check requests are no longer traced, and while the flag is off they no longer appear in OpenTelemetry's HTTP metrics. Set opentelemetry_generate_health_check_spans=True to keep both, for example to watch probe latency.

@lesnik512
lesnik512 merged commit b0e6fba into main Oct 29, 2026
37 checks passed
@lesnik512
lesnik512 deleted the health-check-spans-off branch October 29, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default opentelemetry_generate_health_check_spans to False

1 participant