Per-span call graph construction via eBPF stack frame correlation

[chandanjonson]

Background
OBI currently instruments at the protocol level and generates span-level telemetry (HTTP, gRPC, DB, etc.) with very low overhead. The opentelemetry-ebpf-profiler separately captures continuous stack traces system-wide. However, there is currently no mechanism to correlate profiling stack frames with active OTel spans — meaning we know that a span happened, but not what code ran inside it.

Problem
Given an active span with a known traceId and spanId, there is no way today to answer:

"Which functions were called during the execution of this span?"

This gap means developers cannot get per-span flame graphs or call graphs without manual SDK instrumentation.

Proposal
Introduce a correlation layer that:

Reads active span context from the OBI instrumentation layer (trace ID, span ID, start/end timestamps, thread/goroutine ID)
Correlates eBPF stack frame snapshots captured during the span's lifetime using the thread/goroutine ID as the join key
Constructs a per-span call graph by aggregating the flat stack traces captured between span start and end

This would produce a new signal: per-span call graphs, which can be exported as OTel profile data attached to a span.
Why here
OBI already has span lifecycle awareness and thread-level context. It is the natural place to expose the span context boundary needed for correlation. The profiler provides the raw stack frames. The correlation logic sits between them.
Open questions

Should this live in OBI, in ebpf-profiler, or as a separate integration component?
What is the right export format — OTel profiles signal, span events, or span attributes?
What are the overhead implications of matching stack frames to span boundaries at runtime?

Prior work
I have been prototyping this correlation approach in a personal project (TeleDashboard) and would like to bring the design to the community before writing upstream-targeted code.
Happy to present this at a SIG community call if there is interest.

[grcevski]

Please see #1684. @mmat11 has been working with the OTel Profiler SIG on implementing a standard approach for trace to profile correlation.

[chandanjonson]

Can i contribute there and @mmat11 could you please share me design doc how we are approaching span correlation?

[grcevski]

If you look at the PR I linked, it will point you to the design document and how it's implemented. The traces_ctx_v1 contains the current OBI trace context and it's exposed through the BPF file system. The remaining work is to add the support in the OTel profiler.

[mmat11]

@chandanjonson you can find a proof of concept for the profiler here open-telemetry/opentelemetry-ebpf-profiler#1139. OBI already exports trace contexts in the shared communication channel