[feature request] Log sampling that reduces volume without breaking log-based counts

[Yun-Ting]

Component

OpenTelemetry.Sampler.BottomFloor

Is your feature request related to a problem?

Services that emit high-volume, repetitive logs have three options in .NET
today, and all of them lose information:

• Raise the minimum log level - the rare error inside a flood of routine
  records is discarded along with the flood.
• Fixed-rate head sampling - rare records are dropped at the same rate as
  common ones, and nothing downstream can tell how much was dropped.
• Per-logger rate limiting - whatever is loudest is truncated arbitrarily.

In every case the exported data can no longer answer "how many times did this
actually happen?", so dashboards and alerts built on log counts silently break
once volume controls are switched on.

What is the expected behavior?

A new component, OpenTelemetry.Sampler.BottomFloor, that bounds how many log
records leave the process while keeping counts recoverable.

The idea is bottom-k / priority sampling with Horvitz-Thompson correction. Each
export batch is a sampling window with a single user setting, a budget k.
Frequent callsites are progressively down-weighted and rare ones kept, and every
retained record is stamped with otel.logs.adjusted_count - the reciprocal of
its inclusion probability. Downstream, COUNT(*) becomes
SUM(otel.logs.adjusted_count) and remains an unbiased estimate of the true
pre-sampling count. Memory is bounded by construction, since the weight table is
rebuilt each window from the retained sample and never exceeds k entries.

I would rather agree on the shape before building more of it, so the work is
split into increments. This issue tracks them.

Increment 1 - whole-stream log sampling (#5108, in review)

• The sampling algorithm, kept internal for now
• A log exporter decorator and a ready-to-register processor
• Options, docs and a runnable example

Increment 2 - per-span log coverage

Also retain a bounded sample of records per recording span, so a trace under
investigation keeps its own logs even when the whole-stream budget would have
dropped them. Grouping can use the span id already on the record, so it needs no
cooperation from the application.

• Opt-in per-span retention, off by default
• A separate adjusted count so span-only records do not skew the
  whole-stream estimate
• Agree how this interacts with the budget (question 3)

Increment 3 - expose the sampling algorithm

• Make the underlying stream sampler public for non-log use, if wanted

Which alternative solutions or features have you considered?

• Filing on opentelemetry-specification first. The count-correction
  convention arguably belongs there, which is why attribute naming is called out
  below. The component itself is .NET-specific in that it hooks the .NET log
  pipeline. Happy to take the naming part upstream first if preferred.
• Sampling in the Collector. Does not help the cost that motivates this,
  which is volume leaving the process, and it cannot recover structure once
  records have already been dropped at the source.
• Naming it for the signal, e.g. Extensions.LogSampling. The Sampler.*
  prefix is chosen for where the component is going: increment 2 makes it
  span-aware and the algorithm is a general sampler, so a log-specific name
  would misdescribe the later increments.

Additional context

Questions I would most like input on:

1. Is this in scope for contrib at all? Worth settling before more is built.
2. Attribute naming. otel.logs.adjusted_count and the adequacy signal
   alongside it are not specified anywhere. If a convention for
   sampling-adjusted counts exists or is planned, I would rather match it than
   invent names that later have to be broken.
3. Should span-only records count against the budget? Forwarding them in
   addition to the whole-stream sample means the budget alone no longer bounds
   total output once per-span coverage is on. Charging them to it keeps one hard
   cap but lets a burst of spans crowd out the whole-stream sample. This is a
   user-visible contract, so I would rather agree on it than pick one.
4. Two dependencies on SDK internals. A decorating exporter has to copy
   pooled records before retaining them (LogRecord.Copy()), and has to forward
   ParentProvider to the exporter it wraps or that exporter cannot resolve its
   Resource. Neither has a public seam today. Both are verified rather than
   assumed, are probed once at startup, and degrade safely - if the copy cannot
   be bound the component forwards everything unsampled rather than emitting
   recycled records. Is an upstream SDK request the right path, and is the
   component acceptable in the interim?

Increment 1 is implemented in #5108, with a runnable example showing ~10x
compression while the recovered total stays within a fraction of a percent of
the true arrival count. Happy to land infrastructure and implementation as two
PRs per REVIEW.md if preferred.

[Yun-Ting]

cc: @jmacd

[jmacd]

I give my full support for this development, and I've given a thorough review to the PR #5108. This is a good approach to reservoir sampling for non-tracing log events that @Yun-Ting and I are experimenting with for our users. Please consider this also development work for re-opening open-telemetry/semantic-conventions#793.