Skip to content

Add experimental FixedSizeRoundRobinReservoir - #8305

Draft
dashpole wants to merge 15 commits into
open-telemetry:mainfrom
dashpole:round_robin
Draft

Add experimental FixedSizeRoundRobinReservoir#8305
dashpole wants to merge 15 commits into
open-telemetry:mainfrom
dashpole:round_robin

Conversation

@dashpole

@dashpole dashpole commented May 5, 2026

Copy link
Copy Markdown
Contributor

This moves the implementation of the round-robin reservoir from #8257 to an experimental module.

Public API Changes

This moves the ConcurrentSafe interface from sdk/metric/exemplar/internal/reservoir to sdk/metric/exemplar, making it part of the public API for exemplar reservoirs. This is required to be able to use it in an experimental module.

Changes

This is an alternative approach to #7447, which had to be reverted in #8249 because of #8238. That approach attempted to make a highly-performant parallel version of algorithm-L, but ultimately is is very difficult to do so correctly.

Instead, this PR tries to improve parallel performance by sharding. Instead of one algorithm-L implementation for a k-sized reservoir, just have k algorithm-L copies for 1-sized reservoirs. This solves #8236 by moving the algorithm-L logic into the storage layer, so that each bucket is individually time-weighted. It also improves the concurrent performance of the fixed-size reservoir by reducing contention on the algorithm-L implementations, since they are distributed across multiple instances.

This should have identical runtime. One k-sized reservoir that handles n Observe calls has a runtime of O(k(1 + log(n/k)). A 1-sized reservoirs that handles n/k Observe calls (we have an even distribution because of our round-robin) would have O(1(1 + log(n/k)) runtime. k 1-sized reservoirs would have a O(k(1 + log(n/k)) runtime.

This has the following downsides:

  • Increased size of the reservoir. It adds two integers and a float64, which might be significant if there are no dropped attributes.
  • Bias from using round-robin.

This uses a simple round-robin strategy. This does introduce a bias. Algorithm-L by itself makes any combination of exemplars equally likely (e.g. the first k and last k are equally likely). Using round-robin makes some exemplar combinations impossible (e.g. the 1st and k+1 th). This would matter if there was periodic behavior that aligned with the reservoir size. I.e. there is something "special" about every kth Offer call. You would be highly likely to get only one "special" exemplar, and less likely to get zero or more than one special events. This tend to makes the results more "average". If this is a concern, I think this could be addressed with some added complexity, and some performance overhead.

FixedSizeRoundRobin benchmarks.

goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/metric/exemplar/x
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
                                     │   new.txt    │
                                     │    sec/op    │
FixedSizeRoundRobinReservoirOffer-24   106.6n ± 27%

FixedSize Reservoir benchmarks

goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/metric/exemplar
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
                           │   old.txt   │
                           │   sec/op    │
FixedSizeReservoirOffer-24   229.0n ± 8%

Gemini helped me write this.

@codecov

codecov Bot commented May 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.43750% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 88.4%. Comparing base (524fc3c) to head (b3708ca).
⚠️ Report is 20 commits behind head on main.

Files with missing lines Patch % Lines
sdk/metric/exemplar/x/reservoir.go 98.4% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@          Coverage Diff          @@
##            main   #8305   +/-   ##
=====================================
  Coverage   88.4%   88.4%           
=====================================
  Files        331     332    +1     
  Lines      21001   21064   +63     
=====================================
+ Hits       18569   18632   +63     
  Misses      2432    2432           
Files with missing lines Coverage Δ
sdk/metric/exemplar/fixed_size_reservoir.go 100.0% <ø> (ø)
sdk/metric/exemplar/histogram_reservoir.go 96.8% <ø> (ø)
...dk/metric/internal/aggregate/filtered_reservoir.go 100.0% <100.0%> (ø)
sdk/metric/exemplar/x/reservoir.go 98.4% <98.4%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dashpole

dashpole commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

Link failure is because the module isn't released yet.

@dashpole
dashpole marked this pull request as ready for review May 5, 2026 18:21
pellared
pellared previously approved these changes Jun 12, 2026
Comment thread sdk/metric/exemplar/x/reservoir.go
Comment thread CHANGELOG.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new experimental exemplar reservoir implementation (FixedSizeRoundRobinReservoir) under go.opentelemetry.io/otel/sdk/metric/exemplar/x, intended to improve concurrent Offer performance by sharding sampling across per-bucket Algorithm‑L state and selecting buckets via round-robin.

Changes:

  • Adds the new experimental reservoir implementation, plus accompanying tests and a parallel benchmark.
  • Introduces a new experimental module (sdk/metric/exemplar/x) with its own go.mod/go.sum and documentation.
  • Wires the module into the repo’s release/module metadata (versions.yaml) and documents it in CHANGELOG.md.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
versions.yaml Adds the new experimental module to the experimental-metrics module-set.
sdk/metric/exemplar/x/reservoir.go Implements FixedSizeRoundRobinReservoir and supporting storage/sampling logic.
sdk/metric/exemplar/x/reservoir_test.go Adds unit tests for the new reservoir.
sdk/metric/exemplar/x/benchmark_test.go Adds a parallel benchmark for Offer.
sdk/metric/exemplar/x/README.md Documents the experimental reservoir and intended usage tradeoffs.
sdk/metric/exemplar/x/go.mod Defines the new experimental module and its dependencies.
sdk/metric/exemplar/x/go.sum Dependency checksums for the new module.
CHANGELOG.md Announces the new experimental reservoir.

Comment thread sdk/metric/exemplar/x/reservoir.go
Comment thread sdk/metric/exemplar/x/reservoir.go
Comment thread sdk/metric/exemplar/x/reservoir.go
Comment thread sdk/metric/exemplar/x/reservoir_test.go
Comment thread sdk/metric/exemplar/x/go.mod
Comment thread CHANGELOG.md Outdated
@pellared
pellared dismissed their stale review June 12, 2026 11:40

approved by accident

@pellared

pellared commented Jun 12, 2026

Copy link
Copy Markdown
Member

I have not found any other issues than the ones that are currently reported.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

Comment thread sdk/metric/exemplar/x/reservoir.go
Comment thread sdk/metric/exemplar/x/reservoir_test.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 3 comments.

Comment thread sdk/metric/exemplar/x/reservoir.go Outdated
Comment thread sdk/metric/exemplar/x/benchmark_test.go
Comment thread CHANGELOG.md
@dashpole
dashpole marked this pull request as draft July 9, 2026 15:22
@dashpole
dashpole force-pushed the round_robin branch 2 times, most recently from 794e941 to e3eacf3 Compare August 13, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants