Repository navigation
[SDK] Stop metric collection between callbacks once a bounded shutdown gives up - #4758
hidingmyname wants to merge 3 commits into
Conversation
…n gives up MetricCollector::Produce() and ObservableRegistry::Observe() stop between meters and callbacks once the periodic reader has stopped waiting for the collection, and the reader no longer exports a collection after shutdown was requested. Fixes open-telemetry#4746
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4758 +/- ##
==========================================
+ Coverage 86.62% 86.65% +0.04%
==========================================
Files 530 531 +1
Lines 21169 21190 +21
==========================================
+ Hits 18335 18360 +25
+ Misses 2834 2830 -4
🚀 New features to boost your workflow:
|
mateenali66
left a comment
There was a problem hiding this comment.
Thanks for the fix.
LGTM, but CI fails on the new test, see comments about clang-tidy and iwyu.
Both new tests fail on main and pass at head 20 times out of 20, and the reader tests are clean under TSan.
See also an optional comment on test coverage for the meter check.
Will approve once CI is green.
| #include <condition_variable> | ||
| #include <cstddef> | ||
| #include <cstdlib> | ||
| #include <memory> |
There was a problem hiding this comment.
The three iwyu jobs fail on this file, and they allow no warnings:
periodic_exporting_metric_reader_test.cc should add these lines:
#include <initializer_list>
#include <ratio>
periodic_exporting_metric_reader_test.cc should remove these lines:
- #include <memory> // lines 10-10
Please add both and remove <memory>, the test still builds and passes that way.
|
|
||
| struct SlowCallbackState | ||
| { | ||
| std::chrono::milliseconds duration; |
There was a problem hiding this comment.
| std::chrono::milliseconds duration; | |
| std::chrono::milliseconds duration{0}; |
Fixes one of the two new clang-tidy warnings that put both clang-tidy jobs over their limit.
| StartedSignal started; | ||
| }; | ||
|
|
||
| void SlowObservableCallback(opentelemetry::metrics::ObserverResult /* result */, void *state) |
There was a problem hiding this comment.
| void SlowObservableCallback(opentelemetry::metrics::ObserverResult /* result */, void *state) | |
| // NOLINTNEXTLINE(performance-unnecessary-value-param) | |
| void SlowObservableCallback(opentelemetry::metrics::ObserverResult /* result */, void *state) |
The callback type requires the by-value ObserverResult, same as async_instruments_test.cc:24.
| ResourceMetrics resource_metrics; | ||
| meter_context_->ForEachMeter([&](const std::shared_ptr<Meter> &meter) noexcept { | ||
| // Stop before the next meter once the reader has stopped waiting for this collection. | ||
| if (CollectionCancellation::IsRequested()) |
There was a problem hiding this comment.
Removing this check leaves both new tests green, because Observe() already skips the callbacks of meter_b.
A MetricCollectorTest that records one counter value, calls Produce() inside a CollectionCancellation::Scope with the flag set, and expects kTimeout and an empty scope_metric_data_, passes at head and fails without this check.
(optional)
|
Thanks for the review. Pushed the changes you listed: the iwyu include changes, I also added the optional test as |
Fixes #4746
Changes
PeriodicExportingMetricReader::Shutdown(timeout)returned only after an in-flight collection had run every remaining meter and observable callback, and the result of that collection could still be exported after the shutdown budget was spent. This follows the direction discussed in #4746:MetricCollector::Produce()checks between meters, andObservableRegistry::Observe()checks between callbacks, whether the reader running the collection has stopped waiting for it. If so, they stop and the collector returnskTimeoutwith whatever it had already collected.PeriodicExportingMetricReader::CollectAndExportOnce()does not callExport()onceis_stop_requested_is set, for any producer status. Before, a partialkTimeoutresult was dropped only when the collection took longer thanexport_timeout_millis_(30 s by default).The reader passes its
is_stop_requested_flag to the collection code through a small internal helper,sdk/src/metrics/collection_cancellation.{h,cc}. It is a thread-local scope set by the reader's worker thread for the duration ofCollectAndExportOnce(). It lives undersdk/src, is not installed, and adds no public API or ABI change. Collections started any other way, such asMetricReader::Collect()from a pull exporter, are unaffected because no scope is active on their thread. For Bazel,sdk/src/metrics/BUILDnow lists the headers in that directory ashdrs.Limits:
join()aroundExport()mentioned in [BUG] BatchSpanProcessor::Shutdown ignores its timeout when the worker thread is blocked in Export() #4532 is not changed.CHANGELOG.mdupdated for non-trivial changesTests
Two new tests in
sdk/test/metrics/periodic_exporting_metric_reader_test.cc, both using the reader's default options (30 s export timeout):ShutdownStopsCollectionBetweenObservableCallbacks: aMeterProviderwith four 500 ms observable gauge callbacks across two meters, andShutdown(100ms)called while the first callback runs. The test checks that only one callback runs, that shutdown returns well before the remaining callbacks would finish, and that nothing is exported.ShutdownSkipsExportOfTimedOutCollection: the case from the review. A producer returnskTimeoutwith one metric after the shutdown budget is spent, and the exporter takes 2 s. The test checks thatExport()is not called and that shutdown does not wait for the exporter.Without the change in
sdk/src, both tests fail: in the first, all four callbacks run (about 2000 ms); in the second, the partial result is exported andShutdownwaits for the exporter. With the change, both pass, and pass 20 out of 20 times with--gtest_repeat=20.ctest --test-dir sdk/test/metrics: 100% of 266 tests pass.Built with CMake 4.4.5, GCC 15.3.0, Debug, Linux x86_64. Formatted with clang-format 18 using the repository's
.clang-format. Not run with Bazel or on Windows/macOS.