Repository navigation
Bug: OTEL_METRICS_KNOWN_FIELD_LIST out of sync with emitted keys (metric series fragmentation and collision) #1723
Description
Summary
OTEL_METRICS_KNOWN_FIELD_LIST in src/otel/metrics.rs is out of sync with the keys the metric flatteners actually emit. compute_series_hash treats every top-level key that is NOT in this list as a series label, so the mismatch corrupts physical-series identity for metrics in two opposite ways: some real per-sample fields are treated as labels (fragmenting a series into many), and some names that are never emitted are listed (which can merge distinct series if a user attribute happens to use one of those names).
Background
compute_series_hash (src/otel/metrics.rs) hashes metric_name plus every top-level key/value on the flattened data point that is not present in OTEL_METRICS_KNOWN_FIELD_LIST. The result is stored per sample as __series_hash and used by the query layer to group samples into series. So the list must exactly match the set of non-label (sample/metric-level) keys the flatteners emit.
Problem 1: emitted keys missing from the list (series fragmentation)
These are emitted as top-level keys but are absent from the list, so their per-sample values get hashed into the series identity. Samples of the same physical series then get different __series_hash values and are split into many spurious series.
min— emitted byflatten_histogram(around line 351, viainsert_number_if_some)max— emitted byflatten_histogram(around line 352)data_point_quantile_values— emitted byflatten_summary(around line 471) as a nested array of{quantile, value}objects
Impact: histogram series fragment whenever min/max change across scrapes (routine, especially for DELTA temporality), and summary series fragment as quantile values change every sample.
Problem 2: listed names that are never emitted (dead entries, with collision risk)
These appear in the list but are never emitted as top-level keys, so they exclude any real attribute of the same name from the series hash:
value(list) — only ever emitted nested inside thedata_point_quantile_valuesobjects (around line 484), never as a top-level key.valueis a very common attribute name, so a user label namedvaluewould be silently dropped from series identity and distinct series would collide.quantile(list) — same situation (nested only, around line 479); collision risk for an attribute namedquantile.data_point_quantile_values_quantileanddata_point_quantile_values_value(list) — never emitted anywhere. Harmless dead entries (names are specific enough to be unlikely user labels), but they should be removed for correctness.
Proposed fix
- Add
min,max, anddata_point_quantile_valuestoOTEL_METRICS_KNOWN_FIELD_LIST. - Remove
value,quantile,data_point_quantile_values_quantile, anddata_point_quantile_values_value. - Update the list length constant and the
OTEL_METRICS_KNOWN_FIELD_LIST.len()test assertion accordingly. - Add tests asserting that differing
min/max(histogram) anddata_point_quantile_values(summary) do not changecompute_series_hash, and that a data point attribute namedvalueorquantiledoes change it.
Related
Same class of issue as exemplar attributes leaking into the series hash, discussed in #1662 (see also #1720). Found while auditing the flattening code after that fix.
Activity
coderabbitai commentedon Jul 12, 2026
🔗 Related PRs
#1679 - perf: field stats more performant for high volume ingestion [closed]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
[-]Bug: Summary metric quantile values fragment physical-series identity (missing from OTEL_METRICS_KNOWN_FIELD_LIST)[/-][+]Bug: OTEL_METRICS_KNOWN_FIELD_LIST out of sync with emitted keys (metric series fragmentation and collision)[/+]prabhaks commentedon Jul 12, 2026
@parmesant I opened this issue similar to what we had earlier (#1662). Please take a look!
fix(otel): sync OTEL_METRICS_KNOWN_FIELD_LIST with emitted keys (pars…
fix(otel): sync OTEL_METRICS_KNOWN_FIELD_LIST with emitted keys (#1723)…