Bug: Multiple exemplars per data point overwrite each other in OTel metrics flattening

@parmesant

Summary

Tracked from PR #1661 (comment: #1661 (comment)).

Current Behavior

In src/otel/metrics.rs, the insert_exemplars helper (introduced in PR #1661 to replace the old flatten_exemplar) iterates over the Vec<Exemplar> for a data point and writes each exemplar's fields directly into the same parent map, using static key names:

exemplar_time_unix_nano
exemplar_span_id
exemplar_trace_id
exemplar_value

Because every exemplar in the slice writes to the same keys, each iteration overwrites the previous one. After the loop only the last exemplar is retained in the flattened record. All prior exemplars are silently dropped.

This affects Sum, Histogram, and ExponentialHistogram metric types — all of which carry exemplars as Vec<Exemplar>.

The previous implementation (flatten_exemplar) returned a Vec<Map<String, Value>> (one Map per exemplar), which at least preserved each exemplar in a separate structure, though the call sites then merged only the first map.

Ideal Behavior

Exemplars for a data point should be preserved in full. Possible approaches:

Array column – Serialize the exemplars as a Value::Array of Value::Object entries and store them under a single column (e.g., exemplars). Each element would contain exemplar_time_unix_nano, exemplar_span_id, exemplar_trace_id, exemplar_value, and any filtered attributes. This is the most semantically faithful representation and aligns with how the OTel spec treats exemplars (a list per data point).
Indexed flat columns – Emit exemplar_0_time_unix_nano, exemplar_1_time_unix_nano, … etc. This avoids nested types but creates a variable, unbounded number of columns.
Emit one row per exemplar – Fan out exemplar data so each exemplar becomes its own row joined to its parent data point's fields. This is queryable without nested-type support but multiplies row count.

Option 1 (array column) is recommended as it matches the OTel data model, keeps schema stable, and allows query-layer unnesting when needed.

References

PR Ingestion optimization #1661: Ingestion optimization
OTel Metrics proto spec: exemplars are repeated Exemplar on NumberDataPoint, HistogramDataPoint, and ExponentialHistogramDataPoint.

🔗 Related PRs

#1177 - refactor: specialized flatten happens at the same semantic location [merged]
#1298 - fix: stringify otel attributes of array types [merged]
#1477 - Fix for otel-metrics JSON [merged]
#1520 - consolidate attributes insertion in otel-metrics [merged]
#1648 - add series hash in metrics ingestion, bounded streaming merge [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

@parmesant

Hey @parmesant — I'd love to take a stab at this! Before I dive in, a couple of quick questions to make sure I pick the right approach:

Schema migration — switching from the flat exemplar_* columns to an exemplars array column will be a breaking change for anyone already querying those fields. Is that acceptable, or do we need a transition period / compatibility shim?
Query-layer support — does the current query layer (DataFusion) handle unnesting a Value::Array of objects cleanly, or is there anything on that side we should be aware of before committing to the array-column approach?
Scope — should this touch all three affected types (Sum/NumberDataPoint, Histogram, ExponentialHistogram) in one go, or would you prefer a narrower first PR?

Happy to wait for your thoughts before opening anything!

@prabhaks

Hey @prabhaks! Thanks a lot for picking this up (and apologies for the delayed response)

I think introducing a flag to modify the behavior would be ideal for someone who might already be using exemplars as is. The default could be the new behavior. We can think about it later
I think there is support for querying Value::Array but not sure about Value::Array of Value::Object. We might need to run a small PoC to guide us to the correct behavior.
I think it would be best if we introduce this as one change rather than a staggered release (even if it takes some time to test and validate)

Happy to discuss this further before you get started

@parmesant

Thanks @parmesant, this all makes sense. Here's my plan:

Flag: I'll add one (e.g. P_OTEL_FLATTEN_EXEMPLARS) and default it to the new array-based behavior, while keeping the legacy flat exemplar_* columns available for anyone already depending on them.
Querying array-of-objects: good news here, this pattern is already used in the metrics flattening path. flatten_summary emits data_point_quantile_values as an array of {quantile, value} objects (see src/otel/metrics.rs), so arrow-json ingestion of Value::Array of Value::Object already works for metrics. That covers the ingestion side. I'll still run a small PoC on the DataFusion query/unnest side and share results before we lock the schema.
Single change: agreed, one PR covering Sum, Histogram, and ExponentialHistogram (all carry Vec<Exemplar>).

One more thing I noticed while digging in: insert_exemplars currently writes each exemplar's filtered_attributes straight into the data-point map, so exemplar attributes leak into compute_series_hash and end up fragmenting the series identity. Nesting exemplars under a single column fixes that too.

I'll start with the PoC and report back before going further. Thanks for the guidance!

Ran the PoC for point 2 and the array-of-objects approach works end to end. Good to proceed.

I tested against the same versions we use here (datafusion 53, arrow/arrow-json 58) and mirrored both real paths: arrow-json decoding for ingestion and a DataFusion SessionContext with default features for querying.

Ingestion: arrow-json infers the exemplars array-of-objects as List<Struct<...>>, and both exemplars are preserved in the single data-point row (no overwrite), including a nested filtered_attributes struct.

Querying: every case I tried worked:

Reading the exemplars column directly returns both objects.
array_length(exemplars) returns 2, so both survive.
unnest(exemplars) fans the array out to one row per exemplar.
Struct field access after unnest works, e.g. ex['span_id'] and ex['value'].
Filtering on a nested field also works, e.g. WHERE value > 2.0 after unnest returns just the matching exemplar.

So DataFusion handles List<Struct> cleanly (unnest, field access, and filtering on nested fields). That clears up the uncertainty around Value::Array of Value::Object.

Next I'll start on the implementation: rewrite insert_exemplars to emit the array column, add the flag defaulting to the new behavior, update OTEL_METRICS_KNOWN_FIELD_LIST, and add tests covering multiple exemplars for Sum, Histogram, and ExponentialHistogram. Will open a PR once it's ready.

fix(otel): preserve all exemplars per data point (parseablehq#1662)

fix(otel): preserve all exemplars per data point (#1662) (#1720)

Navigation Menu

Repository navigation

Bug: Multiple exemplars per data point overwrite each other in OTel metrics flattening #1662

Summary

Current Behavior

Ideal Behavior

References

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bug: Multiple exemplars per data point overwrite each other in OTel metrics flattening #1662

Description

Summary

Current Behavior

Ideal Behavior

References

Activity

coderabbitai commented on Jun 2, 2026

prabhaks commented on Jul 2, 2026

parmesant commented on Jul 6, 2026

prabhaks commented on Jul 11, 2026

prabhaks commented on Jul 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions