[Bug]: ATIF duplicates the user step after a response-less LLM retry

[afourniernv]

Affected area

Observability or exporters; framework integrations

Current behavior

When one logical model call has a response-less failed provider attempt followed by a successful retry, ATOF correctly retains both physical attempts, but the ATIF exporter can emit:

user
user
assistant

The user only sent one message. The retry is being projected as another user turn.

The immediate cause is that Relay's ATIF lookup treats only LLM end events containing response data as paired ends. A failed LLM_END with no response data is therefore treated like a missing end, and the retry start can become another user step.

Expected behavior

The conversational trajectory should not claim that the user sent the message twice:

user
assistant

ATOF should continue to retain every physical attempt. Before changing ATIF, Relay should define how a producer identifies a logical model call, its physical attempts, and the response accepted by the host. If multiple attempts are represented by one ATIF step, the projection should preserve the original call timestamp, actual attempt count, and available status, timing, and cost information.

Relay should not infer a logical retry group from a generic correlation ID unless that meaning is part of the producer contract.

Steps to reproduce

1. Emit an LLM start event for a user request.
2. End that attempt with an error and no response payload.
3. Emit a second LLM start for the host's retry of the same logical call.
4. End the retry successfully with an assistant response.
5. Feed the ATOF events through Relay's AtifExporter.
6. Observe that current Relay produces two user steps before the assistant step.

Validation performed

• Replayed a captured response-less failure followed by a successful retry against Relay main at a5ae6d229b45fc47459aef16530a92ad771aeaa9. It produced user, user, assistant.
• Replayed six provider-backed Hermes traces. Runs without a failed attempt were unchanged; traces containing retries exposed duplicate user steps on current main.
• Tested the implementation formerly proposed in fix: avoid duplicate ATIF turns for response-less retries #954. It reduced the simple example to user, assistant, but selected the retry timestamp instead of the original request timestamp and removed the failed attempt's status, timing, and count from standalone ATIF.
• Tested a failure-success-failure-success sequence sharing one correlation identity. fix: avoid duplicate ATIF turns for response-less retries #954 collapsed only part of the sequence, demonstrating that validating attempts independently is insufficient.

The reproduction uses offline exporter replay after capture, so it does not require a provider credential.

Environment

• NeMo Relay: a5ae6d229b45fc47459aef16530a92ad771aeaa9
• Surface: Rust AtifExporter
• Input: deserialized ATOF JSONL plus constructed multi-attempt edge cases
• Related closed implementation: fix: avoid duplicate ATIF turns for response-less retries #954

Impact

ATIF consumers can interpret a provider retry as another user instruction. This skews turn counts, timing, evaluations, visualizations, and training-oriented trajectory processing.

ATOF remains a complete workaround for inspecting the physical execution, but consumers expecting a conversational ATIF trajectory receive misleading data.

Fix constraints

• Preserve the first logical-call start rather than replacing it with the retry start.
• Preserve the response actually accepted by the host.
• Keep all physical attempts in ATOF.
• Do not silently lose attempt count, error status, timing, or cost when ATIF coalesces attempts.
• Validate the complete attempt group, including interleaved and reused-ID cases.
• Add a producer-level integration test instead of relying only on hand-authored metadata.
• Avoid an all-pairs comparison of cloned provider request bodies.