[u9g]

A simulated job reports entity recognition (the share of key entities, such as names, codes and email addresses, that the listener transcribed correctly) only as entities_recognized / entities_uttered. When a job scores 50%, a client has no way to show which entity was misheard or what was heard in its place, or to take the reader to that moment in the call.

The simulator already decides this for each saying of each entity, and already knows each turn's chat item id. This adds both to JobMetrics.Turn:

repeated Entity entities = 19;
// the agent's chat item id for the first message this turn holds, as its
// stored transcript carries it; "" without a RemoteSession
string item_id = 20;

message Entity {
  string said = 1;      // the entity as the speaker said it
  string heard = 2;     // the listener's normalized words where it landed; "" if dropped
  bool recognized = 3;  // heard with no word errors
}

item_id matches the chat item ids in the agent session's stored transcript, so a client can open the transcript at a turn.

Additive only. Existing counts and rates are unchanged.

[changeset-bot]

🦋 Changeset detected

Latest commit: bb6c86c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
github.com/livekit/protocol	Patch
@livekit/protocol	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR