Repository navigation
Conversation
|
A before/after text simulation on production LiveKit Cloud with an agent that hangs up via EndCallTool and vetoes every run from on_simulation_end. On main, on_simulation_end never runs and the run passes. On this branch, it runs after end_call and its veto fails the run. simulation-finalize-after-end-call.mp4 |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if reason == CloseReason.USER_INITIATED: | ||
| # only an agent ending its own session leaves the simulator yet | ||
| # to finalize; a job shutdown has to stay within the supervisor's | ||
| # shutdown_process_timeout | ||
| await self._session_host.hold_for_simulation_finalize() |
There was a problem hiding this comment.
🔴 Room deletion preempts simulation finalization
When delete_room_on_close is enabled, close triggers RoomIO._on_agent_session_close before the hold starts. Room deletion disconnects the transport, so the simulator cannot deliver finalize_simulation or receive the agent verdict.
Learn more
RoomIO can schedule room deletion from the session's close event when delete_room_on_close is enabled. The close event is emitted before the new simulation hold begins, so holding the host cannot keep the room transport connected. The remote simulator sends its finalize request over that same room transport.
Example: A simulated agent enables delete_room_on_close, ends its session, and RoomIO schedules deletion during close. The hold then waits for a finalize request over a deleted room and eventually times out without an agent verdict.
Recommended fix: Defer RoomIO._on_agent_session_close's room deletion for simulated agent-initiated closes until after hold_for_simulation_finalize returns. Preserve immediate deletion for other close reasons.
Was this helpful? React with 👍 or 👎 to provide feedback.
| with contextlib.suppress(asyncio.TimeoutError): | ||
| await asyncio.wait_for( | ||
| self._simulation_released.wait(), timeout=_SIMULATION_FINALIZE_TIMEOUT | ||
| ) |
There was a problem hiding this comment.
🟡 Replacement sessions lose forwarded events
When attach() replaces an attached session, _events_registered skips subscribing to the new session. The old session keeps forwarding events, while the new session's events never reach the remote client.
Learn more
SessionHost owns event subscriptions for its currently attached AgentSession. Calling attach() again before detach() replaces _session without removing subscriptions from the first session; _events_registered prevents subscriptions on the second. detach() then attempts to remove handlers from the second session and loses its reference to the first. The new shared-host test exercises this exact sequence but checks only host.session, not event delivery.
Example: Attach session A, then attach B. A's user_state_changed reaches the client; B's does not. Calling detach() still leaves A's callback subscribed.
Recommended fix: Either detach the existing session before replacing it in attach(), or explicitly reject an attachment until the previous session is detached. Test event subscriptions when attaching two sessions.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # only an agent ending its own session leaves the simulator yet | ||
| # to finalize; a job shutdown has to stay within the supervisor's | ||
| # shutdown_process_timeout | ||
| await self._session_host.hold_for_simulation_finalize() |
There was a problem hiding this comment.
I don't think it's the right fix, should we just "Drain"?
Like make sure the entire queue is consumed then we just await the main task
Rename register_session to attach, split event unhooking out of aclose into detach, and expose the attached session. Request handling captures the session once per request and finalize_simulation moves into its own method. No behavior change.
…outlives the session The SessionHost is now owned by the job rather than the AgentSession: the primary session attaches on start and detaches on close, and the job closes the host during shutdown. Under a simulation, shutdown keeps the host serving until the simulator finalizes, leaves, or session_end_timeout passes, so on_simulation_end still runs when the agent ends the call. While no session is attached, session requests fail with "no active session" instead of being dropped; finalize_simulation needs only the job and is answered either way.
62bbafa to
fc50c34
Compare
u9g
changed the title
| def _check_simulator_left(*_: Any) -> None: | ||
| if not room.isconnected() or not room.remote_participants: | ||
| simulator_left.set() |
There was a problem hiding this comment.
🔴 Simulator departure delays job shutdown
When the simulator leaves but another participant remains, _check_simulator_left waits for a finalize request that cannot arrive. The job stays open until session_end_timeout, which defaults to five minutes.
Learn more
The job already identifies the simulator by ATTRIBUTE_SIMULATOR in the disconnect callback. This wait instead tests whether every remote participant has left. A SIP leg or avatar can remain after the simulator disconnects, so the wait runs for the entire 300-second default even though no simulator can send finalize. The job retains its room and delays its shutdown callbacks during that time.
Example: A room contains a simulator and a SIP participant. The simulator disconnects after judging its run. The SIP participant stays; remote_participants is nonempty, so shutdown waits five minutes rather than finishing when the simulator leaves.
Recommended fix: Track the participant with ATTRIBUTE_SIMULATOR and finish the wait when that participant disconnects or is absent, regardless of other participants. Handle room disconnect and finalization as separate completion conditions.
Was this helpful? React with 👍 or 👎 to provide feedback.
Stacked on #8099.
Problem
When an agent ends the call itself during a simulation, its
on_simulation_endverdict is silently dropped.The simulator asks for the agent's verdict with
finalize_simulationonly after the run is over and it has judged the transcript, which takes a few seconds of LLM calls. When the agent hangs up (for example withEndCallTool),AgentSessioncloses itsSessionHostalong with itself, andEndCallTool's defaultdelete_room=Truethen deletes the room. By the time the simulator sendsfinalize_simulation, nothing is listening. The simulator logsfinalize_simulation failed; no agent verdictand reports the run using only its own verdict.In production over the last 24h before this PR was opened, this hit 2,945 of ~21.8k simulation runs.
Fix
The
SessionHostbelongs to the job rather than the session.JobContextowns one host. The primaryAgentSessionattaches to it when it starts and detaches when it closes. Outside a job, the session still owns and closes its own host.Explicit attach and detach.
SessionHosthasattach(),detach()and asessionproperty. At most one session is attached at a time.No attached session. Session requests get an explicit
no active sessionerror instead of no reply.finalize_simulationneeds only the job, so it is answered either way.Job shutdown. Under a simulation,
_shutdown_jobkeeps the host up until one of these happens, then closes it beforeShuttingDown,room.disconnectand the shutdown callbacks (whereEndCallTooldeletes the room):session_end_timeoutpasses.The supervisor waits for
ShuttingDownwith no time limit, so this wait doesn't count againstclose_timeout.No wire protocol change in this PR. The agent now stays in the room after it hangs up, so the simulator learns the call ended from
session_closed(feat(voice): send session_closed to the remote session when the session closes #8099). The simulator side is livekit/agents-private#756, and it has to be deployed before this PR is released.The first commit is a pure refactor:
register_sessionbecomesattach, anddetach()is extracted. The second commit is the fix.Verification
tests/test_session_host.py: a request with no attached session is refused, finalize is answered after detach, and detach stops forwarding events.tests/test_simulation_finalize.py: the job owns the host and sessions share it, and the session owns the host outside a job. The shutdown wait ends on finalize, ends when the simulator leaves, skips the wait when the room is gone, and gives up at the timeout.Not covered
RoomIOwithdelete_room_on_close=True(the default isFalse) still deletes the room when the session closes, which takes the host down before finalize.Parity: livekit/agents-js#2793