Skip to content

[LLM serving 2/x] Add batching-backed serving session lifecycle - #23277

Merged
metascroy merged 2 commits into
mainfrom
llm-serving/02-session-lifecycle
Oct 1, 2026
Merged

[LLM serving 2/x] Add batching-backed serving session lifecycle#23277
metascroy merged 2 commits into
mainfrom
llm-serving/02-session-lifecycle

Conversation

@metascroy

@metascroy metascroy commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Introduce an experimental ServingRuntime that manages named sessions on top of the existing batching runtime. It owns one Runner and scheduler, while physical session state and cleanup remain with batching::Session.

  • Add asynchronous open, close, and reset operations, with bounded session capacity and pending operations processed in admission order.
  • Reset destroys and reopens the physical session under the same key. Reopening failure leaves the key explicitly unavailable until reset succeeds or close releases it.
  • Use RAII for physical session closure, and provide synchronized readiness information and idempotent shutdown that settles accepted operations.
  • Add thread-safe Runner::initialization_state() reporting Pending, Ready, or Failed. Initialization failures and exceptions refuse work and settle queued opens without an executor wrapper.

This PR covers lifecycle management only. Generation, tokenization, streaming, and prefix reuse are added separately; no new scheduler or decode loop is introduced.

@pytorch-bot

pytorch-bot Bot commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23277

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b6eb36b with merge base 273ecf9 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 30, 2026
mergennachin
mergennachin approved these changes Sep 30, 2026
View reviewed changes

@mergennachin mergennachin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. When generation is added, please test that resetting or closing a session stops its current response before reporting success, and that other sessions can continue generating.

@metascroy
metascroy force-pushed the llm-serving/01-contracts branch from ffb3ce7 to cd6e869 Compare September 30, 2026 19:51
@metascroy
metascroy force-pushed the llm-serving/02-session-lifecycle branch from 15fa4f2 to 8e060fb Compare September 30, 2026 19:52
@metascroy
metascroy force-pushed the llm-serving/01-contracts branch from cd6e869 to 5f693b4 Compare September 30, 2026 21:40
@metascroy
metascroy force-pushed the llm-serving/02-session-lifecycle branch from 8e060fb to 5127145 Compare September 30, 2026 21:46
@metascroy
metascroy force-pushed the llm-serving/02-session-lifecycle branch from 5127145 to 53cdd22 Compare September 30, 2026 22:56
Base automatically changed from llm-serving/01-contracts to main October 1, 2026 17:00
@metascroy
[LLM] Add batching-backed serving session lifecycle
ce0e06e 
Give serving clients stable logical session keys while keeping physical execution and cleanup in the existing batching runtime. Reset replaces state under the same key and reports reopening failure explicitly; bounded admission and ordered lifecycle operations make later concurrent generation safe to add separately.

Runner owns thread-safe Pending/Ready/Failed initialization state and contains initialization exceptions; serving uses the original executor directly.

Test Plan: Serving and batching CMake targets passed. Standalone validation passed 20 lifecycle tests and 164 batching tests, 100 normal lifecycle repetitions, 25 ThreadSanitizer repetitions, and 19 lifecycle tests with exceptions disabled. Initialization exceptions settle as not-ready errors rather than terminating the engine thread.

Authored with AI assistance from Devmate.
@metascroy
up
b6eb36b
@metascroy
metascroy force-pushed the llm-serving/02-session-lifecycle branch from 53cdd22 to b6eb36b Compare October 1, 2026 17:08
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@metascroy
metascroy marked this pull request as ready for review October 1, 2026 19:13
@metascroy
metascroy merged commit 738fc28 into main Oct 1, 2026
271 of 273 checks passed
@metascroy
metascroy deleted the llm-serving/02-session-lifecycle branch October 1, 2026 22:45

This branch was previously deployed

1 inactive deployment
cadence — b6eb36bf Deployed Oct 1, 2026 by metascroy via hifi-op-test / hifi4 #30863
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mergennachin mergennachin mergennachin approved these changes
@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820 larryliu0820 is a code owner
@kirklandsign kirklandsign Awaiting requested review from kirklandsign

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@metascroy @mergennachin