Skip to content

Commit 51cdd12

Browse files
authored
[LLM][batching][image] Support opaque prepared types in batching (pytorch#23451)
## Summary Add opaque generation inputs to the batching layer while keeping model-specific representations inside the executor. - `GenerationInput` accepts raw token vectors or shared `PreparedInputPtr` backing with a fixed logical decoder-position count. - The runner chunks opaque input by logical positions and forwards shared backing without inspecting or copying its representation. A pending prediction is submitted as raw-token prefill before opaque chunks; generated-token feedback stays on the raw path. - Adapt existing raw-token consumers and add focused coverage for opaque slicing, shared ownership, pending-token handling, and invalid opaque metadata. This PR adds no preparation API, new scheduler work type, or type tag. `ModuleExecutor` remains raw-token-only. Image decoding, vision execution, and serving/HTTP image integration are not implemented here. ## Future Image Use (Sketch) An MG-specific input could hold token runs and image spans: ```text MGInput = [tokens, Image{fixed_positions, bytes | cached_embeddings}, tokens] ``` Image position counts would be determined before submission. During `execute()`, MG could lazily replace image bytes with cached embeddings in backend-private mutable storage, then reuse them across chunks without changing logical size or layout. This is a follow-up use of the opaque input interface, not an implementation in this PR. ## Validation Local checks on macOS: - Clean configure, build, and install using the unchanged CI workflow's LLM/LLM_RUNNER options; optional ModuleExecutor build also passed. - Batching/serving tests plus opaque UBSan and no-exceptions suites passed; installed-worker tests passed. - Server Python tests and MG server tests with rebuilt native workers: 869 passed, 7 skipped. - Public batching headers compile without Tensor includes and with exceptions disabled; scoped lint and change validation passed.
1 parent ba8b711 commit 51cdd12

15 files changed

Lines changed: 548 additions & 69 deletions

‎examples/llm_server/cpp/test_batching_worker_main.cpp‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -150,10 +150,11 @@ class TraceExecutor final : public batching::testing::FakeExecutor {
150150
}
151151
auto& tokens = it->second;
152152
tokens.resize(position);
153+
const auto& source = *std::get<batching::TokenInputPtr>(input.payload);
153154
tokens.insert(
154155
tokens.end(),
155-
input.tokens->begin() + input.offset,
156-
input.tokens->begin() + input.offset + input.size);
156+
source.begin() + input.offset,
157+
source.begin() + input.offset + input.size);
157158
if (output.outputs[i] && !output.outputs[i]->tokens.empty()) {
158159
const auto& generated = output.outputs[i]->tokens;
159160
tokens.insert(tokens.end(), generated.begin(), generated.end() - 1);

‎extension/llm/batching/README.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,26 @@
11
# Batched generation, session cloning, and prefix reuse
22

3-
`Runner` schedules token deltas through an `Executor`. `ModuleExecutor` runs
4-
those batches against a registered off-graph KV cache. Session cloning is a
5-
runtime operation; token matching and snapshot eviction are a separate,
3+
`Runner` schedules generation inputs through an `Executor`. `ModuleExecutor`
4+
runs raw-token batches against a registered off-graph KV cache. Session cloning
5+
is a runtime operation; token matching and snapshot eviction are a separate,
66
caller-owned policy in `batching::PrefixCache`.
77

8+
## Opaque inputs
9+
10+
Ordinary text and decode use raw token IDs. Supporting executors can also consume
11+
an opaque `PreparedInputPtr` through `session.generate_async(input, config,
12+
callback)`. Its model-specific representation must be compatible with that
13+
executor; the framework provides no preparation or construction API.
14+
Executors opt in through `Executor::accepts()`, which defaults to false. The runner
15+
checks it before scheduling, so rejected opaque input fails only its generation.
16+
`PreparedInput::kind()` provides a process-local type identity for this check,
17+
without RTTI; matching a type does not establish model or device compatibility.
18+
19+
The complete backing reports a fixed decoder-position count through `size()`.
20+
Prefill chunks share its owner and slice its logical positions; decode feedback
21+
remains raw tokens. Input metrics count positions, not necessarily text tokens.
22+
Metadata and logical contents stay immutable. `ModuleExecutor` remains raw-token-only.
23+
824
## Cloning a session
925

1026
`Session::clone_async(upto)` returns a future containing a new Session, or

‎extension/llm/batching/executor.h‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,8 @@
99
#pragma once
1010

1111
// The seam between the batched runner and whatever actually runs a forward.
12-
// The interface is expressed in plain tokens and positions, so a fake needs
13-
// neither a .pte nor a GPU.
12+
// Execution carries raw tokens or opaque inputs and logical positions, so a
13+
// fake needs neither a .pte nor a GPU.
1414
//
1515
// Sessions live here because the cache owns their identity. Until a batched
1616
// cache exists an implementation may number them however it likes.
@@ -22,7 +22,7 @@
2222
//
2323
// -- Session state ----------------------------------------------------------
2424
//
25-
// A session's state is a sequence of committed tokens, and its length is the
25+
// A session's state is a sequence of committed positions, and its length is the
2626
// session's position: the absolute position the next token will occupy. A
2727
// freshly opened session has length 0. Nothing here says the state is
2828
// positionally addressable -- a KV cache and a recurrent state both satisfy
@@ -120,8 +120,19 @@ class ET_EXPERIMENTAL Executor {
120120
const SamplingParams& params,
121121
std::optional<std::uint64_t> seed) = 0;
122122

123+
// Metadata-only admission check on the engine thread. False rejects just
124+
// this generation before any tasks are queued. Raw tokens bypass this check.
125+
virtual bool accepts(const PreparedInput& /*input*/) const {
126+
return false;
127+
}
128+
123129
// Run one batch. `out.outputs` is resized to batch.inputs.size() and filled
124-
// position-wise: outputs[i] answers inputs[i].
130+
// position-wise: outputs[i] answers inputs[i]. Opaque backing must be
131+
// compatible with this executor and have stable logical size and layout.
132+
// Lazy cache mutation is engine-thread-only and must preserve logical
133+
// contents. Backing can outlive this executor and be released on another
134+
// thread, so its destruction must be thread-safe and must not depend on this
135+
// executor's life.
125136
//
126137
// The batch arrives shaped as the scheduler packed it, and every input must
127138
// be answered. An implementation whose model needs static shapes pads or
@@ -143,7 +154,9 @@ class ET_EXPERIMENTAL Executor {
143154
//
144155
// A session may appear in more than one input of a batch when consecutive
145156
// prefill chunks of its prompt land together. They arrive in order, with
146-
// contiguous ranges, and at most one has produce_output set.
157+
// contiguous ranges, and at most one has produce_output set. Their payload
158+
// alternatives may differ: raw pending-token prefill can immediately precede
159+
// prepared chunks for the same session, including within one batch.
147160
//
148161
// false = the batch failed as a whole; there is no partial success. The
149162
// runner completes every task in it as Failed and poisons their sessions,

‎extension/llm/batching/metrics.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ struct ET_EXPERIMENTAL GenerationMetrics {
7070
MetricsTime t_first_token{};
7171
MetricsTime t_end{};
7272

73+
// Input counts are decoder positions, which may include expanded image rows.
7374
std::int64_t n_prompt_tokens = 0;
7475
// Successfully executed initial-input tokens, including a carried pending
7576
// token, excluding reused context and later generated-token inputs. Counts

‎extension/llm/batching/module_executor.cpp‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -85,9 +85,10 @@ Result<ModuleExecutor::Step> ModuleExecutor::build_step(
8585
return Error::InvalidArgument;
8686
}
8787
const std::int32_t seq_id = seq_it->second.seq_id;
88-
if (input.size == 0 || !input.tokens ||
89-
input.offset > input.tokens->size() ||
90-
input.size > input.tokens->size() - input.offset) {
88+
const auto* tokens = std::get_if<TokenInputPtr>(&input.payload);
89+
const auto* token_input = tokens && *tokens ? tokens->get() : nullptr;
90+
if (input.size == 0 || !token_input || input.offset > token_input->size() ||
91+
input.size > token_input->size() - input.offset) {
9192
ET_LOG(
9293
Error,
9394
"build_step: session %" PRId64 " gave a slice its tokens do not hold",
@@ -145,7 +146,7 @@ Result<ModuleExecutor::Step> ModuleExecutor::build_step(
145146
return Error::OutOfResources;
146147
}
147148

148-
const Token* slice = input.tokens->data() + input.offset;
149+
const Token* slice = token_input->data() + input.offset;
149150
for (std::size_t k = 0; k < input.size; ++k) {
150151
step.tokens.push_back(static_cast<std::int64_t>(slice[k]));
151152
}

‎extension/llm/batching/runner.cpp‎

Lines changed: 79 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
#include <condition_variable>
1616
#include <cstddef>
1717
#include <exception>
18+
#include <iterator>
1819
#include <limits>
1920
#include <mutex>
2021
#include <thread>
@@ -302,7 +303,7 @@ class RunnerImpl : public std::enable_shared_from_this<RunnerImpl> {
302303
void request_close(SessionId session) noexcept;
303304
GenerationHandle generate_async(
304305
SessionId session,
305-
std::vector<Token> delta,
306+
GenerationInput delta,
306307
GenConfig config,
307308
GenerationCallback on_update,
308309
std::function<void()> on_settled);
@@ -337,7 +338,8 @@ class RunnerImpl : public std::enable_shared_from_this<RunnerImpl> {
337338
// admission. Scheduler tasks own the delta after submission.
338339
struct GenerationRequest {
339340
SessionId session = 0;
340-
std::shared_ptr<const std::vector<Token>> delta;
341+
InputPayload delta;
342+
std::size_t size = 0;
341343
SamplingParams sampling;
342344
std::optional<std::uint64_t> seed;
343345
Generation generation;
@@ -453,21 +455,24 @@ class RunnerImpl : public std::enable_shared_from_this<RunnerImpl> {
453455
std::optional<TerminalOutcome> validate_generation_start_(
454456
const GenerationRequest& request,
455457
const SessionRecord& record) const;
456-
std::shared_ptr<const std::vector<Token>> build_initial_delta_(
458+
// Merge a pending token into raw input, or return it as a separate prefix.
459+
std::optional<Token> carry_pending_(
457460
GenerationRequest& request,
458461
const SessionRecord& record) const;
459462

460-
// Split `tokens` into chunks, the last of which produces output.
463+
// Split `payload` into chunks, the last of which may produce output.
461464
//
462465
// `is_continuation` distinguishes what the executor handed back from the
463466
// caller's opening delta. Only a continuation of exactly one token is a
464467
// decode: an opening delta is prefill however short it is, and a longer
465468
// continuation is re-fed as prefill too.
466469
std::vector<Task> create_tasks_(
467470
SessionId session,
468-
std::shared_ptr<const std::vector<Token>> tokens,
471+
InputPayload payload,
472+
std::size_t size,
469473
Position position,
470-
bool is_continuation);
474+
bool is_continuation,
475+
bool produce_output = true);
471476
InterpretedOutput interpret_output_(
472477
const Generation& generation,
473478
const Output& output) const;
@@ -619,7 +624,7 @@ Session::make_clone_request(Position upto) const {
619624
}
620625

621626
GenerationHandle Session::generate_async(
622-
std::vector<Token> delta,
627+
GenerationInput delta,
623628
GenConfig config,
624629
GenerationCallback on_update,
625630
std::function<void()> on_settled) const {
@@ -1307,15 +1312,22 @@ void RunnerImpl::request_close(SessionId session) noexcept {
13071312

13081313
GenerationHandle RunnerImpl::generate_async(
13091314
SessionId session,
1310-
std::vector<Token> delta,
1315+
GenerationInput delta,
13111316
GenConfig config,
13121317
GenerationCallback on_update,
13131318
std::function<void()> on_settled) {
13141319
auto state = std::make_shared<GenerationHandleState>();
13151320
state->on_settled = std::move(on_settled);
13161321
GenerationRequest request;
13171322
request.session = session;
1318-
request.delta = std::make_shared<const std::vector<Token>>(std::move(delta));
1323+
request.delta = std::visit(
1324+
Overloaded{
1325+
[](std::vector<Token> tokens) -> InputPayload {
1326+
return std::make_shared<const std::vector<Token>>(
1327+
std::move(tokens));
1328+
},
1329+
[](PreparedInputPtr prepared) -> InputPayload { return prepared; }},
1330+
std::move(delta));
13191331
request.seed = config.seed;
13201332
request.sampling = std::move(config.sampling);
13211333
request.generation.remaining_tokens = config.max_new_tokens;
@@ -1353,7 +1365,7 @@ std::optional<TerminalOutcome> RunnerImpl::validate_generation_start_(
13531365
if (request.generation.remaining_tokens <= 0) {
13541366
return TerminalOutcome::failed("max_new_tokens must be greater than zero");
13551367
}
1356-
if (!request.delta || request.delta->empty()) {
1368+
if (request.size == 0) {
13571369
return TerminalOutcome::failed("generation delta must not be empty");
13581370
}
13591371

@@ -1364,7 +1376,7 @@ std::optional<TerminalOutcome> RunnerImpl::validate_generation_start_(
13641376
const auto room = static_cast<std::size_t>(
13651377
std::numeric_limits<Position>::max() - start_position);
13661378
const auto pending_tokens = record.pending ? 1u : 0u;
1367-
if (pending_tokens > room || request.delta->size() > room - pending_tokens) {
1379+
if (pending_tokens > room || request.size > room - pending_tokens) {
13681380
return TerminalOutcome::failed(
13691381
"generation delta exceeds the session position range");
13701382
}
@@ -1375,22 +1387,33 @@ std::optional<TerminalOutcome> RunnerImpl::validate_generation_start_(
13751387
if (record.active_generation) {
13761388
return TerminalOutcome::failed("session already has an active generation");
13771389
}
1390+
if (const auto* prepared = std::get_if<PreparedInputPtr>(&request.delta);
1391+
prepared && !executor_.accepts(**prepared)) {
1392+
return TerminalOutcome::failed("executor does not accept prepared input");
1393+
}
13781394
return std::nullopt;
13791395
}
13801396

1381-
std::shared_ptr<const std::vector<Token>> RunnerImpl::build_initial_delta_(
1397+
std::optional<Token> RunnerImpl::carry_pending_(
13821398
GenerationRequest& request,
13831399
const SessionRecord& record) const {
1384-
auto delta = std::move(request.delta);
13851400
if (!record.pending) {
1386-
return delta;
1401+
return std::nullopt;
1402+
}
1403+
const auto* tokens = std::get_if<TokenInputPtr>(&request.delta);
1404+
if (!tokens) {
1405+
// Opaque backing cannot be prepended to: send a separate raw prefill.
1406+
return record.pending;
13871407
}
13881408

13891409
std::vector<Token> carried;
1390-
carried.reserve(delta->size() + 1);
1410+
carried.reserve((*tokens)->size() + 1);
13911411
carried.push_back(*record.pending);
1392-
carried.insert(carried.end(), delta->begin(), delta->end());
1393-
return std::make_shared<const std::vector<Token>>(std::move(carried));
1412+
carried.insert(carried.end(), (*tokens)->begin(), (*tokens)->end());
1413+
++request.size;
1414+
request.delta =
1415+
std::make_shared<const std::vector<Token>>(std::move(carried));
1416+
return std::nullopt;
13941417
}
13951418

13961419
void RunnerImpl::start_generation_(GenerationRequest request) {
@@ -1415,20 +1438,23 @@ void RunnerImpl::start_generation_(GenerationRequest request) {
14151438
return;
14161439
}
14171440
auto& record = session->second;
1441+
request.size = std::visit(
1442+
[](const auto& payload) -> std::size_t {
1443+
return payload ? payload->size() : 0;
1444+
},
1445+
request.delta);
14181446
if (auto rejection = validate_generation_start_(request, record)) {
14191447
complete_request_(
14201448
std::move(request), std::move(*rejection), /*on_engine_thread=*/true);
14211449
return;
14221450
}
14231451

1424-
const auto start_position = record.position();
1425-
// The caller's own delta, captured before build_initial_delta_ moves it and
1426-
// before any carried token is prepended. The generation tier counts what
1427-
// callers gave; tokens actually fed to the model are EngineMetrics'
1428-
// model_input_tokens().
1452+
auto start_position = record.position();
1453+
// Count the caller's own delta before adding a carried token. EngineMetrics'
1454+
// model_input_tokens() counts all positions actually fed to the model.
14291455
request.generation.m.n_prompt_tokens =
1430-
static_cast<std::int64_t>(request.delta->size());
1431-
auto delta = build_initial_delta_(request, record);
1456+
static_cast<std::int64_t>(request.size);
1457+
const auto prefix = carry_pending_(request, record);
14321458
if (!is_running_() || !record.status->open.load(std::memory_order_acquire) ||
14331459
request.generation.state->cancelled.load()) {
14341460
complete_request_(
@@ -1458,22 +1484,38 @@ void RunnerImpl::start_generation_(GenerationRequest request) {
14581484
return;
14591485
}
14601486

1461-
// advance() clears a carried token only after the input holding it runs.
1462-
submit_(
1487+
std::vector<Task> tasks;
1488+
if (prefix) {
1489+
tasks = create_tasks_(
1490+
request.session,
1491+
std::make_shared<const std::vector<Token>>(1, *prefix),
1492+
1,
1493+
start_position++,
1494+
/*is_continuation=*/false,
1495+
/*produce_output=*/false);
1496+
}
1497+
auto suffix = create_tasks_(
14631498
request.session,
1464-
create_tasks_(
1465-
request.session,
1466-
std::move(delta),
1467-
start_position,
1468-
/*is_continuation=*/false));
1499+
std::move(request.delta),
1500+
request.size,
1501+
start_position,
1502+
/*is_continuation=*/false);
1503+
tasks.insert(
1504+
tasks.end(),
1505+
std::make_move_iterator(suffix.begin()),
1506+
std::make_move_iterator(suffix.end()));
1507+
// advance() clears a carried token only after the input holding it runs.
1508+
submit_(request.session, std::move(tasks));
14691509
}
14701510

14711511
std::vector<Task> RunnerImpl::create_tasks_(
14721512
SessionId session,
1473-
std::shared_ptr<const std::vector<Token>> tokens,
1513+
InputPayload payload,
1514+
std::size_t size,
14741515
Position position,
1475-
bool is_continuation) {
1476-
const auto total = static_cast<std::int32_t>(tokens->size());
1516+
bool is_continuation,
1517+
bool produce_output) {
1518+
const auto total = static_cast<std::int32_t>(size);
14771519
// The scheduler owns this limit, so the runner cannot split a prompt into
14781520
// chunks the scheduler would then refuse. It is non-zero and clamped to the
14791521
// token count, so the loop below always advances.
@@ -1495,10 +1537,10 @@ std::vector<Task> RunnerImpl::create_tasks_(
14951537
t.cancelled = false;
14961538
t.is_decode = decode;
14971539
t.input.sid = session;
1498-
t.input.produce_output = last;
1540+
t.input.produce_output = produce_output && last;
14991541
t.input.offset = static_cast<size_t>(i);
15001542
t.input.size = static_cast<size_t>(n);
1501-
t.input.tokens = tokens;
1543+
t.input.payload = payload;
15021544
t.input.position = position;
15031545
tasks.push_back(std::move(t));
15041546
}
@@ -1707,6 +1749,7 @@ void RunnerImpl::resume_generation_(
17071749
create_tasks_(
17081750
session_id,
17091751
std::move(input),
1752+
1,
17101753
position,
17111754
/*is_continuation=*/true));
17121755
}

0 commit comments

Comments
 (0)