You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c89046c
Browse filesBrowse the repository at this point in the historyBrowse files
Expose committed session cloning and build prefix reuse above it (pytorch#22812)
Expose `Session::clone_async(upto)` for independently writable committed
prefixes. A caller-owned `PrefixCache` handles token matching and LRU
eviction; `capture_prompt(...).wrap(...)` requests the prompt clone
before the first user callback, and caller-thread `collect()` inserts it
after generation. Capture binds to session identity and survives moves
without retaining source ownership. Lookup tries each snapshot only at
its longest matching boundary. The MLX example uses this helper and
forwards the final prompt token for fresh logits. The cache factory API
stays unchanged; session limits are checked through the constructed
cache's `BatchControl::max_seqs()`.
Review the Session/Runner lifecycle first, then ModuleExecutor/backend
clone contracts, prefix policy, and MLX integration.
**214 shared native tests pass on Apple M1 Pro**, including 14 capture
tests for timing, reuse, failures, handle lifetime, and
moved/reassigned/destroyed sources. All five native targets and the
standalone MLX batched runner build; lint passes. GPU execution was not
repeated for this lifetime change; the following GPU/model results are
from `ddf92b40d1`.
All 49 MLX tests pass. Gemma 3 1B validation covers BF16/FP16,
full/selected logits, both layouts, physical ring wrap, branches,
closure, and concurrency: 16 smoke groups, 88 cache/work checks, and 44
cold export comparisons pass. Warm repeats forward 1 of 23/1280 prompt
tokens. **Normal cold/warm initial logits exceed tolerance in 72/88
cases.** The public header recommends opt-in greedy experiments until
non-greedy sampling is validated; even greedy token parity is not
guaranteed. Clone controls with matching schedule/geometry are
byte-exact. A 23-token control also gives byte-exact
cold/split/replay/clone logits in both dtypes with one-token prefill.
Isolated probes identify shape-dependent MLX projection and attention
arithmetic; precision promotions did not resolve the normal cold/warm
gap. The four existing exports were reused and hash-verified.
Authored with OpenAI Codex.
0 commit comments