Support a larger context window #1416

Rewritten after @barhanc's reply. The original text asked for a specific
option on the common/runner / LLMController path; that was the wrong
target, since that API is deprecated and the real ceiling is the exported
model. The underlying need is unchanged, so here it is stated as the outcome
rather than as an API shape.

What we are asking for

The ability to run with a meaningfully larger context window than is possible
today — not a particular API. Whatever form it takes (models exported with a
longer context, a documented path for exporting our own, or a runtime option
where one is feasible), the outcome we need is more usable prompt tokens on
device.

This is deliberately not scoped to the current architecture. If it only ever
lands in 0.10+, that is fine by us.

Why it matters to us

Private Mind does
on-device RAG and document chat. Today we budget 2048 tokens per model and
reserve 768 of those for generation, which leaves roughly 1280 tokens of
usable prompt. Retrieved passages, a system prompt and conversation history do
not fit together in that budget, so the prompt is truncated well before the
model's own capability is reached. It is the single biggest constraint on that
feature.

To be straight about our own side: that 2048 is a constant in our app, and we do
not currently read the runtime value at all. Raising our constant is ours to do
— but it only buys anything up to whatever the exported model actually supports,
and that ceiling is what this issue is about.

What would help

1. Models exported with a larger context window. This is the main ask. We
   saw the note about experimenting with roughly 10x context and would be glad
   to test those builds against a real RAG workload on a range of devices.

2. Publish the context length each shipped model was exported with. This is
   the small, cheap half. Right now the number is not stated on the Hugging Face
   model cards, so an application either queries it at runtime or guesses. We
   guessed, and picked 2048 for every family. Putting the exported context
   length next to each model — in the model card or in the exported model
   constants — would let apps size their prompt budget correctly without
   discovering it by experiment.

On sliding-window strategies

Agreed that they are fragile, and doubly so for RAG — dropping earlier turns
silently changes what the model was grounded on. We truncate rather than slide
for that reason, which is precisely why the raw budget is what binds for us.

Environment

• react-native-executorch 0.9.2
• iOS and Android