feat(dspy): support portable and provider-native compaction

[isaacbmiller]

📝 Changes Description

This PR gives DSPy one compacted-history shape:

History(
    compaction: str | LMCompactionPart | None,
    messages=[...recent signature-shaped turns...],
)

compaction is top-level state, so History's normal signature-field filtering cannot discard it. Its two values preserve the distinction between portable summaries and provider-owned continuation state:

Portable text compaction

history = dspy.History(
    compaction="The user chose PostgreSQL and migrations are pending.",
    messages=[...recent signature-shaped turns...],
)

Adapters render a string before retained turns as ordinary user context:

Here is a summary of the conversation to date:

{compaction}

This follows the common non-native scheme of replacing an old prefix with a user summary and retaining a recent suffix. It can produce adjacent user messages when the retained suffix begins with a user turn; providers and frameworks normally merge or accept that shape.

Provider-native compaction

Provider compaction is returned as LMCompactionPart on typed LM responses and can be placed directly into History:

lm = dspy.LM(
    "openai/gpt-5.3-codex",
    model_type="responses",
    context_management=[
        {"type": "compaction", "compact_threshold": 200_000},
    ],
)

with dspy.context(experimental=True):
    first = lm(dspy.User("Start the task."))

history = dspy.History(
    compaction=first.compactions[-1],
    messages=[...recent signature-shaped turns...],
)

The adapter inserts the typed part at the compacted-history boundary. The LM client then restores the provider's actual wire representation:

• OpenAI's opaque, role-less Responses compaction item remains encrypted provider data.
• Anthropic's readable compaction block remains assistant content.
• Cross-provider replay is rejected rather than silently converting provider state to text.

The same OpenAI-shaped context_management=[...] setting works for both providers: LiteLLM maps it to Anthropic's native compact_20260112 edit.

Additional behavior:

• Typed responses, serialized LMResponses, History, and LMHistoryEntry.messages preserve provider compaction losslessly.
• Legacy string outputs omit provider-only compaction parts.
• inspect_history() displays readable summaries and labels opaque state without printing encrypted content.
• History(compaction=None, messages=...) keeps the previous serialized shape.
• This PR defines representation and replay, not an automatic _compact_if_needed policy.

Why one union field?

Both values occupy the same semantic slot: a replacement for the omitted history prefix. A union makes it impossible to set a portable summary and native state simultaneously.

They intentionally render differently. A string is model-readable, provider-independent application context. An LMCompactionPart is provider-owned state and may be opaque. Converting a string into LMCompactionPart would invent a provider identity; flattening native state into a string would lose or leak OpenAI's encrypted payload.

Ownership remains separated:

• History stores the compacted prefix and retained signature-shaped suffix.
• Adapters place the compacted prefix relative to recent turns.
• LM clients serialize native parts and enforce provider/endpoint compatibility.

References

• OpenAI compaction guide: compaction is an opaque standalone Responses item; append it as provider output or use the compacted output as the next canonical context window.
• Anthropic compaction guide: compaction is a readable block at the start of an assistant response and must be passed back in that assistant content.
• LiteLLM context-management guide: maps the common OpenAI compaction setting to Anthropic's edit shape and preserves returned compaction blocks for subsequent turns.
• LangChain summarization middleware: replaces the summarized prefix with HumanMessage("Here is a summary...") followed by preserved recent messages.
• Pydantic AI message history: history processors replace old history with summarized messages plus a retained suffix; provider-valid history remains typed and serializable.
• Pydantic AI CompactionPart: precedent for retaining provider-specific compaction as a typed history part rather than flattening it into ordinary text.

✅ Contributor Checklist

• Pre-Commit checks are passing locally
• Title follows the required format
• Commit messages follow {label}(dspy): {message}

🧪 Validation

• Live OpenAI Responses smoke test with openai/gpt-5.3-codex: the first call emitted opaque native compaction, the latest LMCompactionPart was assigned to History.compaction, and a subsequent Predict call recovered the compacted codeword COBALT. The encrypted payload's SHA-256 matched between the first response and second provider request, proving lossless replay.
• uv run --frozen pytest --deno tests/adapters -q — 280 passed
• uv run --frozen pytest --deno tests/core/test_types.py tests/clients/test_inspect_global_history.py tests/clients/test_disk_serialization.py -q — 62 passed
• uv run --frozen pytest --deno tests/clients/test_lm.py -q — 83 passed, 8 skipped
• Ruff lint on all touched files
• git diff --check

⚠️ Warnings

The typed direct-call API used to obtain and continue an LMResponse is experimental. Provider compaction is enabled through the upstream context_management API.