feat(gepa): make InstructionProposer the default instruction proposer by dbreunig · Pull Request #10555 · stanfordnlp/dspy

dbreunig · 2026-10-02T02:42:52Z

Changes

GEPA now builds InstructionProposer() by default. The proposer uses dspy.Predict with a JSONAdapter and preserves GEPA's reflection instructions in the ProposeInstruction signature.

Reflective examples remain structured dictionaries through Predict. The adapter renders histories and multimodal values, including nested images. Custom proposers retain their existing reflective dataset format. The adapter and proposer now have a one-way import dependency.

The built-in proposer supports these options:

skills accepts a single source or a sequence of paths, skill directories, or inline strings. YAML frontmatter is parsed with PyYAML, including multiline descriptions. Invalid metadata and empty skill bodies raise at construction.
additional_instructions adds guidance to every proposal.
base_instructions replaces the reflection instructions while retaining the fields.
max_chars sets an exact character limit after stripping outer whitespace. Pydantic validates the draft and allows one compression attempt if it is too long. An empty or still oversized compression raises ValidationError. Validation runs after adapter parsing and also applies to custom adapters. Instructions are never truncated.
truncate_history_outputs shortens long tool and REPL outputs before adapter rendering without mutating the original histories.
adapter selects the adapter used for proposal calls; the default is JSONAdapter().

max_chars replaces the earlier word/token options. There is no tokenizer selection or estimated token count. Exceptions propagate to GEPA for proposal failure handling.

MultiModalInstructionProposer remains available. Its TODO links to #10591 for a vision-quality comparison before any deprecation decision. Structured system-prompt fields, the proposal journal, lesson distillation, and changes to the Flex code proposer remain out of scope.

The docs describe the new API and omit explicit temperature settings from GPT examples.

Validation

211 tests passed against the pinned GEPA 0.1.4:

python3 -m pytest tests/teleprompt/test_gepa.py tests/teleprompt/test_gepa_builtin_instruction_proposer.py tests/teleprompt/test_gepa_instruction_proposer.py tests/flex -q --disable-warnings

Coverage includes character limits and failed compression, scalar skill inputs, YAML parsing, structured examples and nested images, history truncation, and custom proposer compatibility. Recorded reflection prompts were updated; student LM records and recorded responses are unchanged.

The dependency lockfile check and git diff --check pass. Changed Python files introduce no new Ruff findings. These are mocked-LM behavior checks, not an optimization-quality evaluation.

Behavior change

The default reflection request changes from the gepa library's raw prompt and backtick extraction to a system message plus structured input fields and JSON output. Optimization results may differ from earlier releases even with the same seed. The reflection instruction paragraphs remain unchanged.

dspy.GEPA now builds InstructionProposer() when no instruction_proposer is passed. The class keeps GEPA's stock reflection wording as a dspy.Signature and calls the reflection LM through dspy.Predict with a JSONAdapter instead of a raw template and backtick extraction, so dspy.Image and other dspy.Type inputs reach the reflection LM as structured content. InstructionProposer is configurable: reference skills (files, directories with SKILL.md, or inline text), additional instructions, a replacement base prompt, word and token caps enforced with one compression call (never truncation), compaction of long tool results in dspy.History and REPLHistory inputs, and an adapter override. The bridge hands History and REPLHistory objects to the built-in proposer only; custom proposers receive the same reflective dataset as before. The stock gepa InstructionProposalSignature path is removed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RD3qR4dgFVoU82uc7J9WD

greptile-apps · 2026-10-02T02:49:08Z

Confidence Score: 5/5

[Medium risk] Replaces the default instruction proposer in GEPA optimization.

No outstanding findings block merging.

Summary

This PR makes InstructionProposer GEPA’s default and passes reflection examples as structured values rather than markdown text. It adds configurable skills, guidance, character limits, and history-output truncation, alongside updates to the GEPA bridge, tests, documentation, and dependency declaration.

_{Reviews (4) · Last reviewed commit: "fix(gepa): address instruction proposer ..." · Reviewed by Greptile}

…ith spaces Address two review findings on the built-in InstructionProposer. Length caps: when both a word and a token cap are set, a compressed draft that shortened the over-cap unit but left the other, already compliant, unit unchanged was discarded. The proposer now keeps a still-over compression as long as it makes progress on an over-cap unit and does not push any unit over, or further over, its cap. Skills: a missing skill path containing whitespace, such as "./skills/my skill", was loaded as inline text and sent to the reflection LM verbatim. A single-line string now counts as a path when it starts like one, ends with a skill file suffix, or names an entry inside an existing directory, even when it contains spaces. Multi-line text is always inline content. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAqy7HASnTVLve3QPBDwmG

…skill text The previous fix made any single line ending in .md, .markdown, or .txt a path, so inline guidance such as "Consult the file report.txt" raised FileNotFoundError at construction. A single line containing whitespace now counts as a path only when it starts like one (./, ../, ~/, /, or a drive letter) or names an entry inside a directory that exists. The suffix rule still applies to strings without whitespace. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAqy7HASnTVLve3QPBDwmG

isaacbmiller

API/design review and reproduced skill-loading edge cases below. Targeted GEPA/proposer and Flex tests passed locally: 152 passed, 39 skipped (mocked-LM verification, not a quality evaluation).

The structured system-prompt-field use case is recorded in #9829 (comment); this PR need not implement that larger feature. Multimodal comparison/deprecation evaluation is tracked separately in #10591.

isaacbmiller · 2026-10-07T17:13:00Z

+    Read all the assistant responses and the corresponding feedback. Identify all niche and domain specific factual information about the task and include it in the instruction, as a lot of it may not be available to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, if so, include that in the instruction as well."""
+
+    current_instruction: str = dspy.InputField(desc="The instructions I provided to the assistant.")
+    examples_with_feedback: str = dspy.InputField(


Design feedback: could this stay a structured collection of reflective examples instead of a str? The annotation matches the implementation because _render_examples flattens the mappings into Markdown before Predict, but that is the design choice I am questioning. We already have structured inputs, outputs, and feedback; ideally the signature represents that structure and the adapter owns rendering. Is there a specific limitation requiring manual rendering here, including for History and multimodal values?

Updated in 9ec7673. examples_with_feedback is now list[dict[str, Any]], and the proposer passes structured values to Predict. The adapter renders History, REPLHistory, and multimodal values. The bridge also preserves nested inputs and outputs for the built-in proposer; custom proposers keep their existing dataset format. Tests cover structured values reaching the adapter and nested images reaching the LM as image parts. The targeted GEPA, proposer, and Flex suite passed 211 tests against the pinned GEPA 0.1.4. This verifies behavior with mocked LMs, not optimization quality.

isaacbmiller · 2026-10-07T17:13:00Z

        self.rng = rng or random.Random(0)
        self.reflection_lm = reflection_lm
+
+        from dspy.teleprompt.gepa.instruction_proposal import InstructionProposer  # it imports this module


Design feedback: can we remove the circular dependency rather than work around it with an inline import? instruction_proposal imports ReflectiveExample and format_history_for_reflection from this module, while this module now needs InstructionProposer. The deferred import works, but the shared types/rendering and adapter/proposer ownership should have a cleaner dependency direction. Please revisit where these responsibilities belong without introducing a larger abstraction just for this.

Removed the cycle in 9ec7673. ReflectiveExample now lives with the proposer and remains importable from gepa_utils. The legacy history formatter stays in gepa_utils for custom proposers. instruction_proposal no longer imports gepa_utils, so the adapter can import InstructionProposer at module scope. No additional module or abstraction was needed.

isaacbmiller · 2026-10-07T17:13:00Z

+        skills: Sequence[str | Path] | None = None,
+        additional_instructions: str | None = None,
+        base_instructions: str | None = None,
+        max_instruction_words: int | None = None,


Do we need both words and tokens? I would prefer a single token-budget option. Words are not the unit the model/context budget operates in, and having both adds API surface and multi-constraint compression logic without a clear separate use case.

Simplified this to one option, max_chars, in 9ec7673. I chose characters so the limit has one exact definition regardless of the student or reflection model and can be validated with Pydantic. It counts Unicode characters after stripping outer whitespace. Both word/token options and the multi-constraint logic are removed. This is an instruction length limit, not a model context-token budget.

isaacbmiller · 2026-10-07T17:13:00Z

+        base_instructions: str | None = None,
+        max_instruction_words: int | None = None,
+        max_instruction_tokens: int | None = None,
+        compaction: bool = False,


Naming: compaction suggests summarizing history for continued use in the same conversation to me. This actually truncates tool/REPL outputs for reflection. Could we name it for that behavior, e.g. truncate_history_outputs?

Renamed the option to truncate_history_outputs in 9ec7673, including the examples and tests. It still shortens only tool/REPL outputs and leaves the original history unchanged.

isaacbmiller · 2026-10-07T17:13:00Z

+                    _describe(shortened_size),
+                    _describe(caps),
+                )
+                return shortened


A parameter named max_instruction_tokens should not return an oversized instruction with only a warning. Can we validate after compression and fail if it still exceeds the budget, allowing GEPA to apply its retry/skip behavior? Otherwise this should be named suggested_max_tokens. I initially wondered about a Pydantic field constraint, but max_length counts characters, not tokens; this checkout also drops top-level field constraints during adapter parsing/schema construction. A tokenizer-aware check is needed, and structured outputs alone will not enforce it.

Fixed in 9ec7673 with the new max_chars contract. A Pydantic TypeAdapter with StringConstraints(max_length=max_chars) validates the draft, then the result of at most one compression call. An empty or still oversized compression raises ValidationError for GEPA to handle. Validation runs explicitly inside the proposer after adapter parsing, so it does not depend on the top-level constraints that adapters currently drop. Tests cover JSONAdapter and a custom ChatAdapter.

isaacbmiller · 2026-10-07T17:13:00Z

+
+        return litellm.token_counter(model=model, text=text)
+    except Exception:
+        return max(1, round(len(text) / 4))  # rough fallback: ~4 characters per token


Related to the maximum contract: which tokenizer should define this budget? Currently it uses the reflection LM, while the instruction is consumed by the student LM, which can differ. Please make that contract explicit. If this remains a hard maximum, silently estimating characters / 4 after token-counter failure cannot establish that the maximum was honored.

Removed token counting and the characters/4 fallback in 9ec7673. The only limit is now max_chars, measured as Unicode characters after stripping outer whitespace and enforced by Pydantic. It makes no claim about either model's token budget, so there is no tokenizer selection or estimation fallback.

isaacbmiller · 2026-10-07T17:13:00Z

+        if adapter is not None and not isinstance(adapter, Adapter):
+            raise TypeError(f"adapter must be a dspy.Adapter or None, got {type(adapter).__name__}.")
+
+        self.skills: tuple[_Skill, ...] = tuple(_SkillLoader.load(skill) for skill in (skills or ()))


Please guard against scalar skill sources before iterating. Reproduced: InstructionProposer(skills="abc") silently loads three separate skills, a, b, and c. A bare path string is similarly processed character by character. Either normalize scalar str/Path values into a single source or reject them clearly, with a regression test.

Fixed in 9ec7673. A scalar str or Path is normalized into one skill source before iteration. Regression tests cover inline text, string paths, Path objects, and an empty scalar string.

isaacbmiller · 2026-10-07T17:13:00Z

+            raise ValueError("Empty skill content.")
+
+        meta, content = cls._parse_frontmatter(text)
+        content = content.strip()


Please check for empty content after stripping frontmatter too. Reproduced: InstructionProposer(skills=["---\nname: empty\n---"]) succeeds with an empty skill body, despite the documented empty-content rejection.

Fixed in 9ec7673. The loader checks for empty content again after removing frontmatter. Regression tests cover frontmatter-only inline content and skill files, including trailing whitespace.

isaacbmiller · 2026-10-07T17:13:00Z

+                    if raw.startswith((" ", "\t")) or ":" not in raw:
+                        continue
+                    key, _, value = raw.partition(":")
+                    meta[key.strip()] = value.strip().strip("'\"")


The public API/docs promise YAML frontmatter, but this is a flat line parser. For example, description: | followed by indented description lines produces the literal description | and discards its text. Please use proper YAML parsing or explicitly narrow the supported format and reject unsupported constructs rather than silently misreading them. Add coverage for multiline descriptions.

Replaced the flat parser with yaml.safe_load and declared PyYAML as a direct dependency in 9ec7673. Tests cover literal and folded multiline descriptions, including an indented --- inside a description. Invalid YAML, non-mapping metadata, non-string name/description values, and an unclosed frontmatter block raise instead of being silently misread.

isaacbmiller · 2026-10-07T17:13:00Z

        return multimodal_content


+# TODO: InstructionProposer now delivers images to the reflection LM as structured content.


Tracked the comparison and possible deprecation follow-up in #10591. Please link that issue from this TODO. We should compare quality on a vision task before treating structured image transport as sufficient to replace the older proposer.

Linked #10591 from the TODO in 9ec7673. MultiModalInstructionProposer remains available. The vision-quality comparison and any deprecation decision remain in that follow-up; the structured-image tests do not establish equivalent optimization quality.

isaacbmiller

LGTM. Reviewed the updated structured examples, dependency direction, skill loading, and strict max_chars contract. The remaining wrapper/dataset distinction is accepted and tracked in #10592; multimodal evaluation is tracked in #10591. Targeted local tests: 172 passed, 39 skipped. Core CI is green; the latest-dependency job has failures in LM-client/MCP/server-setup tests outside the changed proposer code.

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread dspy/teleprompt/gepa/instruction_proposal.py Outdated

Comment thread dspy/teleprompt/gepa/instruction_proposal.py

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread dspy/teleprompt/gepa/instruction_proposal.py Outdated

This was referenced Oct 7, 2026

[Feature] Add append_instructions to Signature #9829

Closed

Evaluate MultiModalInstructionProposer against the new built-in proposer before deprecation #10591

Open

isaacbmiller reviewed Oct 7, 2026

View reviewed changes

fix(gepa): address instruction proposer review feedback

9ec7673

isaacbmiller mentioned this pull request Oct 7, 2026

Document GEPA proposer dataset differences for built-ins and forwarding wrappers #10592

Open

isaacbmiller approved these changes Oct 7, 2026

View reviewed changes

isaacbmiller merged commit b6c8aaf into stanfordnlp:main Oct 7, 2026
25 of 26 checks passed

This was referenced Oct 7, 2026

fix(tests): support latest LiteLLM and MCP fixtures #10593

Merged

fix(adapters): serialize non-JSON values one at a time in serialize_for_json #10634

Merged

		return multimodal_content


		# TODO: InstructionProposer now delivers images to the reflection LM as structured content.

Conversation

dbreunig commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Validation

Behavior change

Uh oh!

greptile-apps Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

isaacbmiller left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isaacbmiller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dbreunig commented Oct 2, 2026 •

edited

Loading

greptile-apps Bot commented Oct 2, 2026 •

edited

Loading