diff --git a/docs/docs/api/optimizers/GEPA/GEPA_Advanced.md b/docs/docs/api/optimizers/GEPA/GEPA_Advanced.md index 9e8572fe1e..64ccd15656 100644 --- a/docs/docs/api/optimizers/GEPA/GEPA_Advanced.md +++ b/docs/docs/api/optimizers/GEPA/GEPA_Advanced.md @@ -8,33 +8,25 @@ The `instruction_proposer` is the component responsible for invoking the `reflec ### Default Implementation -By default, GEPA uses the built-in instruction proposer from the [GEPA library](https://github.com/gepa-ai/gepa), which implements the [`ProposalFn`](https://github.com/gepa-ai/gepa/blob/main/src/gepa/core/adapter.py). The [default proposer](https://github.com/gepa-ai/gepa/blob/main/src/gepa/proposer/reflective_mutation/reflective_mutation.py#L53-L75) uses this prompt template: +By default, GEPA uses `InstructionProposer` from `dspy.teleprompt.gepa`. When you pass no `instruction_proposer`, GEPA builds `InstructionProposer()` with its defaults. The proposer passes reflective examples as a list of dictionaries to `dspy.Predict`. The adapter renders their inputs, outputs, and feedback, including history and multimodal values. The proposer asks the `reflection_lm` for a new instruction through `dspy.Predict` with a `JSONAdapter`. Inputs that are `dspy.Type` instances, such as `dspy.Image`, reach the reflection LM as structured content. -```` -I provided an assistant with the following instructions to perform a task for me: -``` - -``` +The default prompt is the `ProposeInstruction` signature. Its instructions are: -The following are examples of different task inputs provided to the assistant along with the assistant's response for each of them, and some feedback on how the assistant's response could be better: -``` - -``` +```` +I provided an assistant with instructions to perform a task for me. You are given those instructions, along with examples of different task inputs provided to the assistant, the assistant's response for each of them, and some feedback on how the assistant's response could be better. Your task is to write a new instruction for the assistant. Read the inputs carefully and identify the input format and infer detailed task description about the task I wish to solve with the assistant. Read all the assistant responses and the corresponding feedback. Identify all niche and domain specific factual information about the task and include it in the instruction, as a lot of it may not be available to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, if so, include that in the instruction as well. - -Provide the new instructions within ``` blocks. ```` -This template is automatically filled with: - -- ``: The current instruction being optimized -- ``: Structured markdown containing predictor inputs, generated outputs, and evaluation feedback +The signature has two input fields and one output field: +- `current_instruction`: The current instruction being optimized +- `examples_with_feedback`: A list of dictionaries containing predictor inputs, generated outputs, and evaluation feedback +- `new_instruction`: The proposed instruction Example of default behavior: @@ -42,22 +34,52 @@ Example of default behavior: # Default instruction proposer is used automatically gepa = dspy.GEPA( metric=my_metric, - reflection_lm=dspy.LM(model="gpt-5", temperature=1.0, max_tokens=32000, api_key=api_key), + reflection_lm=dspy.LM(model="gpt-5", max_tokens=32000, api_key=api_key), auto="medium" ) optimized_program = gepa.compile(student, trainset=examples) ``` +#### Configuring the default proposer + +Pass an `InstructionProposer(...)` instance to configure the default behavior. `skills`, `additional_instructions`, and `max_chars` each add one input field to the proposal prompt when set. `InstructionProposer()` sends only the two base fields. + +| Option | Effect | +|---|---| +| `skills` | Reference material shown to the reflection LM, as one source or a sequence of sources. Each entry is a path to a markdown or text file, a directory holding `SKILL.md` (the Agent Skills layout), or an inline string. A path that does not exist raises at construction. A leading YAML frontmatter block supplies `name` and `description`, including multiline descriptions. Empty bodies and invalid frontmatter raise at construction. | +| `additional_instructions` | Guidance applied to every proposal, such as "Write instructions in imperative voice." | +| `base_instructions` | Replaces the prompt text above. The input and output fields stay the same. | +| `max_chars` | Maximum Unicode characters in each proposed instruction after stripping outer whitespace. Pydantic validates this limit. `None` means no limit. | +| `truncate_history_outputs` | When `True`, long tool results inside `dspy.History` inputs (for example from `dspy.ReActV2`) and long outputs inside `REPLHistory` inputs are cut to 500 characters before the examples are rendered. Nothing else is shortened. | +| `adapter` | The adapter for the proposer's own LM calls. Defaults to `JSONAdapter()`. | + +When a proposal exceeds `max_chars`, the proposer makes one more call asking the reflection LM to shorten the draft. Pydantic validates the result and raises `ValidationError` if it is empty or still too long. The proposer never truncates an instruction. Validation runs after adapter parsing, so the limit is enforced with custom adapters too. + +```python +from dspy.teleprompt.gepa import InstructionProposer + +gepa = dspy.GEPA( + metric=my_metric, + reflection_lm=dspy.LM(model="gpt-5", max_tokens=32000, api_key=api_key), + instruction_proposer=InstructionProposer( + skills=["./skills/prompt-engineering", "./skills/prompt-engineering/models/openai.md"], + additional_instructions="Write instructions in imperative voice.", + max_chars=1500, + truncate_history_outputs=True, + ), + auto="medium", +) +``` + +Exceptions raised while proposing propagate to GEPA for proposal failure handling. + ### When to Use Custom instruction_proposer -**Note:** Custom instruction proposers are an advanced feature. Most users should start with the default proposer, which works well for most text-based optimization tasks. +**Note:** Custom instruction proposers are an advanced feature. Most users should start with the default proposer, which works well for most optimization tasks, and reach for its options (`skills`, `additional_instructions`, `base_instructions`, `max_chars`, and `truncate_history_outputs`) before writing their own. Consider implementing a custom instruction proposer when you need: -- **Multi-modal handling**: Process images (dspy.Image) alongside textual information in your inputs -- **Nuanced control on limits and length constraints**: Have more fine-grained control over instruction length, format, and structural requirements -- **Domain-specific information**: Inject specialized knowledge, terminology, or context that the default proposer lacks and cannot be provided via feedback_func. This is an advanced feature, and most users should not need to use this. -- **Provider-specific prompting guides**: Optimize instructions for specific LLM providers (OpenAI, Anthropic, etc.) with their unique formatting preferences +- **Nuanced control on format and structure**: Requirements on instruction format or structure that go beyond what `additional_instructions` and `max_chars` express - **Coupled component updates**: Handle situations where 2 or more components need to be updated together in a coordinated manner, rather than optimizing each component independently (refer to component_selector parameter, in [Custom Component Selection](#custom-component-selection) section, for related functionality) - **External knowledge integration**: Connect to databases, APIs, or knowledge bases during instruction generation @@ -65,22 +87,22 @@ Consider implementing a custom instruction proposer when you need: **Built-in Options:** -- **Default Proposer**: The standard GEPA instruction proposer (used when `instruction_proposer=None`). The default instruction proposer IS an instruction proposer as well! It is the most general one, that was used for the diverse experiments reported in the GEPA paper and tutorials. -- **MultiModalInstructionProposer**: Handles `dspy.Image` inputs and structured multimodal content. +- **InstructionProposer**: The default (used when `instruction_proposer=None`), configurable as shown above. It uses GEPA's standard reflection prompt, which was used for the diverse experiments reported in the GEPA paper and tutorials, and it sends `dspy.Image` inputs to the reflection LM as structured content. +- **MultiModalInstructionProposer**: An earlier proposer for `dspy.Image` inputs with its own vision-oriented prompt. The default proposer now handles images too, so this class is kept for comparison and may be deprecated. ```python from dspy.teleprompt.gepa.instruction_proposal import MultiModalInstructionProposer -# For tasks involving images or multimodal inputs +# A vision-specific prompt for tasks involving images gepa = dspy.GEPA( metric=my_metric, - reflection_lm=dspy.LM(model="gpt-5", temperature=1.0, max_tokens=32000, api_key=api_key), + reflection_lm=dspy.LM(model="gpt-5", max_tokens=32000, api_key=api_key), instruction_proposer=MultiModalInstructionProposer(), auto="medium" ) ``` -We invite community contributions of new instruction proposers for specialized domains as the [GEPA library](https://github.com/gepa-ai/gepa) continues to grow. +We invite community contributions of new instruction proposers for specialized domains. ### How to Implement Custom Instruction Proposers @@ -113,59 +135,17 @@ def custom_instruction_proposer(candidate, reflective_dataset, components_to_upd - `Generated_Outputs: dict[str, Any] | str` - Success: output fields dict, Failure: error message - `Feedback: str` - Always a string from metric function or auto-generated by GEPA -#### Basic Example: Word Limit Proposer - -```python -import dspy -from gepa.core.adapter import ProposalFn -from dspy.teleprompt.gepa.gepa_utils import ReflectiveExample - -class GenerateWordLimitedInstruction(dspy.Signature): - """Given a current instruction and feedback examples, generate an improved instruction with word limit constraints.""" - - current_instruction = dspy.InputField(desc="The current instruction that needs improvement") - feedback_summary = dspy.InputField(desc="Feedback from examples that might include both positive and negative cases") - max_words = dspy.InputField(desc="Maximum number of words allowed in the new instruction") - - improved_instruction = dspy.OutputField(desc="A new instruction that fixes the issues while staying under the max_words limit") - -class WordLimitProposer(ProposalFn): - def __init__(self, max_words: int = 1000): - self.max_words = max_words - self.instruction_improver = dspy.ChainOfThought(GenerateWordLimitedInstruction) - - def __call__(self, candidate: dict[str, str], reflective_dataset: dict[str, list[ReflectiveExample]], components_to_update: list[str]) -> dict[str, str]: - updated_components = {} - - for component_name in components_to_update: - if component_name not in candidate or component_name not in reflective_dataset: - continue - - current_instruction = candidate[component_name] - component_examples = reflective_dataset[component_name] - - # Create feedback summary - feedback_text = "\n".join([ - f"Example {i+1}: {ex.get('Feedback', 'No feedback')}" - for i, ex in enumerate(component_examples) # Limit examples to prevent context overflow - ]) +#### Basic Example: Character Limit - # Use the module to improve the instruction - result = self.instruction_improver( - current_instruction=current_instruction, - feedback_summary=feedback_text, - max_words=self.max_words - ) +A character limit needs no custom proposer. The default proposer validates it with Pydantic and allows one compression call when a draft is too long: - updated_components[component_name] = result.improved_instruction - - return updated_components +```python +from dspy.teleprompt.gepa import InstructionProposer -# Usage gepa = dspy.GEPA( metric=my_metric, - reflection_lm=dspy.LM(model="gpt-5", temperature=1.0, max_tokens=32000, api_key=api_key), - instruction_proposer=WordLimitProposer(max_words=700), + reflection_lm=dspy.LM(model="gpt-5", max_tokens=32000, api_key=api_key), + instruction_proposer=InstructionProposer(max_chars=3500), auto="medium" ) ``` @@ -278,7 +258,7 @@ collection = client.get_collection("instruction_guidelines") gepa = dspy.GEPA( metric=task_specific_metric, - reflection_lm=dspy.LM(model="gpt-5", temperature=1.0, max_tokens=32000, api_key=api_key), + reflection_lm=dspy.LM(model="gpt-5", max_tokens=32000, api_key=api_key), instruction_proposer=DocumentationEnhancedProposer(collection), auto="medium" ) @@ -410,7 +390,7 @@ By default, GEPA uses a **round-robin strategy** (`RoundRobinReflectionComponent # Default round-robin component selection gepa = dspy.GEPA( metric=my_metric, - reflection_lm=dspy.LM(model="gpt-5", temperature=1.0, max_tokens=32000, api_key=api_key), + reflection_lm=dspy.LM(model="gpt-5", max_tokens=32000, api_key=api_key), # component_selector="round_robin" # This is the default auto="medium" ) @@ -511,12 +491,14 @@ gepa = dspy.GEPA( Component selectors work seamlessly with custom instruction proposers. The selector determines which components to update, then the instruction proposer generates new instructions for those components: ```python -# Combined custom selector + custom proposer +from dspy.teleprompt.gepa import InstructionProposer + +# Combined custom selector and configured instruction proposer gepa = dspy.GEPA( metric=my_metric, reflection_lm=reflection_lm, component_selector=alternating_half_selector, - instruction_proposer=WordLimitProposer(max_words=500), + instruction_proposer=InstructionProposer(max_chars=2500), auto="medium" ) ``` diff --git a/docs/docs/diving-deeper/gepa-in-depth.md b/docs/docs/diving-deeper/gepa-in-depth.md index 730d04e3a7..0a14743ce8 100644 --- a/docs/docs/diving-deeper/gepa-in-depth.md +++ b/docs/docs/diving-deeper/gepa-in-depth.md @@ -93,8 +93,8 @@ Reads a minibatch of low-scoring traces and emits a candidate instruction for th **`reflection_minibatch_size`** — examples shown to the reflection LM per mutation Default `3`. Larger minibatches give the proposer more context (better proposals) at the cost of longer reflection prompts (more tokens). -**`instruction_proposer`** — custom proposer hook -A callable matching the `gepa.ProposalFn` protocol: takes a `{predictor_name: current_instruction}` dict, a reflective dataset of low-scoring examples, and a list of components to update; returns a new `{predictor_name: new_instruction}` dict. Override when the default proposer doesn’t handle your modality (signatures with `dspy.Image` fields, say) or when you want domain-specific constraints on the instructions. +**`instruction_proposer`** — the proposer that writes new instructions +A callable matching the `gepa.ProposalFn` protocol: takes a `{predictor_name: current_instruction}` dict, a reflective dataset of low-scoring examples, and a list of components to update; returns a new `{predictor_name: new_instruction}` dict. Defaults to `InstructionProposer()` from `dspy.teleprompt.gepa`, which handles `dspy.Image` inputs and takes options for reference skills, extra guidance, a replacement prompt, length caps, and compaction of long tool outputs. Pass a configured `InstructionProposer(...)` for those; write your own callable only for needs the options don't cover. See [Custom Instruction Proposers](../api/optimizers/GEPA/GEPA_Advanced.md#custom-instruction-proposers). **`code_proposer`** — custom proposer hook for `dspy.Flex` code components The code-side counterpart. Receives the same `candidate` and `reflective_dataset` as `instruction_proposer`, plus two per-component maps: `task_descriptions` (the component's signature — name, objective, and input/output fields) and `context_blurbs` (the tools in scope and the sandbox rules for using them). Returns a full replacement `dspy.Module` source per component. Only `Flex` submodules route to it, so it does nothing on programs without them. See [Custom Code Proposers](../api/optimizers/GEPA/GEPA_Advanced.md#custom-code-proposers). diff --git a/dspy/teleprompt/gepa/__init__.py b/dspy/teleprompt/gepa/__init__.py index e4498168de..64fc2ca196 100644 --- a/dspy/teleprompt/gepa/__init__.py +++ b/dspy/teleprompt/gepa/__init__.py @@ -1,3 +1,4 @@ from dspy.teleprompt.gepa.gepa import GEPA +from dspy.teleprompt.gepa.instruction_proposal import InstructionProposer -__all__ = ["GEPA"] +__all__ = ["GEPA", "InstructionProposer"] diff --git a/dspy/teleprompt/gepa/gepa.py b/dspy/teleprompt/gepa/gepa.py index 8282ccb521..c202c0c9d4 100644 --- a/dspy/teleprompt/gepa/gepa.py +++ b/dspy/teleprompt/gepa/gepa.py @@ -263,31 +263,28 @@ def metric( a strong reflection model. Consider using `dspy.LM(model='gpt-5', temperature=1.0, max_tokens=32000)` for optimal performance. skip_perfect_score: Whether to skip examples with perfect scores during reflection. Default is True. - instruction_proposer: Optional custom instruction proposer implementing GEPA's ProposalFn protocol. - **Default: None (recommended for most users)** - Uses GEPA's proven instruction proposer from - the [GEPA library](https://github.com/gepa-ai/gepa), which implements the - [`ProposalFn`](https://github.com/gepa-ai/gepa/blob/main/src/gepa/core/adapter.py). This default - proposer is highly capable and was validated across diverse experiments reported in the GEPA - paper and tutorials. - - See documentation on custom instruction proposers + instruction_proposer: The proposer that writes new instructions, implementing GEPA's ProposalFn + protocol. **Default: None** - builds `InstructionProposer()` from `dspy.teleprompt.gepa`, the + built-in proposer. It asks the reflection LM for a new instruction through `dspy.Predict` with + a `JSONAdapter`, using GEPA's standard reflection prompt, and sends `dspy.Image` and other + `dspy.Type` inputs to the reflection LM as structured content. + + To configure the built-in proposer, pass an `InstructionProposer(...)` instance. Its options: + - `skills`: reference material (skill files, directories with `SKILL.md`, or inline text) + shown to the reflection LM + - `additional_instructions`: guidance applied to every proposal + - `base_instructions`: replaces the reflection prompt + - `max_chars`: character limit validated with Pydantic after one compression attempt; + an oversized result raises instead of being returned or truncated + - `truncate_history_outputs`: shorten long tool results in `dspy.History` inputs and long outputs in + `REPLHistory` inputs before rendering + - `adapter`: the adapter for the proposer's own LM calls (default `JSONAdapter()`) + + See documentation on instruction proposers [here](https://dspy.ai/api/optimizers/GEPA/GEPA_Advanced/#custom-instruction-proposers). - **Advanced Feature**: Only needed for specialized scenarios: - - **Multi-modal handling**: Processing dspy.Image inputs alongside textual information - - **Nuanced control over constraints**: Fine-grained control over instruction length, format, - and structural requirements beyond standard feedback mechanisms - - **Domain-specific knowledge injection**: Specialized terminology or context that cannot be - provided through feedback_func alone - - **Provider-specific prompting**: Optimizations for specific LLM providers (OpenAI, Anthropic) - with unique formatting preferences - - **Coupled component updates**: Coordinated updates of multiple components together rather - than independent optimization - - **External knowledge integration**: Runtime access to databases, APIs, or knowledge bases - - The default proposer handles the vast majority of use cases effectively. Use - MultiModalInstructionProposer() from dspy.teleprompt.gepa.instruction_proposal for visual - content or implement custom ProposalFn for highly specialized requirements. + Implement a custom ProposalFn only for needs the built-in options do not cover, such as + coupled updates of several components or runtime access to external knowledge. Note: When both instruction_proposer and reflection_lm are set, the instruction_proposer is called in the reflection_lm context. However, reflection_lm is optional when using a custom instruction_proposer. @@ -509,7 +506,8 @@ def __init__( raise ValueError( "reflection_prompt_template cannot be passed via gepa_kwargs when using dspy.GEPA. " "DspyAdapter implements its own propose_new_texts, so reflection_prompt_template is unused. " - "To customize reflection behavior, pass a custom ProposalFn via the instruction_proposer parameter instead." + "To customize the reflection prompt, pass " + "instruction_proposer=InstructionProposer(base_instructions=...) instead." ) def auto_budget( diff --git a/dspy/teleprompt/gepa/gepa_utils.py b/dspy/teleprompt/gepa/gepa_utils.py index 3276f751d9..6a3e81131b 100644 --- a/dspy/teleprompt/gepa/gepa_utils.py +++ b/dspy/teleprompt/gepa/gepa_utils.py @@ -2,11 +2,10 @@ import random from concurrent.futures import ThreadPoolExecutor from contextvars import copy_context -from typing import Any, Callable, Protocol, TypedDict +from typing import Any, Callable, Protocol from gepa import EvaluationBatch, GEPAAdapter from gepa.core.adapter import ProposalFn -from gepa.strategies.instruction_proposal import InstructionProposalSignature import dspy from dspy.adapters.chat_adapter import ChatAdapter @@ -24,6 +23,7 @@ propose_code, rebind_flex_code, ) +from dspy.teleprompt.gepa.instruction_proposal import InstructionProposer, ReflectiveExample logger = logging.getLogger(__name__) @@ -38,20 +38,14 @@ def log(self, x: str): DSPyTrace = list[tuple[Any, dict[str, Any], Prediction]] -ReflectiveExample = TypedDict( - "ReflectiveExample", - { - "Inputs": dict[str, Any], - "Generated Outputs": dict[str, Any] | str, - "Feedback": str, - }, -) - -ReflectiveExample.__doc__ = """ -Structure of individual examples in the reflective dataset. -Each example contains the predictor inputs, generated outputs, and feedback from evaluation. -""" +def format_history_for_reflection(history: History) -> str: + """Render a `dspy.History` as the `Context` string shown to the reflection LM.""" + s = "```json\n" + for i, message in enumerate(history.messages): + s += f" {i}: {message}\n" + s += "```" + return s class CodeProposalFn(Protocol): @@ -135,7 +129,11 @@ def __init__( self.add_format_failure_as_feedback = add_format_failure_as_feedback self.rng = rng or random.Random(0) self.reflection_lm = reflection_lm + + if custom_instruction_proposer is None: + custom_instruction_proposer = InstructionProposer() self.custom_instruction_proposer = custom_instruction_proposer + self._builtin_instruction_proposer = isinstance(custom_instruction_proposer, InstructionProposer) self.custom_code_proposer = custom_code_proposer self.warn_on_score_mismatch = warn_on_score_mismatch self.reflection_minibatch_size = reflection_minibatch_size @@ -182,38 +180,29 @@ def propose_new_texts( if not components_to_update: return results - # A custom proposer overrides the default *instruction* proposer only. - if self.custom_instruction_proposer: - if code_keys and not self.custom_code_proposer and not self._warned_custom_proposer_skips_code: - logger.warning( - "A custom instruction_proposer is set, but %d dspy.Flex code component(s) are " - "being optimized. The custom proposer handles instruction components only; the " - "built-in code proposer rewrites the flex source.", - len(code_keys), - ) - self._warned_custom_proposer_skips_code = True - with dspy.context(lm=reflection_lm): - results.update( - self.custom_instruction_proposer( - candidate=candidate, - reflective_dataset=reflective_dataset, - components_to_update=components_to_update, - ) - ) - return results - + # Instruction components go to the instruction proposer. A custom one handles instruction + # components only, so warn when it would leave Flex code to the built-in code proposer. + if ( + not self._builtin_instruction_proposer + and code_keys + and not self.custom_code_proposer + and not self._warned_custom_proposer_skips_code + ): + logger.warning( + "A custom instruction_proposer is set, but %d dspy.Flex code component(s) are " + "being optimized. The custom proposer handles instruction components only; the " + "built-in code proposer rewrites the flex source.", + len(code_keys), + ) + self._warned_custom_proposer_skips_code = True with dspy.context(lm=reflection_lm): - for name in components_to_update: - base_instruction = candidate[name] - dataset_with_feedback = reflective_dataset[name] - results[name] = InstructionProposalSignature.run( - lm=(lambda x: self.stripped_lm_call(x)[0]), - input_dict={ - "current_instruction_doc": base_instruction, - "dataset_with_feedback": dataset_with_feedback, - }, - )["new_instruction"] - + results.update( + self.custom_instruction_proposer( + candidate=candidate, + reflective_dataset=reflective_dataset, + components_to_update=components_to_update, + ) + ) return results def build_program(self, candidate: dict[str, str]): @@ -414,18 +403,19 @@ def make_reflective_dataset( assert history_key_name is None history_key_name = input_key + # The built-in proposer receives history objects for optional output truncation + # and adapter rendering. Custom proposers receive the same strings as before. if contains_history: - s = "```json\n" - for i, message in enumerate(inputs[history_key_name].messages): - s += f" {i}: {message}\n" - s += "```" - new_inputs["Context"] = s + history = inputs[history_key_name] + new_inputs["Context"] = ( + history if self._builtin_instruction_proposer else format_history_for_reflection(history) + ) for input_key, input_val in inputs.items(): if contains_history and input_key == history_key_name: continue - if isinstance(input_val, Type) and self.custom_instruction_proposer is not None: + if self._builtin_instruction_proposer or isinstance(input_val, Type): # Keep original object - will be properly formatted when sent to reflection LM new_inputs[input_key] = input_val else: @@ -439,7 +429,7 @@ def make_reflective_dataset( new_outputs = s else: for output_key, output_val in outputs.items(): - new_outputs[output_key] = str(output_val) + new_outputs[output_key] = output_val if self._builtin_instruction_proposer else str(output_val) d = {"Inputs": new_inputs, "Generated Outputs": new_outputs} if isinstance(outputs, FailedPrediction): diff --git a/dspy/teleprompt/gepa/instruction_proposal.py b/dspy/teleprompt/gepa/instruction_proposal.py index 9a0e26d35f..024f5f4258 100644 --- a/dspy/teleprompt/gepa/instruction_proposal.py +++ b/dspy/teleprompt/gepa/instruction_proposal.py @@ -1,13 +1,453 @@ -import logging -from typing import Any +import re +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Annotated, Any, TypedDict +import yaml from gepa.core.adapter import ProposalFn +from pydantic import StringConstraints, TypeAdapter, ValidationError import dspy +from dspy.adapters.base import Adapter +from dspy.adapters.json_adapter import JSONAdapter +from dspy.adapters.types import History from dspy.adapters.types.base_type import Type -from dspy.teleprompt.gepa.gepa_utils import ReflectiveExample +from dspy.adapters.types.tool import ToolCallResults, ToolCalls +from dspy.primitives.repl_types import REPLHistory +from dspy.utils.annotation import experimental + +ReflectiveExample = TypedDict( + "ReflectiveExample", + { + "Inputs": dict[str, Any], + "Generated Outputs": dict[str, Any] | str, + "Feedback": str, + }, +) +ReflectiveExample.__doc__ = "An example's predictor inputs, generated outputs, and evaluation feedback." + + +# --------------------------------------------------------------------------- # +# Skills: reference material for the reflection LM +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class _Skill: + name: str + content: str + description: str | None = None + + +class _SkillLoader: + """Resolve a skill source (a directory, a file, or inline text) into a `_Skill`. + + Resolution order for `load`: + + 1. A `Path` is always a path (`~` is expanded). A missing path raises `FileNotFoundError`. + 2. A `str` naming an existing file or directory (after `~` expansion) is a path. + 3. A `str` that does not exist but looks like a path raises `FileNotFoundError`. Text spanning + more than one line never looks like a path. A single line without whitespace looks like a + path when it contains a path separator, starts with `.` or `~`, or ends with `.md`, + `.markdown`, or `.txt`. A single line with whitespace looks like a path only when it starts + like one (`./`, `../`, `~/`, `/`, or a drive letter) or names an entry inside a directory that + exists, so `./skills/my skill` and `skills/my skill` are caught while prose such as + `Consult the file report.txt` stays inline. + 4. Any other `str` is inline skill content. Empty content raises `ValueError`. + + A directory must contain `SKILL.md` (the Agent Skills layout); only that file is read. A file is + read as UTF-8. A leading YAML frontmatter block supplies `name` and `description` and is + stripped from the content. The fallback name is the directory name, the file stem, or the first + line of an inline skill. + + Note: a single line of inline content can be mistaken for a path when it matches one of the rules + above; write it on more than one line, or pass a `Path` for strict path checking. + """ + + SKILL_FILE = "SKILL.md" + FILE_SUFFIXES = (".md", ".markdown", ".txt") + # `./`, `../`, `~/`, `/`, `\`, or a Windows drive letter, with either separator. + _PATH_PREFIX_RE = re.compile(r"^(?:\.{1,2}[\\/]|~[\\/]|[\\/]|[A-Za-z]:[\\/])") + + @classmethod + def load(cls, source: "str | Path") -> _Skill: + if isinstance(source, Path): + return cls._from_path(source.expanduser()) + if not isinstance(source, str): + raise TypeError(f"A skill must be a str or a Path, not {type(source).__name__}.") + + if not source.strip(): + raise ValueError("Empty skill content.") + path = cls._existing_path(source) + if path is not None: + return cls._from_path(path) + if cls._looks_like_path(source): + raise FileNotFoundError( + f"Skill {source!r} looks like a path, but no such file or directory exists. " + "Pass the path to an existing skill file or directory. Inline skill content that " + "resembles a path must span more than one line." + ) + return cls._from_text(source, fallback_name=None) + + @classmethod + def _from_path(cls, path: Path) -> _Skill: + if not path.exists(): + raise FileNotFoundError(f"Skill path {path} does not exist.") + if path.is_dir(): + skill_md = path / cls.SKILL_FILE + if not skill_md.exists(): + raise FileNotFoundError(f"Skill directory {path} has no {cls.SKILL_FILE}.") + return cls._from_text(skill_md.read_text(encoding="utf-8"), fallback_name=path.name) + return cls._from_text(path.read_text(encoding="utf-8"), fallback_name=path.stem) + + @classmethod + def _from_text(cls, text: str, fallback_name: str | None) -> _Skill: + text = text.strip() + if not text: + raise ValueError("Empty skill content.") + + meta, content = cls._parse_frontmatter(text) + content = content.strip() + if not content: + raise ValueError("Empty skill content after removing frontmatter.") + + if fallback_name is None: + # The first non-empty line doubles as a display name for inline skills. + first_line = content.splitlines()[0].lstrip("# ").strip() if content else "" + fallback_name = ((first_line[:60] + "…") if len(first_line) > 60 else first_line) or "inline-skill" + + return _Skill(name=meta.get("name") or fallback_name, content=content, description=meta.get("description")) + + @staticmethod + def _existing_path(source: str) -> Path | None: + try: + path = Path(source).expanduser() + return path if path.exists() else None + except (OSError, ValueError): # e.g. an inline string too long to be a valid path + return None + + @classmethod + def _looks_like_path(cls, source: str) -> bool: + """Whether a string that names nothing on disk was meant as a path rather than inline content.""" + if "\n" in source or "\r" in source: + return False + source = source.strip() + has_separator = "/" in source or "\\" in source + if not re.search(r"\s", source): + return has_separator or source.startswith((".", "~")) or source.lower().endswith(cls.FILE_SUFFIXES) + # Paths may contain spaces ("./skills/my skill"), so a single line with whitespace is still a path + # when it carries a stronger signal than a bare separator or suffix: prose such as + # "Consult the file report.txt" must stay inline. + if cls._PATH_PREFIX_RE.match(source): + return True + return has_separator and cls._parent_exists(source) + + @staticmethod + def _parent_exists(source: str) -> bool: + try: + return Path(source).expanduser().parent.exists() + except (OSError, ValueError): + return False + + @staticmethod + def _parse_frontmatter(text: str) -> tuple[dict[str, str], str]: + """Read YAML frontmatter, retaining string name/description metadata.""" + lines = text.splitlines() + if not lines or lines[0].strip() != "---": + return {}, text + for end, line in enumerate(lines[1:], start=1): + if line.rstrip() == "---": + try: + meta = yaml.safe_load("\n".join(lines[1:end]) + "\n") + except yaml.YAMLError as exc: + raise ValueError("Invalid YAML skill frontmatter.") from exc + if meta is None: + meta = {} + if not isinstance(meta, dict): + raise ValueError("Skill frontmatter must be a YAML mapping.") + for key in ("name", "description"): + if key in meta and not isinstance(meta[key], str): + raise ValueError(f"Skill frontmatter {key!r} must be a string.") + return {key: meta[key] for key in ("name", "description") if key in meta}, "\n".join(lines[end + 1 :]) + raise ValueError("Skill frontmatter is missing its closing '---'.") + + +def _render_skill(skill: _Skill) -> str: + """Render a skill as one `` block for the reflection prompt.""" + attrs = f"name={skill.name!r}" + if skill.description: + attrs += f" description={skill.description!r}" + return f"\n{skill.content.strip()}\n" + + +# --------------------------------------------------------------------------- # +# Signatures +# --------------------------------------------------------------------------- # + + +class ProposeInstruction(dspy.Signature): + """I provided an assistant with instructions to perform a task for me. You are given those instructions, along with examples of different task inputs provided to the assistant, the assistant's response for each of them, and some feedback on how the assistant's response could be better. + + Your task is to write a new instruction for the assistant. + + Read the inputs carefully and identify the input format and infer detailed task description about the task I wish to solve with the assistant. + + Read all the assistant responses and the corresponding feedback. Identify all niche and domain specific factual information about the task and include it in the instruction, as a lot of it may not be available to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, if so, include that in the instruction as well.""" + + current_instruction: str = dspy.InputField(desc="The instructions I provided to the assistant.") + examples_with_feedback: list[dict[str, Any]] = dspy.InputField( + desc="Task inputs provided to the assistant, the assistant's response for each, " + "and feedback on how the response could be better." + ) + new_instruction: str = dspy.OutputField(desc="The new instruction for the assistant.") + + +class CompressInstruction(dspy.Signature): + """Shorten the instruction to fit within the stated limit. Preserve every + strategy, rule, and constraint; cut redundancy and verbosity. Do not add + new content. Output only the shortened instruction.""" + + instruction: str = dspy.InputField() + length_limit: str = dspy.InputField() + shortened_instruction: str = dspy.OutputField() + -logger = logging.getLogger(__name__) +# --------------------------------------------------------------------------- # +# Compaction of long inputs +# --------------------------------------------------------------------------- # + +_COMPACTED_CHARS = 500 + + +def compact_history(history: History, max_chars: int = _COMPACTED_CHARS) -> History: + """Return a copy of `history` with every tool result longer than `max_chars` cut to that length. + + Only the `value` of each `ToolCallResult` attached to a `ToolCalls` message value is shortened; a cut + value ends with a marker naming how many characters were removed. Thoughts, tool names, tool + arguments, user inputs, and error flags are untouched. The input is never mutated: `dspy.ReActV2` + appends to its `History` in place and the trace shares that object. + """ + messages = [] + for message in history.messages: + new_message = dict(message) + for key, value in message.items(): + if isinstance(value, ToolCalls) and isinstance(value.tool_call_results, ToolCallResults): + new_message[key] = _compact_tool_calls(value, max_chars) + messages.append(new_message) + return history.model_copy(update={"messages": messages}) + + +def _compact_tool_calls(tool_calls: ToolCalls, max_chars: int) -> ToolCalls: + results = tool_calls.tool_call_results + new_results = [] + for result in results.tool_call_results: + text = result.value if isinstance(result.value, str) else str(result.value) + if len(text) > max_chars: + cut = len(text) - max_chars + marker = f" [{cut:,} of {len(text):,} characters cut]" + result = result.model_copy(update={"value": text[:max_chars] + marker}) + new_results.append(result) + new_results = results.model_copy(update={"tool_call_results": new_results}) + return tool_calls.model_copy(update={"tool_call_results": new_results}) + + +def compact_repl_history(history: REPLHistory, max_chars: int = _COMPACTED_CHARS) -> REPLHistory: + """Return a copy of `history` whose rendered REPL outputs are capped at `max_chars` characters. + + `REPLHistory.format()` already cuts each output to its head and tail and reports the omitted count, so + only `max_output_chars` changes, and only downwards. + """ + return history.model_copy(update={"max_output_chars": min(history.max_output_chars, max_chars)}) + + +# --------------------------------------------------------------------------- # +# The proposer +# --------------------------------------------------------------------------- # + + +@experimental(version="3.4.1") +class InstructionProposer(ProposalFn): + """GEPA's default instruction proposer. + + `dspy.GEPA` builds `InstructionProposer()` when no `instruction_proposer` is passed. The proposer + passes each component's reflective examples as structured values and asks the reflection LM, through + `dspy.Predict` and a `JSONAdapter`, for a new instruction. Inputs that are `dspy.Type` instances + (for example `dspy.Image`) reach the reflection LM as structured content. + + Args: + skills: Reference material for the reflection LM, as a single source or a sequence of sources. + Each source is a path to a skill file, a directory holding `SKILL.md`, or an inline string. + Loaded once, at construction; a path that does not exist raises. + additional_instructions: Guidance applied to every proposal, for example "Write instructions in + imperative voice." + base_instructions: Replaces the proposal prompt (the `ProposeInstruction` docstring). The input and + output fields are unchanged. + max_chars: Maximum number of Unicode characters in each proposed instruction, after stripping + outer whitespace. Enforced with Pydantic validation; None means no limit. + truncate_history_outputs: When True, long tool results inside `dspy.History` inputs and long outputs inside + `REPLHistory` inputs are cut before the examples are rendered. Nothing else is shortened. + adapter: The adapter for the proposer's own LM calls. Defaults to `JSONAdapter()`. + + When a proposal exceeds max_chars, the proposer makes one call asking the reflection LM to + shorten the draft. Pydantic rejects an empty or still oversized result with ValidationError. + The proposer never truncates an instruction. Exceptions propagate to GEPA for proposal failure handling. + + Example: + ```python + from dspy.teleprompt.gepa import InstructionProposer + + dspy.GEPA( + metric=metric, + reflection_lm=lm, + instruction_proposer=InstructionProposer( + skills=["./skills/prompt-engineering"], + additional_instructions="Write instructions in imperative voice.", + max_chars=1500, + truncate_history_outputs=True, + ), + auto="medium", + ) + ``` + """ + + def __init__( + self, + skills: str | Path | Sequence[str | Path] | None = None, + additional_instructions: str | None = None, + base_instructions: str | None = None, + max_chars: int | None = None, + truncate_history_outputs: bool = False, + adapter: Adapter | None = None, + ): + if max_chars is not None and (isinstance(max_chars, bool) or not isinstance(max_chars, int) or max_chars <= 0): + raise ValueError(f"max_chars must be a positive int or None, got {max_chars!r}.") + if not isinstance(truncate_history_outputs, bool): + raise TypeError(f"truncate_history_outputs must be a bool, got {type(truncate_history_outputs).__name__}.") + if adapter is not None and not isinstance(adapter, Adapter): + raise TypeError(f"adapter must be a dspy.Adapter or None, got {type(adapter).__name__}.") + + if isinstance(skills, str | Path): + skills = [skills] + self.skills: tuple[_Skill, ...] = tuple(_SkillLoader.load(skill) for skill in (skills or ())) + self.additional_instructions = _clean_text("additional_instructions", additional_instructions) + self.base_instructions = _clean_text("base_instructions", base_instructions) + self.max_chars = max_chars + # Adapter parsing currently drops top-level field constraints. Validate here so the + # same Pydantic contract holds with JSONAdapter and custom adapters alike. + self._instruction_validator = TypeAdapter( + Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=max_chars)] + ) + self.truncate_history_outputs = truncate_history_outputs + self.adapter = adapter if adapter is not None else JSONAdapter() + + # Static fields come first so provider-side prompt caching can reuse them across calls. Each + # prepend lands at index 0, so they are added in reverse order of their final position. + signature = ProposeInstruction + if self._length_limit_text(): + signature = signature.prepend( + "length_limit", dspy.InputField(desc="Hard limit on the length of the new instruction."), str + ) + if self.additional_instructions: + signature = signature.prepend( + "additional_instructions", + dspy.InputField(desc="Additional requirements from the user for the new instruction. Follow them."), + str, + ) + if self.skills: + signature = signature.prepend( + "reference_skills", + dspy.InputField( + desc="Trusted reference material chosen by the user. Draw on it when writing the new instruction." + ), + str, + ) + if self.base_instructions: + signature = signature.with_instructions(self.base_instructions) + + self.propose = dspy.Predict(signature) + self.compress = dspy.Predict(CompressInstruction) if self._length_limit_text() else None + + def __call__( + self, + candidate: dict[str, str], + reflective_dataset: Mapping[str, Sequence[Mapping[str, Any]]], + components_to_update: list[str], + ) -> dict[str, str]: + """Propose a new instruction for each component in `components_to_update`. + + A component missing from `candidate` or from `reflective_dataset` is left out of the result. + """ + results: dict[str, str] = {} + for name in components_to_update: + if name not in candidate or name not in reflective_dataset: + continue + + kwargs: dict[str, Any] = {} + if self.skills: + kwargs["reference_skills"] = "\n\n".join(_render_skill(skill) for skill in self.skills) + if self.additional_instructions: + kwargs["additional_instructions"] = self.additional_instructions + if self.compress is not None: + kwargs["length_limit"] = self._length_limit_text() + kwargs["current_instruction"] = candidate[name] + kwargs["examples_with_feedback"] = self._prepare_examples(reflective_dataset[name]) + + with dspy.context(adapter=self.adapter): + pred = self.propose(**kwargs) + + draft = pred.new_instruction.strip() + if not draft: + raise ValueError(f"The reflection LM returned an empty instruction for component {name!r}.") + + results[name] = self._enforce_length(draft) + + return results + + def _prepare_examples(self, examples: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]: + """Optionally shorten history outputs, leaving rendering to the adapter.""" + + def prepare(value: Any) -> Any: + if isinstance(value, History): + return compact_history(value) + if isinstance(value, REPLHistory): + return compact_repl_history(value) + if isinstance(value, Mapping): + return {key: prepare(item) for key, item in value.items()} + if isinstance(value, list | tuple): + return [prepare(item) for item in value] + return value + + if self.truncate_history_outputs: + return [prepare(example) for example in examples] + return [dict(example) for example in examples] + + def _length_limit_text(self) -> str: + return f"The new instruction must be at most {self.max_chars} characters." if self.max_chars is not None else "" + + def _enforce_length(self, draft: str) -> str: + """Validate with Pydantic, allowing one attempt to compress an oversized draft.""" + try: + return self._instruction_validator.validate_python(draft) + except ValidationError: + if self.compress is None: + raise + with dspy.context(adapter=self.adapter): + shortened = self.compress(instruction=draft, length_limit=self._length_limit_text()).shortened_instruction + return self._instruction_validator.validate_python(shortened) + + +def _clean_text(label: str, value: str | None) -> str | None: + if value is None: + return None + if not isinstance(value, str): + raise TypeError(f"{label} must be a str or None, got {type(value).__name__}.") + return value.strip() or None + + +# --------------------------------------------------------------------------- # +# Multimodal proposer +# --------------------------------------------------------------------------- # class GenerateEnhancedMultimodalInstructionFromFeedback(dspy.Signature): @@ -206,7 +646,7 @@ def render_value_with_images(value, level=3, example_images=None): if not value: s += "\n" return s - elif isinstance(value, (list, tuple)): + elif isinstance(value, list | tuple): s = "" for i, item in enumerate(value): s += f"{'#' * level} Item {i + 1}\n" @@ -269,6 +709,9 @@ def _create_multimodal_examples(self, formatted_text: str, image_map: dict[int, return multimodal_content +# TODO: InstructionProposer now delivers images to the reflection LM as structured content. +# Compare quality on a vision task before considering deprecation: +# https://github.com/stanfordnlp/dspy/issues/10591 class MultiModalInstructionProposer(ProposalFn): """GEPA-compatible multimodal instruction proposer. diff --git a/pyproject.toml b/pyproject.toml index fd920dc361..e0793cd5db 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,6 +27,7 @@ dependencies = [ "tqdm>=4.66.1", "requests>=2.31.0", "pydantic>=2.11.0", + "pyyaml>=6.0.1", "litellm>=1.65.8", "diskcache>=5.6.0", "json-repair>=0.54.2", diff --git a/tests/teleprompt/gepa_dummy_lm.json b/tests/teleprompt/gepa_dummy_lm.json index a532d5a89e..25db2113dd 100644 --- a/tests/teleprompt/gepa_dummy_lm.json +++ b/tests/teleprompt/gepa_dummy_lm.json @@ -1 +1,306 @@ -{"lm": [{"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.0001624, "timestamp": "2025-08-13T18:20:44.059820", "uuid": "93b03c3f-7e96-43a0-bac1-2ececc179365", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.000136, "timestamp": "2025-08-13T18:20:44.060309", "uuid": "b3367637-304b-46cf-9147-6a976a8db439", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.000136, "timestamp": "2025-08-13T18:20:44.074500", "uuid": "34df9ccb-128c-4a66-8ee7-c17de080d2ad", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.0001624, "timestamp": "2025-08-13T18:20:44.076517", "uuid": "eae79449-93ab-474c-887e-e8ecde69f8e8", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.0001624, "timestamp": "2025-08-13T18:20:44.078480", "uuid": "6768dd52-0e72-4648-891b-4d60bea205ec", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.000234, "timestamp": "2025-08-13T18:20:44.093296", "uuid": "3d8eee4c-f7f0-4f74-a67a-8a43da85da15", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.0002492, "timestamp": "2025-08-13T18:20:44.094009", "uuid": "ef866572-bbe0-4b47-b64a-af377d18d786", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.0002492, "timestamp": "2025-08-13T18:20:44.094555", "uuid": "5b223989-d453-4c94-bf74-7c94d328c94d", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.0002492, "timestamp": "2025-08-13T18:20:44.123406", "uuid": "64ce9bc4-5a84-4eb4-b8fa-5cceddfb8c7c", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}, {"prompt": null, "messages": [{"role": "system", "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user."}, {"role": "user", "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}], "kwargs": {}, "response": "[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]", "outputs": ["[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]"], "usage": {}, "cost": 0.000234, "timestamp": "2025-08-13T18:20:44.125206", "uuid": "5b38b61a-e139-4962-b788-570b3cbb5c38", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}], "reflection_lm": [{"prompt": "I provided an assistant with the following instructions to perform a task for me:\n```\nGiven the fields `input`, produce the fields `output`.\n```\n\nThe following are examples of different task inputs provided to the assistant along with the assistant's response for each of them, and some feedback on how the assistant's response could be better:\n```\n# Example 1\n## Inputs\n### input\nWhat is the color of the sky?\n\n## Generated Outputs\n### output\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n\n## Feedback\nWrong answer.\n\n\n\n# Example 2\n## Inputs\n### input\nWhat does the fox say?\n\n## Generated Outputs\n### output\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n## Feedback\nWrong answer.\n\n\n\n# Example 3\n## Inputs\n### input\nWhat does the fox say?\n\n## Generated Outputs\n### output\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n## Feedback\nWrong answer.\n\n\n```\n\nYour task is to write a new instruction for the assistant.\n\nRead the inputs carefully and identify the input format and infer detailed task description about the task I wish to solve with the assistant.\n\nRead all the assistant responses and the corresponding feedback. Identify all niche and domain specific factual information about the task and include it in the instruction, as a lot of it may not be available to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, if so, include that in the instruction as well.\n\nProvide the new instructions within ``` blocks.", "messages": null, "kwargs": {}, "response": "``` \nGiven the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n\nKey details and guidelines:\n\n1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n\n2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n\n3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n\n4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n\n5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n\n6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n\n7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n\nExample:\n\n- Input: \"What is the color of the sky?\"\n- Output: \"Blue.\"\n\n- Input: \"What does the fox say?\"\n- Output: \"Ring-ding-ding-ding-dingeringeding!\"\n\nThis approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user.\n```", "outputs": ["``` \nGiven the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n\nKey details and guidelines:\n\n1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n\n2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n\n3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n\n4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n\n5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n\n6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n\n7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n\nExample:\n\n- Input: \"What is the color of the sky?\"\n- Output: \"Blue.\"\n\n- Input: \"What does the fox say?\"\n- Output: \"Ring-ding-ding-ding-dingeringeding!\"\n\nThis approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user.\n```"], "usage": {}, "cost": 0.000774, "timestamp": "2025-08-13T18:20:44.080463", "uuid": "c71eee51-af0b-4469-a365-343105013d66", "model": "openai/gpt-4.1-mini", "response_model": "gpt-4.1-mini-2025-04-14", "model_type": "chat"}]} \ No newline at end of file +{ + "lm": [ + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.0001624, + "timestamp": "2025-08-13T18:20:44.059820", + "uuid": "93b03c3f-7e96-43a0-bac1-2ececc179365", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.000136, + "timestamp": "2025-08-13T18:20:44.060309", + "uuid": "b3367637-304b-46cf-9147-6a976a8db439", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nThe color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.000136, + "timestamp": "2025-08-13T18:20:44.074500", + "uuid": "34df9ccb-128c-4a66-8ee7-c17de080d2ad", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.0001624, + "timestamp": "2025-08-13T18:20:44.076517", + "uuid": "eae79449-93ab-474c-887e-e8ecde69f8e8", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `input`, produce the fields `output`." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nThe fox says a variety of sounds including barks, yips, and howls. However, the phrase \"What does the fox say?\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.0001624, + "timestamp": "2025-08-13T18:20:44.078480", + "uuid": "6768dd52-0e72-4648-891b-4d60bea205ec", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.000234, + "timestamp": "2025-08-13T18:20:44.093296", + "uuid": "3d8eee4c-f7f0-4f74-a67a-8a43da85da15", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.0002492, + "timestamp": "2025-08-13T18:20:44.094009", + "uuid": "ef866572-bbe0-4b47-b64a-af377d18d786", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.0002492, + "timestamp": "2025-08-13T18:20:44.094555", + "uuid": "5b223989-d453-4c94-bf74-7c94d328c94d", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat does the fox say?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nRing-ding-ding-ding-dingeringeding!\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.0002492, + "timestamp": "2025-08-13T18:20:44.123406", + "uuid": "64ce9bc4-5a84-4eb4-b8fa-5cceddfb8c7c", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `input` (str):\nYour output fields are:\n1. `output` (str):\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## input ## ]]\n{input}\n\n[[ ## output ## ]]\n{output}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\n \n Key details and guidelines:\n \n 1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\n \n 2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\n \n 3. If the `input` is a well-known phrase or question from popular culture (e.g., \"What does the fox say?\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\n \n 4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\n \n 5. The goal is to produce the answer that the user expects or the \"correct\" answer in the context of the question, including culturally recognized or meme-based answers.\n \n 6. If the `input` is a straightforward factual question (e.g., \"What is the color of the sky?\"), provide the commonly accepted direct answer (e.g., \"Blue\") rather than a detailed scientific explanation.\n \n 7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\n \n Example:\n \n - Input: \"What is the color of the sky?\"\n - Output: \"Blue.\"\n \n - Input: \"What does the fox say?\"\n - Output: \"Ring-ding-ding-ding-dingeringeding!\"\n \n This approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user." + }, + { + "role": "user", + "content": "[[ ## input ## ]]\nWhat is the color of the sky?\n\nRespond with the corresponding output fields, starting with the field `[[ ## output ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`." + } + ], + "kwargs": {}, + "response": "[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]", + "outputs": [ + "[[ ## output ## ]]\nBlue.\n\n[[ ## completed ## ]]" + ], + "usage": {}, + "cost": 0.000234, + "timestamp": "2025-08-13T18:20:44.125206", + "uuid": "5b38b61a-e139-4962-b788-570b3cbb5c38", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + } + ], + "reflection_lm": [ + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `current_instruction` (str): The instructions I provided to the assistant.\n2. `examples_with_feedback` (list[dict[str, Any]]): Task inputs provided to the assistant, the assistant's response for each, and feedback on how the response could be better.\nYour output fields are:\n1. `new_instruction` (str): The new instruction for the assistant.\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\nInputs will have the following structure:\n\n[[ ## current_instruction ## ]]\n{current_instruction}\n\n[[ ## examples_with_feedback ## ]]\n{examples_with_feedback}\n\nOutputs will be a JSON object with the following fields.\n\n{\n \"new_instruction\": \"{new_instruction}\"\n}\nIn adhering to this structure, your objective is: \n I provided an assistant with instructions to perform a task for me. You are given those instructions, along with examples of different task inputs provided to the assistant, the assistant's response for each of them, and some feedback on how the assistant's response could be better.\n \n Your task is to write a new instruction for the assistant.\n \n Read the inputs carefully and identify the input format and infer detailed task description about the task I wish to solve with the assistant.\n \n Read all the assistant responses and the corresponding feedback. Identify all niche and domain specific factual information about the task and include it in the instruction, as a lot of it may not be available to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, if so, include that in the instruction as well." + }, + { + "role": "user", + "content": "[[ ## current_instruction ## ]]\nGiven the fields `input`, produce the fields `output`.\n\n[[ ## examples_with_feedback ## ]]\n[{\"Inputs\": {\"input\": \"What is the color of the sky?\"}, \"Generated Outputs\": {\"output\": \"The color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\"}, \"Feedback\": \"Wrong answer.\"}, {\"Inputs\": {\"input\": \"What does the fox say?\"}, \"Generated Outputs\": {\"output\": \"The fox says a variety of sounds including barks, yips, and howls. However, the phrase \\\"What does the fox say?\\\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\"}, \"Feedback\": \"Wrong answer.\"}, {\"Inputs\": {\"input\": \"What does the fox say?\"}, \"Generated Outputs\": {\"output\": \"The fox says a variety of sounds including barks, yips, and howls. However, the phrase \\\"What does the fox say?\\\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\"}, \"Feedback\": \"Wrong answer.\"}]\n\nRespond with a JSON object in the following order of fields: `new_instruction`." + } + ], + "kwargs": {}, + "response": "{\"new_instruction\": \"Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\\n\\nKey details and guidelines:\\n\\n1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\\n\\n2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\\n\\n3. If the `input` is a well-known phrase or question from popular culture (e.g., \\\"What does the fox say?\\\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\\n\\n4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\\n\\n5. The goal is to produce the answer that the user expects or the \\\"correct\\\" answer in the context of the question, including culturally recognized or meme-based answers.\\n\\n6. If the `input` is a straightforward factual question (e.g., \\\"What is the color of the sky?\\\"), provide the commonly accepted direct answer (e.g., \\\"Blue\\\") rather than a detailed scientific explanation.\\n\\n7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\\n\\nExample:\\n\\n- Input: \\\"What is the color of the sky?\\\"\\n- Output: \\\"Blue.\\\"\\n\\n- Input: \\\"What does the fox say?\\\"\\n- Output: \\\"Ring-ding-ding-ding-dingeringeding!\\\"\\n\\nThis approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user.\"}", + "outputs": [ + "{\"new_instruction\": \"Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\\n\\nKey details and guidelines:\\n\\n1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\\n\\n2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\\n\\n3. If the `input` is a well-known phrase or question from popular culture (e.g., \\\"What does the fox say?\\\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\\n\\n4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\\n\\n5. The goal is to produce the answer that the user expects or the \\\"correct\\\" answer in the context of the question, including culturally recognized or meme-based answers.\\n\\n6. If the `input` is a straightforward factual question (e.g., \\\"What is the color of the sky?\\\"), provide the commonly accepted direct answer (e.g., \\\"Blue\\\") rather than a detailed scientific explanation.\\n\\n7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\\n\\nExample:\\n\\n- Input: \\\"What is the color of the sky?\\\"\\n- Output: \\\"Blue.\\\"\\n\\n- Input: \\\"What does the fox say?\\\"\\n- Output: \\\"Ring-ding-ding-ding-dingeringeding!\\\"\\n\\nThis approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user.\"}" + ], + "usage": {}, + "cost": 0.000774, + "timestamp": "2025-08-13T18:20:44.080463", + "uuid": "c71eee51-af0b-4469-a365-343105013d66", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + }, + { + "prompt": null, + "messages": [ + { + "role": "system", + "content": "Your input fields are:\n1. `current_instruction` (str): The instructions I provided to the assistant.\n2. `examples_with_feedback` (list[dict[str, Any]]): Task inputs provided to the assistant, the assistant's response for each, and feedback on how the response could be better.\nYour output fields are:\n1. `new_instruction` (str): The new instruction for the assistant.\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\nInputs will have the following structure:\n\n[[ ## current_instruction ## ]]\n{current_instruction}\n\n[[ ## examples_with_feedback ## ]]\n{examples_with_feedback}\n\nOutputs will be a JSON object with the following fields.\n\n{\n \"new_instruction\": \"{new_instruction}\"\n}\nIn adhering to this structure, your objective is: \n I provided an assistant with instructions to perform a task for me. You are given those instructions, along with examples of different task inputs provided to the assistant, the assistant's response for each of them, and some feedback on how the assistant's response could be better.\n \n Your task is to write a new instruction for the assistant.\n \n Read the inputs carefully and identify the input format and infer detailed task description about the task I wish to solve with the assistant.\n \n Read all the assistant responses and the corresponding feedback. Identify all niche and domain specific factual information about the task and include it in the instruction, as a lot of it may not be available to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, if so, include that in the instruction as well." + }, + { + "role": "user", + "content": "[[ ## current_instruction ## ]]\nGiven the fields `input`, produce the fields `output`.\n\n[[ ## examples_with_feedback ## ]]\n[{\"Inputs\": {\"input\": \"What is the color of the sky?\"}, \"Generated Outputs\": {\"output\": \"The color of the sky is typically blue during the day due to the scattering of sunlight by the atmosphere. However, it can appear different colors at sunrise, sunset, or under various weather conditions.\"}, \"Feedback\": \"Return the expected answer.\"}, {\"Inputs\": {\"input\": \"What does the fox say?\"}, \"Generated Outputs\": {\"output\": \"The fox says a variety of sounds including barks, yips, and howls. However, the phrase \\\"What does the fox say?\\\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\"}, \"Feedback\": \"Return the expected answer.\"}, {\"Inputs\": {\"input\": \"What does the fox say?\"}, \"Generated Outputs\": {\"output\": \"The fox says a variety of sounds including barks, yips, and howls. However, the phrase \\\"What does the fox say?\\\" is famously known from the viral song by Ylvis, where humorous and nonsensical sounds are suggested as the fox's call.\"}, \"Feedback\": \"Return the expected answer.\"}]\n\nRespond with a JSON object in the following order of fields: `new_instruction`." + } + ], + "kwargs": {}, + "response": "{\"new_instruction\": \"Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\\n\\nKey details and guidelines:\\n\\n1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\\n\\n2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\\n\\n3. If the `input` is a well-known phrase or question from popular culture (e.g., \\\"What does the fox say?\\\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\\n\\n4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\\n\\n5. The goal is to produce the answer that the user expects or the \\\"correct\\\" answer in the context of the question, including culturally recognized or meme-based answers.\\n\\n6. If the `input` is a straightforward factual question (e.g., \\\"What is the color of the sky?\\\"), provide the commonly accepted direct answer (e.g., \\\"Blue\\\") rather than a detailed scientific explanation.\\n\\n7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\\n\\nExample:\\n\\n- Input: \\\"What is the color of the sky?\\\"\\n- Output: \\\"Blue.\\\"\\n\\n- Input: \\\"What does the fox say?\\\"\\n- Output: \\\"Ring-ding-ding-ding-dingeringeding!\\\"\\n\\nThis approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user.\"}", + "outputs": [ + "{\"new_instruction\": \"Given the field `input` containing a question or phrase, produce the field `output` containing the exact, direct, and contextually appropriate answer or response that the user expects, without additional explanations, commentary, or general knowledge unless explicitly requested.\\n\\nKey details and guidelines:\\n\\n1. The `input` field contains a question or phrase that may be literal, factual, or culturally specific (e.g., references to popular culture or memes).\\n\\n2. The `output` must be the precise answer or response that directly addresses the `input` as intended by the user, not a general or encyclopedic explanation.\\n\\n3. If the `input` is a well-known phrase or question from popular culture (e.g., \\\"What does the fox say?\\\"), the `output` should reflect the expected or canonical answer associated with that phrase, rather than a factual or scientific explanation.\\n\\n4. Avoid providing additional background information, scientific explanations, or alternative interpretations unless explicitly requested.\\n\\n5. The goal is to produce the answer that the user expects or the \\\"correct\\\" answer in the context of the question, including culturally recognized or meme-based answers.\\n\\n6. If the `input` is a straightforward factual question (e.g., \\\"What is the color of the sky?\\\"), provide the commonly accepted direct answer (e.g., \\\"Blue\\\") rather than a detailed scientific explanation.\\n\\n7. The output should be concise, clear, and focused solely on answering the question or phrase in the `input`.\\n\\nExample:\\n\\n- Input: \\\"What is the color of the sky?\\\"\\n- Output: \\\"Blue.\\\"\\n\\n- Input: \\\"What does the fox say?\\\"\\n- Output: \\\"Ring-ding-ding-ding-dingeringeding!\\\"\\n\\nThis approach ensures that the assistant provides the expected, contextually appropriate answers rather than general or overly detailed responses that may be considered incorrect by the user.\"}" + ], + "usage": {}, + "cost": 0.000774, + "timestamp": "2025-08-13T18:20:44.080463", + "uuid": "c71eee51-af0b-4469-a365-343105013d66", + "model": "openai/gpt-4.1-mini", + "response_model": "gpt-4.1-mini-2025-04-14", + "model_type": "chat" + } + ] +} diff --git a/tests/teleprompt/test_gepa.py b/tests/teleprompt/test_gepa.py index 5af96120f2..cc95bd4fda 100644 --- a/tests/teleprompt/test_gepa.py +++ b/tests/teleprompt/test_gepa.py @@ -342,7 +342,7 @@ def test_gepa_compile_with_track_usage_no_tuple_error(caplog): trainset = [dspy.Example(question="What is 2+2?", answer="4").with_inputs("question")] task_lm = DummyLM([{"answer": "mock answer 1"}]) - reflection_lm = DummyLM([{"new_instruction": "Something new."}]) + reflection_lm = DummyLM([{"new_instruction": "Something new."}], adapter=dspy.JSONAdapter()) compiled_container: dict[str, Any] = {} exc_container: dict[str, BaseException] = {} @@ -410,10 +410,11 @@ def test_selector(state, trajectories, subsample_scores, candidate_idx, candidat task_lm = DummyLM([{"category": "test_category", "output": "test_output"}] * 20) reflection_lm = DummyLM( [ - {"improved_instruction": "Improved classifier instruction"}, - {"improved_instruction": "Improved generator instruction"}, + {"new_instruction": "Improved classifier instruction"}, + {"new_instruction": "Improved generator instruction"}, ] - * 10 + * 10, + adapter=dspy.JSONAdapter(), ) trainset = [dspy.Example(input="test", output="expected").with_inputs("input")] @@ -439,7 +440,7 @@ def test_component_selector_default_behavior(): # Provide enough responses for all possible LM calls task_lm = DummyLM([{"category": "test_category", "output": "test_output"}] * 15) - reflection_lm = DummyLM([{"improved_instruction": "Better instruction"}] * 8) + reflection_lm = DummyLM([{"new_instruction": "Better instruction"}] * 8, adapter=dspy.JSONAdapter()) trainset = [dspy.Example(input="test", output="expected").with_inputs("input")] with dspy.context(lm=task_lm): @@ -460,7 +461,7 @@ def test_component_selector_string_round_robin(): # Provide enough responses for all possible LM calls task_lm = DummyLM([{"category": "test_category", "output": "test_output"}] * 15) - reflection_lm = DummyLM([{"improved_instruction": "Better instruction"}] * 8) + reflection_lm = DummyLM([{"new_instruction": "Better instruction"}] * 8, adapter=dspy.JSONAdapter()) trainset = [dspy.Example(input="test", output="expected").with_inputs("input")] with dspy.context(lm=task_lm): @@ -498,10 +499,11 @@ def improving_metric(example, prediction, trace=None, pred_name=None, pred_trace task_lm = DummyLM([{"category": "test_category", "output": "test_output"}] * 20) reflection_lm = DummyLM( [ - {"improved_instruction": "Updated classifier instruction"}, - {"improved_instruction": "Updated generator instruction"}, + {"new_instruction": "Updated classifier instruction"}, + {"new_instruction": "Updated generator instruction"}, ] - * 10 + * 10, + adapter=dspy.JSONAdapter(), ) trainset = [dspy.Example(input="test", output="expected").with_inputs("input")] @@ -552,7 +554,7 @@ def random_component_selector(state, trajectories, subsample_scores, candidate_i # Provide enough responses for all possible LM calls task_lm = DummyLM([{"category": "test_category", "output": "test_output"}] * 15) - reflection_lm = DummyLM([{"improved_instruction": "Better instruction"}] * 8) + reflection_lm = DummyLM([{"new_instruction": "Better instruction"}] * 8, adapter=dspy.JSONAdapter()) trainset = [dspy.Example(input="test", output="expected").with_inputs("input")] with dspy.context(lm=task_lm): @@ -603,7 +605,7 @@ def alternating_half_selector(state, trajectories, subsample_scores, candidate_i # Provide enough responses for multiple iterations task_lm = DummyLM([{"category": "test_category", "output": "test_output"}] * 20) - reflection_lm = DummyLM([{"improved_instruction": "Better instruction"}] * 10) + reflection_lm = DummyLM([{"new_instruction": "Better instruction"}] * 10, adapter=dspy.JSONAdapter()) trainset = [dspy.Example(input="test", output="expected").with_inputs("input")] with dspy.context(lm=task_lm): diff --git a/tests/teleprompt/test_gepa_builtin_instruction_proposer.py b/tests/teleprompt/test_gepa_builtin_instruction_proposer.py new file mode 100644 index 0000000000..ef0f957a8d --- /dev/null +++ b/tests/teleprompt/test_gepa_builtin_instruction_proposer.py @@ -0,0 +1,701 @@ +"""Tests for `InstructionProposer`, GEPA's built-in instruction proposer.""" + +import json +import logging +from pathlib import Path +from typing import Any + +import pytest +from pydantic import ValidationError + +import dspy +from dspy.adapters.types.tool import ToolCallResults, ToolCalls +from dspy.primitives.repl_types import REPLHistory +from dspy.teleprompt.gepa import InstructionProposer +from dspy.teleprompt.gepa.gepa_utils import DspyAdapter +from dspy.teleprompt.gepa.instruction_proposal import ( + ProposeInstruction, + _Skill, + _SkillLoader, + compact_history, + compact_repl_history, +) +from dspy.utils.dummies import DummyLM + +GEPA_PARAGRAPH_1 = ( + "Read the inputs carefully and identify the input format and infer detailed task description about the " + "task I wish to solve with the assistant." +) +GEPA_PARAGRAPH_2 = ( + "Read all the assistant responses and the corresponding feedback. Identify all niche and domain specific " + "factual information about the task and include it in the instruction, as a lot of it may not be available " + "to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, " + "if so, include that in the instruction as well." +) + +EXAMPLES = [ + {"Inputs": {"question": "2+2?"}, "Generated Outputs": {"answer": "5"}, "Feedback": "Wrong answer."}, + {"Inputs": {"question": "3+3?"}, "Generated Outputs": {"answer": "6"}, "Feedback": "Correct."}, +] + + +@pytest.fixture(autouse=True) +def capture_dspy_logs(monkeypatch): + # DSPy's logger has its own handler and normally disables propagation to pytest. + monkeypatch.setattr(logging.getLogger("dspy"), "propagate", True) + + +def json_lm(*answers: str, **kwargs) -> DummyLM: + """A reflection LM whose answers are proposals (and, after the first, compressions).""" + scripted = [] + for i, text in enumerate(answers): + scripted.append({"new_instruction" if i == 0 else "shortened_instruction": text}) + return DummyLM(scripted, adapter=dspy.JSONAdapter(), **kwargs) + + +def last_messages(lm: DummyLM) -> tuple[str, str]: + messages = lm.history[-1]["messages"] + assert [m["role"] for m in messages] == ["system", "user"] + return messages[0]["content"], messages[1]["content"] + + +def propose(proposer: InstructionProposer, lm: DummyLM, examples=EXAMPLES, components=("pred",)) -> dict[str, str]: + with dspy.context(lm=lm): + return proposer( + candidate={"pred": "Answer the question."}, + reflective_dataset={"pred": examples}, + components_to_update=list(components), + ) + + +# --- the proposer, called directly ------------------------------------------ + + +def test_default_prompt_has_only_the_two_base_fields_and_gepa_wording(): + lm = json_lm("Add the numbers.") + assert propose(InstructionProposer(), lm) == {"pred": "Add the numbers."} + + system, user = last_messages(lm) + assert GEPA_PARAGRAPH_1 in system + assert GEPA_PARAGRAPH_2 in system + assert "Provide the new instructions within" not in system + assert "[[ ## current_instruction ## ]]\nAnswer the question." in user + assert json.dumps(EXAMPLES) in user + for absent in ("reference_skills", "additional_instructions", "length_limit"): + assert absent not in user + assert absent not in system + assert list(InstructionProposer().propose.signature.input_fields) == [ + "current_instruction", + "examples_with_feedback", + ] + + +@pytest.mark.parametrize( + ("options", "field"), + [ + ({"skills": ["Be terse."]}, "reference_skills"), + ({"additional_instructions": "Use imperative voice."}, "additional_instructions"), + ({"max_chars": 50}, "length_limit"), + ], +) +def test_each_option_adds_only_its_field(options, field): + proposer = InstructionProposer(**options) + assert list(proposer.propose.signature.input_fields) == [field, "current_instruction", "examples_with_feedback"] + + lm = json_lm("New instruction.") + propose(proposer, lm) + _, user = last_messages(lm) + assert user.index(f"[[ ## {field} ## ]]") < user.index("[[ ## current_instruction ## ]]") + + +def test_static_fields_precede_the_dynamic_ones(): + proposer = InstructionProposer(skills=["Be terse."], additional_instructions="Use imperative voice.", max_chars=50) + assert list(proposer.propose.signature.input_fields) == [ + "reference_skills", + "additional_instructions", + "length_limit", + "current_instruction", + "examples_with_feedback", + ] + lm = json_lm("New instruction.") + propose(proposer, lm) + _, user = last_messages(lm) + assert "[[ ## reference_skills ## ]]\n\nBe terse.\n" in user + assert "[[ ## additional_instructions ## ]]\nUse imperative voice." in user + assert "[[ ## length_limit ## ]]\nThe new instruction must be at most 50 characters." in user + + +def test_base_instructions_replace_the_docstring_and_keep_the_fields(): + proposer = InstructionProposer(base_instructions="Rewrite the instruction so it generalizes.") + assert proposer.propose.signature.instructions == "Rewrite the instruction so it generalizes." + assert list(proposer.propose.signature.input_fields) == list(ProposeInstruction.input_fields) + assert list(proposer.propose.signature.output_fields) == ["new_instruction"] + + lm = json_lm("New instruction.") + propose(proposer, lm) + system, _ = last_messages(lm) + assert "Rewrite the instruction so it generalizes." in system + assert GEPA_PARAGRAPH_1 not in system + + +def test_default_adapter_is_json_even_when_settings_adapter_differs(): + with dspy.context(adapter=dspy.ChatAdapter()): + lm = json_lm("New instruction.") + proposer = InstructionProposer() + assert isinstance(proposer.adapter, dspy.JSONAdapter) + assert propose(proposer, lm) == {"pred": "New instruction."} + _, user = last_messages(lm) + assert "Respond with a JSON object" in user + + chat_lm = DummyLM([{"new_instruction": "Chat instruction."}], adapter=dspy.ChatAdapter()) + proposer = InstructionProposer(adapter=dspy.ChatAdapter()) + assert propose(proposer, chat_lm) == {"pred": "Chat instruction."} + _, user = last_messages(chat_lm) + assert "Respond with a JSON object" not in user + assert "[[ ## new_instruction ## ]]" in user + + +def test_constructor_validation(): + with pytest.raises(ValueError, match="max_chars"): + InstructionProposer(max_chars=0) + with pytest.raises(TypeError, match="truncate_history_outputs"): + InstructionProposer(truncate_history_outputs="yes") + with pytest.raises(TypeError, match="adapter"): + InstructionProposer(adapter=object()) + assert InstructionProposer(additional_instructions=" ").additional_instructions is None + assert InstructionProposer(base_instructions="").base_instructions is None + + +def test_component_missing_from_the_reflective_dataset_is_left_out(): + lm = json_lm("New instruction.") + with dspy.context(lm=lm): + out = InstructionProposer()( + candidate={"pred": "old", "other": "old too"}, + reflective_dataset={"pred": EXAMPLES}, + components_to_update=["pred", "other", "unknown"], + ) + assert out == {"pred": "New instruction."} + assert len(lm.history) == 1 + + +def test_empty_instruction_raises(): + lm = json_lm(" ") + with pytest.raises(ValueError, match="empty instruction"): + propose(InstructionProposer(), lm) + + +def test_special_characters_round_trip_exactly(): + text = ( + 'Report R&D spend. If score < 5, write "low" and name the .\n\n' + "```python\nprint('ok')\n```\n\nIt's done." + ) + lm = json_lm(text) + assert propose(InstructionProposer(), lm) == {"pred": text} + + +# --- skills ----------------------------------------------------------------- + + +def test_skills_load_from_paths_and_inline_text_and_render_in_order(tmp_path: Path): + skill_dir = tmp_path / "prompt-engineering" + skill_dir.mkdir() + (skill_dir / "SKILL.md").write_text( + "---\nname: prompt-engineering\ndescription: Clear instructions\n---\nBe explicit.", encoding="utf-8" + ) + proposer = InstructionProposer(skills=[skill_dir, "Be terse.", "# Style\n\nUse active voice."]) + assert [skill.name for skill in proposer.skills] == ["prompt-engineering", "Be terse.", "Style"] + + lm = json_lm("New instruction.") + propose(proposer, lm) + _, user = last_messages(lm) + assert ( + "[[ ## reference_skills ## ]]\n" + "\nBe explicit.\n\n\n" + "\nBe terse.\n\n\n" + "\n# Style\n\nUse active voice.\n\n\n" + ) in user + + +def test_skill_directory_loads_skill_md_only(tmp_path: Path): + skill_dir = tmp_path / "prompt-engineering" + skill_dir.mkdir() + (skill_dir / "SKILL.md").write_text("# Prompt engineering\n\nBe explicit.", encoding="utf-8") + (skill_dir / "notes.md").write_text("ignored", encoding="utf-8") + + skill = _SkillLoader.load(str(skill_dir)) + assert skill == _Skill(name="prompt-engineering", content="# Prompt engineering\n\nBe explicit.") + + empty_dir = tmp_path / "empty" + empty_dir.mkdir() + with pytest.raises(FileNotFoundError, match=r"SKILL\.md"): + _SkillLoader.load(empty_dir) + + +def test_skill_file_loads_with_frontmatter(tmp_path: Path): + path = tmp_path / "openai.md" + path.write_text( + "---\nname: openai-style\ndescription: 'Prompting tips for OpenAI models'\n---\n\nUse markdown headers.\n", + encoding="utf-8", + ) + skill = _SkillLoader.load(path) + assert skill == _Skill( + name="openai-style", content="Use markdown headers.", description="Prompting tips for OpenAI models" + ) + assert _SkillLoader.load(str(path)) == skill + + plain = tmp_path / "plain.txt" + plain.write_text("Keep it short.", encoding="utf-8") + assert _SkillLoader.load(plain) == _Skill(name="plain", content="Keep it short.") + + +def test_missing_skill_path_raises(tmp_path: Path): + with pytest.raises(FileNotFoundError, match="prompt-enginering"): + _SkillLoader.load("./skills/prompt-enginering") + with pytest.raises(FileNotFoundError): + _SkillLoader.load("notes.markdown") + with pytest.raises(FileNotFoundError): + _SkillLoader.load(tmp_path / "missing") + with pytest.raises(FileNotFoundError): + InstructionProposer(skills=[str(tmp_path / "missing")]) + + +@pytest.mark.parametrize( + "source", + [ + "./skills/my skill", + "../skills/my skill", + "~/skills/my skill", + "/skills/my skill", + "C:\\skills\\my skill", + ], +) +def test_missing_skill_path_with_whitespace_raises(source): + with pytest.raises(FileNotFoundError, match="looks like a path"): + _SkillLoader.load(source) + + +def test_missing_skill_inside_an_existing_directory_raises_even_with_whitespace(tmp_path: Path, monkeypatch): + (tmp_path / "skills").mkdir() + with pytest.raises(FileNotFoundError, match="my skill"): + _SkillLoader.load(str(tmp_path / "skills" / "my skill")) + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError, match="my skill"): + InstructionProposer(skills=["skills/my skill"]) + + +@pytest.mark.parametrize( + "source", + [ + "Use JSON/YAML formatting.", + "...and then stop.", + "Approximately ~5 words.", + "Cite the file as docs/style.md where relevant.", + "Consult the file report.txt", + "Use active voice.\nSee ./docs/style for examples.", + ], +) +def test_single_line_prose_and_multi_line_text_are_inline_skills(source): + assert _SkillLoader.load(source).content == source + + +def test_inline_skill_and_empty_skill(): + assert _SkillLoader.load("Be terse.") == _Skill(name="Be terse.", content="Be terse.") + assert _SkillLoader.load("x" * 80 + "\nmore").name == "x" * 60 + "…" + for empty in ("", " \n"): + with pytest.raises(ValueError, match="Empty skill"): + _SkillLoader.load(empty) + with pytest.raises(TypeError): + _SkillLoader.load(42) + + +# --- length caps -------------------------------------------------------------- + + +def test_no_cap_means_no_length_field_and_no_compress_call(): + draft = "x" * 20_000 + lm = json_lm(draft) + proposer = InstructionProposer() + assert proposer.compress is None + assert propose(proposer, lm) == {"pred": draft} + assert len(lm.history) == 1 + + +@pytest.mark.parametrize("draft", ["x" * 5, "é猫🙂ab", "a\nb\nc"]) +def test_draft_at_character_limit_makes_one_call(draft): + lm = json_lm(draft) + assert propose(InstructionProposer(max_chars=5), lm) == {"pred": draft} + assert len(lm.history) == 1 + + +def test_over_length_draft_is_compressed_within_the_cap(): + lm = json_lm("x" * 10, "a\nb") + assert propose(InstructionProposer(max_chars=5), lm) == {"pred": "a\nb"} + assert len(lm.history) == 2 + _, user = last_messages(lm) + assert "[[ ## instruction ## ]]\n" + "x" * 10 in user + assert "The new instruction must be at most 5 characters." in user + + +@pytest.mark.parametrize("compressed", ["x" * 7, "x" * 12, "", " "]) +def test_invalid_compression_raises_pydantic_validation_error(compressed): + lm = json_lm("x" * 10, compressed) + with pytest.raises(ValidationError): + propose(InstructionProposer(max_chars=5), lm) + assert len(lm.history) == 2 + + +@pytest.mark.parametrize("cap", [0, -1, True, 1.5, "5"]) +def test_invalid_character_limit(cap): + with pytest.raises(ValueError, match="max_chars"): + InstructionProposer(max_chars=cap) + + +def test_character_limit_applies_after_stripping_outer_whitespace(): + lm = json_lm(" abcde ") + assert propose(InstructionProposer(max_chars=5), lm) == {"pred": "abcde"} + assert len(lm.history) == 1 + + +def test_character_limit_is_enforced_with_custom_adapter(): + lm = DummyLM( + [{"new_instruction": "too long"}, {"shortened_instruction": "still too long"}], + adapter=dspy.ChatAdapter(), + ) + with pytest.raises(ValidationError): + propose(InstructionProposer(max_chars=5, adapter=dspy.ChatAdapter()), lm) + + +# --- truncate_history_outputs ----------------------------------------------------------------- + + +def tool_history(result: str) -> dspy.History: + tool_calls = ToolCalls.from_dict_list([{"name": "search", "args": {"query": "cats"}}]) + results = ToolCallResults.from_tool_calls_and_values(tool_calls, [result]) + tool_calls = tool_calls.model_copy(update={"tool_call_results": results}) + return dspy.History(messages=[{"question": "cats?", "next_thought": "Search first.", "tool_calls": tool_calls}]) + + +def test_compact_history_cuts_long_tool_results_and_leaves_the_input_alone(): + long_result = "r" * 1_200 + history = tool_history(long_result) + + compacted = compact_history(history) + + [message] = compacted.messages + assert message["question"] == "cats?" + assert message["next_thought"] == "Search first." + call = message["tool_calls"].tool_calls[0] + assert (call.name, call.args) == ("search", {"query": "cats"}) + [result] = message["tool_calls"].tool_call_results.tool_call_results + assert result.value == "r" * 500 + " [700 of 1,200 characters cut]" + assert result.name == "search" + assert result.is_error is False + + # The input is untouched: ReActV2 appends to its History in place and the trace shares that object. + assert history.messages[0]["tool_calls"].tool_call_results.tool_call_results[0].value == long_result + assert compacted is not history + assert compacted.messages is not history.messages + + short = compact_history(tool_history("short")) + assert short.messages[0]["tool_calls"].tool_call_results.tool_call_results[0].value == "short" + + +def test_compact_history_treats_field_markers_in_tool_results_as_plain_text(): + history = tool_history("[[ ## thought_1 ## ]]\n" + "z" * 1_000) + [result] = compact_history(history).messages[0]["tool_calls"].tool_call_results.tool_call_results + assert result.value.startswith("[[ ## thought_1 ## ]]\n" + "z" * 478) + assert result.value.endswith(" [522 of 1,022 characters cut]") + + +def test_compact_history_returns_a_plain_chat_history_unchanged(): + history = dspy.History(messages=[{"question": "a" * 2_000, "answer": "b" * 2_000}, {"question": "c"}]) + assert compact_history(history) == history + + +def test_compact_repl_history_lowers_max_output_chars_and_never_raises_it(): + history = REPLHistory().append(reasoning="look", code="print(x)", output="o" * 3_000) + compacted = compact_repl_history(history) + assert compacted.max_output_chars == 500 + assert compacted.entries == history.entries + assert history.max_output_chars == 10_000 + assert "(2,500 characters omitted)" in compacted.format() + + small = REPLHistory(max_output_chars=100) + assert compact_repl_history(small).max_output_chars == 100 + + +def test_history_and_repl_history_use_adapter_serialization(): + history = tool_history("r" * 1_200) + repl = REPLHistory().append(reasoning="look", code="print(x)", output="o" * 3_000) + examples = [{"Inputs": {"Context": history, "repl": repl}, "Generated Outputs": {"a": "b"}, "Feedback": "ok"}] + + lm = json_lm("New instruction.") + propose(InstructionProposer(), lm, examples=examples) + _, user = last_messages(lm) + assert '"Context": {"messages":' in user + assert "r" * 1_200 in user + assert "o" * 3_000 in user + assert "characters cut" not in user + assert "characters omitted" not in user + + lm = json_lm("New instruction.") + propose(InstructionProposer(truncate_history_outputs=True), lm, examples=examples) + _, user = last_messages(lm) + assert "r" * 500 + " [700 of 1,200 characters cut]" in user + assert "r" * 501 not in user + assert "(2,500 characters omitted)" in user + + +# --- GEPA end to end ------------------------------------------------------------- + + +def no_reflection_errors(caplog) -> bool: + return "Exception during reflection/proposal" not in caplog.text + + +def test_gepa_default_proposer_on_a_text_program(caplog): + student = dspy.Predict("input -> output") + trainset = [ + dspy.Example(input="What is the color of the sky?", output="blue").with_inputs("input"), + dspy.Example(input="What does the fox say?", output="ring").with_inputs("input"), + ] + task_lm = DummyLM([{"output": "unsure"}] * 30) + reflection_lm = DummyLM([{"new_instruction": "Answer in one word."}] * 5, adapter=dspy.JSONAdapter()) + + def metric(gold, pred, trace=None, pred_name=None, pred_trace=None): + return dspy.Prediction(score=float(gold.output == pred.output), feedback="Wrong answer.") + + with dspy.context(lm=task_lm), caplog.at_level(logging.INFO, logger="dspy"): + optimized = dspy.GEPA(metric=metric, reflection_lm=reflection_lm, max_metric_calls=5).compile( + student, trainset=trainset, valset=trainset + ) + + assert no_reflection_errors(caplog) + assert reflection_lm.history, "The reflection LM should have been called" + system, user = last_messages(reflection_lm) + assert GEPA_PARAGRAPH_1 in system + assert "[[ ## current_instruction ## ]]\n" + student.signature.instructions in user + assert '"Inputs": {"input":' in user + assert '"Feedback": "Wrong answer."' in user + assert "Proposed new text for self: Answer in one word." in caplog.text + assert optimized is not None + + +LONG_TOOL_OUTPUT = "cat facts " * 300 # 3,000 characters + + +def react_program_and_lm(): + def lookup(query: str) -> str: + return LONG_TOOL_OUTPUT + + task_lm = DummyLM( + [ + { + "next_thought": "I should look this up.", + "tool_calls": ToolCalls.from_dict_list([{"name": "lookup", "args": {"query": "cats"}}]), + }, + { + "next_thought": "I can answer now.", + "tool_calls": ToolCalls.from_dict_list([{"name": "submit", "args": {"answer": "cats purr"}}]), + }, + ] + * 20 + ) + return dspy.ReActV2("question -> answer", tools=[lookup]), task_lm + + +def react_metric(gold, pred, trace=None, pred_name=None, pred_trace=None): + return dspy.Prediction(score=0.3, feedback="Answer with more detail.") + + +def test_gepa_truncate_history_outputs_on_a_react_program_and_custom_proposers_see_the_context_string(caplog): + trainset = [dspy.Example(question="cats?", answer="cats purr loudly").with_inputs("question")] + + # The built-in proposer with truncate_history_outputs: the reflection prompt omits the long tool output. + program, task_lm = react_program_and_lm() + reflection_lm = DummyLM([{"new_instruction": "Search, then answer in detail."}] * 5, adapter=dspy.JSONAdapter()) + with dspy.context(lm=task_lm, adapter=dspy.ChatAdapter()), caplog.at_level(logging.INFO, logger="dspy"): + dspy.GEPA( + metric=react_metric, + reflection_lm=reflection_lm, + instruction_proposer=InstructionProposer(truncate_history_outputs=True), + max_metric_calls=4, + ).compile(program, trainset=trainset, valset=trainset) + + assert no_reflection_errors(caplog) + _, user = last_messages(reflection_lm) + assert '"Context": {"messages":' in user + assert '"next_thought": "I should look this up."' in user + assert "characters cut]" in user + assert LONG_TOOL_OUTPUT not in user + + # A custom proposer receives the Context string exactly as the bridge built it on main. + seen: list[dict[str, Any]] = [] + + def custom_proposer(candidate, reflective_dataset, components_to_update): + seen.extend(reflective_dataset[name][0] for name in components_to_update) + return dict.fromkeys(components_to_update, "Custom instruction.") + + program, task_lm = react_program_and_lm() + with dspy.context(lm=task_lm, adapter=dspy.ChatAdapter()): + dspy.GEPA(metric=react_metric, instruction_proposer=custom_proposer, max_metric_calls=4).compile( + program, trainset=trainset, valset=trainset + ) + + [example] = seen + context = example["Inputs"]["Context"] + assert isinstance(context, str) + assert context.startswith("```json\n 0: {") + assert context.endswith("```") + assert LONG_TOOL_OUTPUT in context + assert "characters cut" not in context + assert list(example["Inputs"]) == ["Context", "tools"] + assert isinstance(example["Inputs"]["tools"], str) + + +def test_explicit_builtin_proposer_with_a_flex_submodule_logs_no_custom_proposer_warning(caplog): + class Echo(dspy.Signature): + q: str = dspy.InputField() + a: str = dspy.OutputField() + + class Prog(dspy.Module): + def __init__(self): + super().__init__() + self.flex = dspy.Flex(Echo) + self.sibling = dspy.Predict("x -> y") + + def forward(self, **kwargs): + return self.flex(**kwargs) + + prog = Prog() + reflection_lm = DummyLM( + { + "current_source": {"revised_source": prog.flex.module_src}, + "current_instruction": {"new_instruction": "new instruction"}, + }, + adapter=dspy.JSONAdapter(), + ) + adapter = DspyAdapter( + student_module=prog, + metric_fn=lambda gold, pred, trace=None, pred_name=None, pred_trace=None: 0.0, + feedback_map={}, + reflection_lm=reflection_lm, + custom_instruction_proposer=InstructionProposer(additional_instructions="Be brief."), + ) + assert adapter._builtin_instruction_proposer + candidate = {"flex": prog.flex.module_src, "sibling": "old instruction"} + reflective = {"flex": [], "sibling": EXAMPLES} + + with caplog.at_level(logging.WARNING), dspy.context(adapter=dspy.JSONAdapter()): + out = adapter.propose_new_texts(candidate, reflective, ["flex", "sibling"]) + + assert out["sibling"] == "new instruction" + assert not any("custom instruction_proposer" in r.message for r in caplog.records) + + +@pytest.mark.parametrize("source", ["abc", ""]) +def test_scalar_skill_text_is_one_source(source): + if not source: + with pytest.raises(ValueError, match="Empty skill"): + InstructionProposer(skills=source) + else: + assert [s.content for s in InstructionProposer(skills=source).skills] == [source] + + +def test_scalar_skill_path_is_one_source(tmp_path): + path = tmp_path / "skill.md" + path.write_text("Be terse.") + for source in (path, str(path)): + assert [s.content for s in InstructionProposer(skills=source).skills] == ["Be terse."] + + +@pytest.mark.parametrize("source", ["---\nname: empty\n---", "---\nname: empty\n---\n "]) +def test_frontmatter_without_body_is_rejected(source, tmp_path): + with pytest.raises(ValueError, match="Empty skill"): + InstructionProposer(skills=[source]) + path = tmp_path / "SKILL.md" + path.write_text(source) + with pytest.raises(ValueError, match="Empty skill"): + InstructionProposer(skills=[path]) + + +@pytest.mark.parametrize( + ("style", "description"), [("|", "First line.\nSecond line.\n"), (">", "First line. Second line.\n")] +) +def test_yaml_multiline_description(style, description): + source = f"---\nname: example\ndescription: {style}\n First line.\n Second line.\n---\nSkill body." + [skill] = InstructionProposer(skills=[source]).skills + assert skill.description == description + assert skill.content == "Skill body." + + +@pytest.mark.parametrize("metadata", ["[unclosed", "- item", "name: [nested]", "description: 123"]) +def test_invalid_yaml_metadata_is_rejected(metadata): + with pytest.raises(ValueError): + InstructionProposer(skills=[f"---\n{metadata}\n---\nSkill body."]) + + +def test_examples_reach_adapter_as_structured_values(): + seen = [] + + class InspectAdapter(dspy.JSONAdapter): + def format(self, signature, demos, inputs): + seen.append(inputs["examples_with_feedback"]) + return super().format(signature, demos, inputs) + + history = tool_history("short result") + image = dspy.Image(url="https://example.com/image.png") + repl = REPLHistory().append(code="print(1)", output="1") + examples = [ + { + "Inputs": {"Context": history, "nested": [{"image": image}], "repl": repl}, + "Generated Outputs": {"answer": "cat"}, + "Feedback": "ok", + } + ] + lm = json_lm("New instruction.") + propose(InstructionProposer(adapter=InspectAdapter()), lm, examples=examples) + assert seen == [examples] + assert seen[0][0]["Inputs"]["Context"] is history + assert seen[0][0]["Inputs"]["repl"] is repl + content = lm.history[-1]["messages"][-1]["content"] + assert any(part.get("type") == "image_url" for part in content) + + +@pytest.mark.parametrize("builtin", [True, False]) +def test_reflective_dataset_preserves_builtin_values_and_custom_string_format(builtin): + from gepa import EvaluationBatch + + student = dspy.Predict("payload: dict -> answer: list") + payload = {"items": [{"image": dspy.Image(url="https://example.com/image.png"), "count": 3}]} + output = dspy.Prediction(answer=["cat", "dog"]) + example = dspy.Example(payload=payload).with_inputs("payload") + proposer = None if builtin else lambda **kwargs: {} + adapter = DspyAdapter( + student_module=student, + metric_fn=lambda *args, **kwargs: 0, + feedback_map={"self": lambda **kwargs: {"score": 0, "feedback": "Try again."}}, + custom_instruction_proposer=proposer, + ) + batch = EvaluationBatch( + outputs=[output], + scores=[0], + trajectories=[ + { + "trace": [(student, {"payload": payload}, output)], + "example": example, + "prediction": output, + "score": 0, + } + ], + ) + [record] = adapter.make_reflective_dataset({"self": student.signature.instructions}, batch, ["self"])["self"] + assert record["Inputs"]["payload"] == (payload if builtin else str(payload)) + assert record["Generated Outputs"]["answer"] == (output.answer if builtin else str(output.answer)) + + +def test_yaml_description_can_contain_an_indented_separator(): + source = "---\nname: example\ndescription: |\n Before.\n ---\n After.\n---\nSkill body." + [skill] = InstructionProposer(skills=[source]).skills + assert skill.description == "Before.\n---\nAfter.\n" + assert skill.content == "Skill body." diff --git a/tests/teleprompt/test_gepa_instruction_proposer.py b/tests/teleprompt/test_gepa_instruction_proposer.py index 7e168f7bd0..003252526a 100644 --- a/tests/teleprompt/test_gepa_instruction_proposer.py +++ b/tests/teleprompt/test_gepa_instruction_proposer.py @@ -331,13 +331,14 @@ def test_default_proposer(reasoning: bool, caplog): reflection_lm = DummyLM( [ - {"improved_instruction": "Be more specific about image analysis"}, - {"improved_instruction": "Focus on visual features when classifying"}, - {"improved_instruction": "Consider contextual clues in the image"}, - {"improved_instruction": "Analyze shape, color, and texture patterns"}, - {"improved_instruction": "Look for distinguishing characteristics"}, + {"new_instruction": "Be more specific about image analysis"}, + {"new_instruction": "Focus on visual features when classifying"}, + {"new_instruction": "Consider contextual clues in the image"}, + {"new_instruction": "Analyze shape, color, and texture patterns"}, + {"new_instruction": "Look for distinguishing characteristics"}, ], reasoning=reasoning, + adapter=dspy.JSONAdapter(), ) gepa = dspy.GEPA( @@ -363,8 +364,10 @@ def test_default_proposer(reasoning: bool, caplog): assert len(lm.history) > 0, "LM should have been called" assert len(reflection_lm.history) > 0, "Reflection LM should have been called" + # The default InstructionProposer sends images to the reflection LM as structured content. images_in_history = check_images_in_history(reflection_lm.history) - assert images_in_history.has_text_serialized_images, ( - "Expected to find serialized images (CUSTOM-TYPE-START-IDENTIFIER)" + assert images_in_history.has_structured_images, "Expected structured image_url parts in the reflection prompt" + assert not images_in_history.has_text_serialized_images, ( + "Expected no serialized images (CUSTOM-TYPE-START-IDENTIFIER) in the reflection prompt" ) diff --git a/uv.lock b/uv.lock index fd0ad11760..0c3076ea08 100644 --- a/uv.lock +++ b/uv.lock @@ -868,7 +868,7 @@ wheels = [ [[package]] name = "dspy" -version = "3.4.0b1" +version = "3.4.0" source = { editable = "." } dependencies = [ { name = "anyio" }, @@ -881,6 +881,7 @@ dependencies = [ { name = "openai" }, { name = "orjson" }, { name = "pydantic" }, + { name = "pyyaml" }, { name = "regex" }, { name = "requests" }, { name = "tenacity" }, @@ -983,6 +984,7 @@ requires-dist = [ { name = "pytest-asyncio", marker = "extra == 'dev'", specifier = ">=0.26.0" }, { name = "pytest-mock", marker = "extra == 'dev'", specifier = ">=3.12.0" }, { name = "pytest-xdist", marker = "extra == 'dev'", specifier = ">=3.5.0" }, + { name = "pyyaml", specifier = ">=6.0.1" }, { name = "regex", specifier = ">=2023.10.3" }, { name = "requests", specifier = ">=2.31.0" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.3.0" },