Show and Tell: A package with StructuredOutputAdapter for getting better structured output from smaller models

[rohitgarudls]

Showcasing a compact-schema DSPy adapter for small local models. I have benchmarked it across 15 model×corpus. DSPy silently downgrades local models out of constrained decoding, and a grammar does not guarantee parseable output. Both are below, with confidence intervals and the raw replies committed.

Repo: https://github.com/rohitgarud/llm-schema-lite · pip install "llm-schema-lite[dspy]" · MIT

The one-liner

import dspy
from llm_schema_lite.dspy_integration import OutputMode, StructuredOutputAdapter

dspy.configure(adapter=StructuredOutputAdapter(output_mode=OutputMode.JSONISH))

Signatures don't change. What changes is the schema DSPy puts in the prompt, and how the reply is parsed back. Constraints ride inline in the type line instead of a $defs block:

// A litigation case record.
// Fields marked with * are required
{
  docket*: string (PATTERN: ^[0-9]{2}-cv-[0-9]{4}$),
  parties*: [{
    name*: string (2-60 chars),
    role*: string // one of: "plaintiff", "defendant", "witness"
  }],
  damages_usd: float OR null // (default=null)
}

231 tokens of JSON Schema → 91. Typically 40-70% fewer schema tokens; check yours with
compare_tokens().

And you keep your own Pydantic models

The ergonomic half, which I think matters more day-to-day than the token count: your existing models go in as-is, inputs as well as outputs. No base class, no mixin, no decorator, no registration step. The adapter reads the raw annotation, so the Company model already serving your API layer and your DB code drops straight into a signature instead of being flattened into multiple InputFields:

class Summarise(dspy.Signature):
    company: Company = dspy.InputField()   # nested BaseModel, not flattened
    summary: str = dspy.OutputField()

Nested models and list[Model] are followed, and the field metadata survives into the prompt: description, ge/le, min_length/max_length, pattern:

[[ ## company ## ]]
{company}        # note: this value follows the schema:
{
  name*: string // Company name,
  hq*: {
    street*: string (>= 3 chars) // Street name,
    zipcode*: string (PATTERN: ^[0-9]{5}$) // ZIP
  },
  branches*: [{ ... }]  // Branch offices
}

So Pydantic stays the single source of truth for constraints and descriptions, rather than having them restated in a signature docstring where they drift from the model. (Compact input schemas are a JSONISH/YAML feature; OutputMode.JSON sends the verbose schema.)

The comparison I should have run first

The obvious objection to any prompt-and-parse library: why not constrain the sampler?
Outlines, XGrammar, vLLM guided decoding, llama.cpp GBNF and Ollama, which honours a json_schema response_format even though LiteLLM reports the capability unsupported for every Ollama prefix. Because of that, DSPy's stock JSONAdapter silently downgrades to {"type": "json_object"}: valid JSON of any shape, not your schema. If you thought you were getting structured outputs from a local model through DSPy, you may not be.

So I added a json-constrained arm that forces the real thing, and pointed it at my own adapters. Three models × five corpora × 30 cases, paired bootstrap, 5000 resamples, 95% CI on the recall difference.

Verdict per cell, mine vs constrained:

corpus	falcon3:1b	llama3.2:1b	qwen3.5:0.8b
synthetic	lose	lose	tie
financial-ner	tie	tie	lose
pii	tie	tie	tie
patient-notes	lose	lose	lose
insurance-claims	lose	lose	tie

0 wins, 7 ties, 8 losses. Worst single cell: patient-notes on qwen, −0.487 [−0.618, −0.345]. The constrained arm was also ok on 30/30 cases in 13 of 15 cells, and I took that to mean it cannot fail to produce parseable output. It can, and a later section is about exactly how.

What DSPy actually does about grammars

DSPy supports OpenAI's response_format through the JSONAdapter. ChatAdapter never sets it and the BAMLAdapter subclasses JSONAdapter and inherits the same path. That path picks one of three tiers, per call (json_adapter.py:57-66):

tier	when	what constrains the model
nothing	response_format not in lm.supported_params	free text
{"type": "json_object"}	open-ended mapping, tool-calls with native FC off, or not lm.supports_response_schema	valid JSON of any shape
the signature's schema	none of the above	your schema

Two things here surprised me.

The capability table is model-keyed, and wrong for local models:

openai/gpt-4o                True
anthropic/claude-sonnet-4-5  True
ollama_chat/llama3.2:1b      False
ollama/llama3.2:1b           False
openai/falcon3:1b            False   <- not an Ollama prefix

That last row is the one to notice, which is not an Ollama quirk. Point an openai/ prefix at any OpenAI-compatible local server and LiteLLM still answers False, because it doesn't recognise the model name, so the downgrade hits vLLM and llama.cpp servers too. Ollama honours a real json_schema response_format regardless of what the table claims and we silently get any-shape JSON.

required does not mean non-null. enforce_required sets required = [every property] and additionalProperties: false recursively, which reads like "the model must fill everything". It isn't: optional fields still serialise as anyOf: [X, null], so the key is mandatory while null remains a legal value. The schema never compelled the fabrication described below. null was available on nearly every field and the models filled them anyway.

One thing to note is that DSPy is not performing constrained decoding, it is requesting it. Whether a grammar actually constrains sampling is entirely handled by the backend. DSPy does not provide grammer api, which can include guided_json, guided_regex, GBNF, Outlines or XGrammar integration.

What the package does about it: today, exactly the same thing

llm-schema-lite ships now constrained decoding as one flag, default off:

StructuredOutputAdapter(output_mode=OutputMode.JSONISH, force_response_schema=True)

The flag suppresses only the capability clause. An open-ended mapping or a ToolCalls output still falls back to json_object, YAML still sends nothing, a tool-carrying signature still backs off, and an LM that doesn't advertise response_format at all is left alone. It overrides a stale capability table, never an endpoint that tells you it cannot accept the parameter.

The adapter alone fails 37% of the time on small models, while adding the grammar takes the failures down to 2.4%. The compact prompt does not beat grammar, but it composes with one at a large discount.

But note that a grammar guarantees shape, not truth. It removes the model's ability to fail, and a model that can't fail confabulates instead. On the other hand, compact prompts do not make a model more truthful. Not having a grammar does, by leaving it the option to fail. I checked whether this was a schema artifact, but it isn't. Nearly every field permits null in the generated schema (patient-notes 12/13, pii 56/57). The model could emit null and declined. On one synthetic case, it invented an entire employer and clamped years to 50, the schema's exact maximum.

A bug worth mentioning

While benchmarking, I found the adapter reporting success on replies that were just the own rendered schema handed back: {"name": "string", "age": "int"}. Against a signature whose fields are all str, that parses and validates perfectly. The failure mode isn't an exception; it's a silent 100% invention rate that scores as a win.

0.7.0 detects and rejects it. Re-scoring the full sweep moved exactly 3 cells, all reclassifying echoes that had counted as successes. Worth checking your own eval isn't counting these.

A grammar does not guarantee parseable output

The sharpest thing I found, and it isn't about the package at all.

19 cases across the constrained arms silently fell back to json_object mid-run.

model / corpus	arm	logger	fallbacks	exception
llama3.2:1b / insurance-claims	composed	ours	9/30	ValidationError
llama3.2:1b / synthetic	composed	ours	1/30	ValidationError
llama3.2:1b / patient-notes	json-constrained	dspy	2/30	ValidationError
llama3.2:1b / patient-notes	composed	ours	1/30	ValidationError

13 of 13 were a Pydantic ValidationError on the model's reply; none was a schema-build failure. The dspy row is the one that generalises, which is the stock JSONAdapter behaviour, not the package subclass.

The failing replies are truncated mid-structure, and several are looping a repeated unit: ' Cedar Lane,' ×33, or a duplicated array element like '"make_model": "Honda CR-V", "year": …'. A grammar permits every bit of this. An arbitrarily long string is legal JSON; so is an arbitrarily long array. The sampler stays perfectly schema-compliant right up until the token budget severs it.

Constrained decoding constrains structure, not termination. If you lean on a grammar for reliability, you still need a parse-failure path.

Caveats

• Three models, all sub-1.2B, one local Ollama. Not a general claim.
• 7 of 15 constrained cells are impure: DSPy fell back to json_object on 19 cases total. Those rows read mixed and the case ids are in the repo. The cause is not a failed schema build.
• pii and patient-notes sit below their all-null floor (0.949 / 0.356) for most arms; extracting nothing often scores better than extracting.
• Ollama moves a cell by up to 0.02 on identical code; one in twenty comparisons clears zero by chance. Treat thin results accordingly; mine included.

Every raw reply is committed, so any number above re-scores with no model and no network:

make bench-dspy BENCH_ARGS="--accuracy --corpus pii --replay <csv>"

How much of JSON Schema does it actually handle?

Fair question for anything claiming to compact schemas, so I ran the whole of JSONSchemaBench through it: 10 configs, 9,542 real-world schemas, Kubernetes and the JSON Schema Store included.

Measured over all 9,542
Ingested without error	9,542 / 9,542 (100.0%)
Schemas that raise	0
Too slow to render (10 s budget)	0
Median token reduction	46.8% (per-config 31.4%–57.0%)
Keywords rendered	JSONish 32/39 · YAML 33/39 · TypeScript 31/39
Validation	full Draft 2020-12

Two things I would rather say myself than have someone find:

Rendering ≠ validation. Every keyword is validated; 32/39 is what survives into the prompt string. Those gaps are also the corpus's rarest keyword: minProperties 1.6%, maxProperties 0.6%, if/then 0.5%, else 0.2%, unevaluatedProperties 0.02%, and dependentRequired appears nowhere in all 9,542.

Quote the median, not the mean. Github_hard medians 40.1% against a mean of 13.9%, because a few pathological schemas drag the average far below the typical one.

oneOf, patternProperties, not and contains are ❌ for every constrained-decoding engine in that benchmark: LLGuidance,
llama.cpp, Outlines, XGrammar, OpenAI, Gemini and ✅ here. Their gaps are architectural: an FSM cannot express not or unbounded recursion. For this package, they are implementation debt. That asymmetry is the one durable argument for prompt-and-parse I found in this whole exercise, and it is worth noting it cuts the opposite way from every benchmark above.

The failure mode

A $ref expands inline at its use site, so a cyclic definition graph can render larger than the schema it came from. Back-references (object // defined above: Address) keep that rare: 56 of 9,542 (0.59%) still expand, and past the worst case the next is 8.67×. The worst, o13029, renders at 82×, with seven mutually recursive definitions, the accepted limit of repeat-suppression.

The part worth reporting: the corpus's worst schema used to render at 45× its input and time out. That was not an architectural limit; it was one over-broad cache-invalidation rule and a truncation anywhere poisoned every ancestor frame, so back-references almost never fired (103 of 51,808 expansions cached). Scoping it took that schema to 0.35× and removed every timeout in the corpus.

So when would you use this?

Honestly: if you have a grammar, use the grammar and put this underneath it. The composed model performs better against the verbose-schema grammar at ~30-45% fewer tokens, and this adapter's own failure rate falls from 37% to 2.4%.

If you can't constrain the sampler, which is the case for plenty of endpoints, this is prompt-and-parse, and it is much weaker: 0 wins, 7 ties, 8 losses, with a 37% failure rate on sub-1.2B models. The invention advantage exists only in that weaker mode, and a fabricated value has to cost you more than a missing one for it to be worth the trade.

The benchmarks were taken from https://github.com/thedataquarry/structured-outputs/ by @prrao87, which benchmarked the BAMLAdapter.

Your honest feedback, even if it's criticism, is much appreciated. Will be running extensive benchmarks on small models to check if the compaction also helps with the weight fine-tuning of the small models with DSPy's BootstrapFinetune

[rohitgarud]

Added JevLM and JevAdapter support with OpenRouter provider as default

[rohitgarudls]

Added SemIfLM for SemIf-OpenJev support. This is the best open-source alternative to Typesafe AI's Jev