Repository navigation
Vendor lm15 with ogg, opus, flac, aac, aiff, webm and mpeg input_audio - #10532
Draft
jonasherfort wants to merge 6 commits into
Draft
jonasherfort wants to merge 6 commits into
jonasherfort wants to merge 6 commits into
Conversation
git-subtree-dir: dspy/_vendor/lm15 git-subtree-split: 63b6bb319f38ad1d31060138da54cb8ac8718711
6a745c84e fix(openai-chat): read ogg, opus, flac, aac, aiff, webm and mpeg input_audio as their true media types git-subtree-dir: dspy/_vendor/lm15 git-subtree-split: 6a745c84e83a58aa4ed1237132d940f78ce65c35
git-subtree-dir: dspy/_vendor/lm15 git-subtree-mainline: 9c900c7 git-subtree-split: 63b6bb319f38ad1d31060138da54cb8ac8718711
Source: https://github.com/cmpnd-ai/lm15-python.git Python commit: b0bffc13bc4a12e115e4c2e92fbba40ca8f3450c Contract: b721ce4f03877d72a91431b3a5ce653ffc0b3e38 Package split: 6a745c84e83a58aa4ed1237132d940f78ce65c35
Before the lm15 update, lm15's Chat Completions ingest refused every input_audio format but wav and mp3, so an ogg recording never reached the native Gemini request (DSPy fell back to LiteLLM, or raised on the lm15 engine). The test drives dspy.Predict through a fake transport and checks the inlineData part.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dspy.Audiorenders as a Chat Completionsinput_audioblock whoseformatis the file's MIME subtype. The vendored lm15 reads that body into a canonical request before it chooses a provider. That reader accepted onlywavandmp3, so an ogg/opus recording sent to a Gemini model never reached lm15's Gemini request. On the lm15 engine it raisedinput_audio.format must be one of ['mp3', 'wav']; otherwise DSPy fell back to LiteLLM. An.mp3file, which DSPy labelsmpeg, hit the same wall.This updates the vendored lm15 with
scripts/update_vendored_lm15.pyto cmpnd-ai/lm15-pythonb0bffc1(cmpnd-ai/lm15-python#2), the one-line-table fix on the 1.0.0rc3 line. Each format now reads as its true media type:ogg,opus,flac,aac,aiff,webmandmpegjoinwavandmp3. Gemini sends the part asinlineDatawith that type. The OpenAI Chat Completions and Anthropic builders still refuse audio before the wire. The vendored diff is that table and the provenance record.tests/clients/test_lm15_audio_formats.pyrunsdspy.Predictwith an oggdspy.Audiothrough a fake transport and checks the GeminiinlineDatapart. It fails on the previous vendor and passes now. The full suite passes (2314 passed, 519 skipped, 2 xfailed), andruff checkis clean.Draft until cmpnd-ai/lm15-python#2 lands; if it lands under another SHA, the updater is re-run against it. The format list itself is lm15-dev/lm15-contract#1.