feat(adk): support multimodal input types (image/audio/video/document) - #1410
Merged
Conversation
…document) Replace reliance on deprecated BinaryInputContent with support for the newer modality-specific types: ImageInputContent, AudioInputContent, VideoInputContent, and DocumentInputContent. Both InputContentDataSource (inline base64 → Part.inline_data) and InputContentUrlSource (URI → Part.file_data) are now handled, leveraging ADK's native URI support. Legacy BinaryInputContent continues to work for backward compatibility. Closes #1405 https://claude.ai/code/session_01RavM3Kc8e7nGXrhQ5LikEY
Add end-to-end tests that send real multimodal content to Google Gemini through the ADK middleware. Tests are gated on GOOGLE_API_KEY and skipped when the key is absent. Covers: - Inline base64 image → model describes dominant colour (red PNG) - Two inline images compared → model identifies both colours - URL-based image (Wikimedia Commons apple) → model identifies subject - Mixed text + inline image → model answers question about image https://claude.ai/code/session_01RavM3Kc8e7nGXrhQ5LikEY
gemini-2.0-flash is approaching deprecation. https://claude.ai/code/session_01RavM3Kc8e7nGXrhQ5LikEY
Add end-to-end tests that send real multimodal content to Google Gemini through the ADK middleware. Tests are gated on GOOGLE_API_KEY and skipped when the key is absent. Covers: - Inline base64 image (256x256 solid red) → model acknowledges image - Two inline images (red + blue) → model acknowledges both - Document via HTTPS URL (RFC 2549 text) → model summarizes content - Mixed text + image (colour stripes) → model identifies stripe colours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add end-to-end tests that send real multimodal content to Gemini 2.5 Flash through the ADK middleware. Tests are gated on GOOGLE_API_KEY. Covers: - Inline base64 image (256x256 solid red) - model acknowledges image - Two inline images (red + blue) - model acknowledges both - Document via HTTPS URL (RFC 2549 text) - model summarizes content - Mixed text + image (colour stripes) - model identifies stripe colours Also adds an Unreleased section to CHANGELOG.md describing the new multimodal input type support added in the prior commit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
Python Preview PackagesVersion
Install with uvAdd the TestPyPI index to your [[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
explicit = trueThen install the packages you need: # Core SDK
uv add 'ag-ui-protocol==0.0.0.dev1775106306' --index testpypi
# Integrations (each already depends on the matching ag-ui-protocol preview)
uv add 'ag-ui-langgraph==0.0.0.dev1775106306' --index testpypi
uv add 'ag-ui-crewai==0.0.0.dev1775106306' --index testpypi
# NOTE: ag-ui-agent-spec depends on pyagentspec (git-only, not on PyPI).
# You will need to install pyagentspec separately from its git repo.
uv add 'ag-ui-agent-spec==0.0.0.dev1775106306' --index testpypi
uv add 'ag_ui_adk==0.0.0.dev1775106306' --index testpypi
uv add 'ag_ui_strands==0.0.0.dev1775106306' --index testpypiInstall with pippip install \
--index-url https://test.pypi.org/simple/ \
--extra-index-url https://pypi.org/simple/ \
ag-ui-protocol==0.0.0.dev1775106306
Commit: 00d0ff5 |
@ag-ui/a2a-middleware
@ag-ui/a2ui-middleware
@ag-ui/mcp-apps-middleware
@ag-ui/middleware-starter
@ag-ui/a2a
@ag-ui/adk
@ag-ui/ag2
@ag-ui/agno
@ag-ui/aws-strands
@ag-ui/claude-agent-sdk
@ag-ui/crewai
@ag-ui/langchain
@ag-ui/langgraph
@ag-ui/langroid
@ag-ui/llamaindex
@ag-ui/mastra
@ag-ui/pydantic-ai
@ag-ui/server-starter
@ag-ui/server-starter-all-features
@ag-ui/vercel-ai-sdk
create-ag-ui-app
@ag-ui/client
@ag-ui/core
@ag-ui/encoder
@ag-ui/proto
commit: |
ADK's runner was replacing inline_data parts with artifact text
placeholders ("Uploaded file: artifact_xxx") before the model could
see them, causing the model to report it cannot view images.
Setting save_input_blobs_as_artifacts=False in RunConfig preserves
inline binary data so the model receives actual image/audio/video
content.
Also adds a test_multimodal_live.py script for interactive testing
of multimodal messaging against a running ADK server.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The new multimodal input types (ImageInputContent, AudioInputContent, etc.) were added in ag-ui-protocol 0.1.15. Without this pin, CI installs an older version from PyPI which lacks these types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_default_run_config_returns_valid_config: expect False instead of True now that we preserve inline data for multimodal support - test_from_app_with_unsupported_mime_type: allow RUN_ERROR since the invalid mime type blob now reaches the API instead of being stored as an artifact Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ImageInputContent,AudioInputContent,VideoInputContent,DocumentInputContent) in the ADK middleware converter, replacing reliance on the deprecatedBinaryInputContentInputContentDataSource(inline base64) maps toPart(inline_data=Blob(...))andInputContentUrlSource(HTTPS/GCS URLs) maps toPart(file_data=FileData(file_uri=...)), leveraging ADK's native URI supportBinaryInputContentcontinues to work for backward compatibilitysave_input_blobs_as_artifactsinRunConfig— ADK's runner was replacing inline image data with artifact text placeholders before the model could see them, causing the model to say "I cannot see the image"GOOGLE_API_KEYcovering inline images, document URLs, multi-image messages, and mixed text+image contenttest_multimodal_live.pyinteractive script for manual multimodal testing against a running serverCloses #1405
Test plan
GOOGLE_API_KEYis absent🤖 Generated with Claude Code