Skip to content

mime_type and resolution parameters aded to the dspy.File type - #10500

Open
tyrael147 wants to merge 13 commits into
stanfordnlp:mainfrom
tyrael147:gemini-fix
Open

tyrael147 wants to merge 13 commits into
stanfordnlp:mainfrom
tyrael147:gemini-fix

Conversation

@tyrael147

Copy link
Copy Markdown

This is a very simple fix that adds mime_type and resolution arguments to the functions that instantiate a dspy.File object. This is necessary when using gemini features (e.g., File api and different resolutions for media objects), which is not possible at the current version of dspy.

Now, dspy files can be instantiated like this:

# When passing a file from the file-api
my_pdf = dspy.File.from_file_id(
        file_id=<uri-from-gemini-file-api>,
        filename=<filename-from-gemini-file-api>,
        resolution="low",
        mime_type="application/pdf",
    )

# When passing the bytes of the file but you need a custom resolution
my_pdf = dspy.File.from_bytes(
            file_bytes=<bytes>,
            filename=<my-pdf-name>,
            resolution="low",
            mime_type="application/pdf",
        )
        
 # When reading from a local file:
my_pdf = dspy.File.from_path(
            <my/pdf/path>, 
            mime_type="application/pdf", 
            resolution="low"
        )

I have to admit that I still don't finish understanding the implications of the 3.4.X update at the deeper level and my changes cover just the dspy.File contract.

I just added the LiteLLM-required fields ('format' and 'resolution') and it works fine on my side and passes the tests.

This fix is a modern solution (3.4.X) for the issue raise in this PR #9114 .

Please feel to free to reject this PR and let me know if more attention is needed on this.

Cheers,
Gustavo

@greptile-apps

greptile-apps Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium impact] Adds optional file metadata fields to the file type.

Not safe to merge while file-ID metadata can cause a native request to fail before it reaches the provider.

Findings

  1. P1 File-ID Metadata Blocks Requests ▶
  2. P2 MIME File Copies Fail ▶

Summary

The PR adds MIME type and resolution metadata to dspy.File and its constructors. The latest revision updates a documentation dependency and adjusts interpreter timeout tests. No new findings are reported, but a previously reported file-ID metadata issue remains outstanding.

Reviews (5) · Last reviewed commit: "Merge branch 'stanfordnlp:main' into gem..." · Reviewed by Greptile

Comment thread dspy/adapters/types/file.py Outdated
Comment thread dspy/adapters/types/file.py Outdated
Comment on lines +91 to +94
if self.mime_type:
file_dict["format"] = self.mime_type
if self.resolution:
file_dict["resolution"] = self.resolution

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 File-ID Metadata Blocks Requests

When from_file_id sets a MIME type or resolution, format() emits format or resolution in the file block. Native conversion rejects either field, so a native Gemini request using the new metadata fails before reaching the provider. This must be fixed before merging.

Knowledge Base Used: Language model integration

Artifacts

Executable Gemini file-ID reproduction script

  • The authored script runs both revisions through the same DSPy call and mocks only the provider boundary, making pre-provider rejection observable.

Gemini file-ID run before the change

  • The captured baseline run shows a plain file ID reaching the mock provider while the constructor rejects the two optional arguments.

Gemini file-ID run after the change

  • The captured HEAD run shows both optional fields rejected during native conversion with zero provider calls.

View artifacts

T-Rex Ran code and verified through T-Rex

Comment on lines +204 to +205
if file_input.mime_type is not None:
result["format"] = file_input.mime_type

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 MIME File Copies Fail

File(existing_file) converts an existing mime_type into a format field, which the File model forbids. Copying a file with MIME metadata therefore raises a validation error instead of producing a usable copy. This is a non-blocking concern, but callers cannot reliably reuse those files.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Executed File constructor reproduction script

  • The authored script constructs and copies Files with and without a MIME type, asserting the observed constructor outcomes.

Constructor output without a MIME type

  • The executed control command copied an existing File without a MIME type successfully.

Constructor output with a MIME type

  • The executed command copied an existing File with a MIME type and captured a `format` extra-forbidden validation error.

View artifacts

T-Rex Ran code and verified through T-Rex

@tyrael147 tyrael147 Oct 12, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The File model has aalready been updated in this fix to inlcude mime_type and resolution fields.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. In the revision under review, File declares mime_type and resolution, but it does not declare a format field. format is only the outbound key produced by format(); model_config still has extra="forbid".

Therefore, File(existing_file) still fails when either new field is set: encode_file_to_dict() emits {"format": ..., "resolution": ...}, which is passed back into Pydantic, where format is rejected as an extra field. The copy path should preserve the model field name (mime_type) and translate it to format only when serializing for the provider.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that File now declares both fields; the issue is the conversion key, not the model definition. encode_file_to_dict() is used by File(existing_file) and returns {"format": ...} here, but format is not a File model field (extra="forbid"), so copying a file with mime_type still raises a validation error.

format should only be emitted by the adapter-facing format() method. The constructor normalization should preserve the model field name:

Suggested change
if file_input.mime_type is not None:
result["format"] = file_input.mime_type
if file_input.mime_type is not None:
result["mime_type"] = file_input.mime_type

The resolution mapping can remain unchanged. This lets File(existing_file) retain both mime_type and resolution, while File.format() still translates mime_type to LiteLLM’s format key.

@greptile-apps

This comment has been minimized.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants