Add note in Readme about different API use with OpenRouter connection types?

[Jonseed]

In the README.md in the section how to connect to OpenRouter, you might consider adding a note that if a user chooses to use the OpenAI API Connection then any parameters added must follow the OpenAI API spec, and if they are using the Direct Web Connection then any parameters added must follow the OpenRouter API spec (unified API). This difference in API types is also noted in the OpenRouter docs here. This tripped me up for a minute when adding structured output parameters for OpenRouter.

[glibsonoran]

Open Router has become popular enough that I was thinking about adding it as a first class AI_Service like Anthropic and Groq. But I wonder which type of connection would be best to choose if I do, and presenting the user with that overwhelming list of models... I dunno.

Does this difference in parameters affect anything beyond response_format?

[Jonseed]

Yeah. OpenRouter is an easy way to access many different models with different architectures under a unified API (which is why I'm using it instead of directly using Gemini via Google's API). But that unified API is only accessible via the Direct Web Connection. It seems there are more capabilities with the Direct Web Connection (such as structured outputs with JSON schema), than with the OpenAI API, but the OpenAI API is still quite popular. Maybe you could just have two first class options: OpenRouter (OpenAI API), or OpenRouter (unified API). For most purposes, users probably wouldn't care one way or the other, since it seems to only make much of a difference when you are adding parameters. I don't know how I would format the parameters for the Gemini API for structured output, or if that is more like the complex structured outputs with OpenAI, which is why the unified API is nice. The unified API standardizes and hides all those differences between the APIs. Which makes me wonder if you can get structured outputs with provided JSON schemas if you use the unifed API with OpenAI models?

[Jonseed]

Ha! Indeed you can. You can get the complex structured outputs (using a JSON schema) from OpenAI's models if you use the unified API via OpenRouter (Direct Web Connection). That's pretty cool.

(Edit: although it didn't follow my text prompt to write a story about the provided image, which is odd... It just described the image in the returned JSON. I did notice in the Gemini API docs that "When using a single image with text, place the text prompt after the image part in the contents array." That makes me think that the order that the image and text prompt are provided to the APIs matters, and if it will tend to ignore the text prompt if it receives the image after the text prompt?)

[Jonseed]

It seems the order does matter, but it might depend on the particular API/model. It seems some AI models, like Claude and Gemini, recommend placing the image before the text for optimal results, but OpenAI's API examples often show text-first ordering.

[Jonseed]

I added a note in my JSON schema for the "Description" value to follow the instructions in the prompt when writing the story, and that seems to have made the model write a story rather than just describe the image. Maybe I should change the key to "Story" rather than "Description" to help this along.

[glibsonoran]

It would be a while before I would be able to do that. But it's on my radar.