Issue and solution with Ollama provider

[benjaminpillot]

Hi there,
thank you very much for that awesome package !
For your information, regarding the Ollama provider, when performing title/abstract review, an issue raises with most of the models I have tested (llama3.1, mistral, llama3.2-vision, etc. ), returning a KeyError due to an empty json output. Actually, it seems response_format is given to system prompt in both the _build_system_prompt method (line 98) of the BasicReviewer class and in the get_json_response method of the OllamaProvider class (line 98 to 104), causing some kind of conflict, which results in the model returning schema_str itself rather than a response complying with it.

By commenting the previous lines in get_json_response of the OllamaProvider class (where the system message is actually updated to request JSON object), it eventually works perfectly (the model returns a dict with both reasoning and evaluation score)

I'm not sure why the response format was defined twice in the first place. The JSON format is already given through cleaned_kwargs in get_json_response (line 108) and used as it by the underlying chat method of the AsyncClient class in the _fetch_response protected method. I think the definition is redundant, but do not really see why it would make some of the ollama models (and not all of them --> llama3.2) return schema_str or an empty json object.

System characteristics : Ubuntu 22.04, Python 3.12, CPU/GPU

I hope this feedback will help.

Benjamin

[PouriaRouzrokh]

Thank you Benjamin for your kind words and detailed logs.

I believe those lines were legacy from the early dev time where I was passing the structured outputs differently. I commented us those lines as you said and will remove them later if nothing breaks. Can you please update the package in your env and recheck this to make sure the issue is now fixed?

Thanks, Pouria!

[benjaminpillot]

Hi Pouria,

I'm not sure commenting did actually solved anything. The error continues to "randomly" raise, either with the "reasoning" keyword or with the "evaluation" keyword". I think the trick is in the way the response is formatted by the model. So I'm quite perplex.

For the moment I will dig further. Let me know if you can reproduce the error on your side

Benjamin

[PouriaRouzrokh]

Sorry that the issue persists. I did try to reproduce it but it seems to run fine on my mac.

Please check the following tutorial notebook that I just updated: tutorials/base_functionalities
If you scroll down to the last cell, you see I can use gemma3 running on ollama for reviewing a CSV file.

Can you try running this notebook on your end? If you run into any issues, please give me the logs / screenshot of erros, etc.

Best,
Pouria

[benjaminpillot]

Hi Pouria,
after digging a little bit, it seems the ollama-python accept JSON format in the following way (see here):

from ollama import chat
from pydantic import BaseModel

class Country(BaseModel):
  name: str
  capital: str
  languages: list[str]

response = chat(
  messages=[
    {
      'role': 'user',
      'content': 'Tell me about Canada.',
    }
  ],
  model='llama3.1',
  format=Country.model_json_schema(),
)

country = Country.model_validate_json(response.message.content)
print(country)

I would recommend commenting both dict and JSON system prompt in ollama_provider and basic_reviewer, and try the previous hack. I did it here and it worked. Essentially, the structure is:

class Review(BaseModel):
    evaluation: int
    reasoning: str

cleaned_kwargs["format"] = Review.model_json_schema()
response = await self._fetch_response(message_list, cleaned_kwargs)

Benjamin