Evaluating (open) LLMs for more languages using translated prompts
As part of MPEP, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language. We can do other things with these translated prompts, but our first next step is to use them to evaluate the performance of LLMs for a particular language.
Does LLM as a judge work outside of English?
Many people use LLMs as judges to evaluate the performance of open LLMs without requiring human input. https://github.com/tatsu-lab/alpaca_eval is a well-known example of this approach, and they have demonstrated high agreement with human rankings. However, most of these approaches are targeted towards evaluating models in English. What happens when we want to evaluate non-English models?
Ideally, it would still be compelling to leverage LLMs to judge models for non-English since this lowers the barrier to evaluating models significantly (although it doesn't remove this barrier altogether).
What we want to know is:
- does auto/LLM eval work in general for a particular language
- which model(s) works best as a judge
- do LLMs judgements for non-English models match with human preferences?
A possible approach
For each language with 500 translated prompts we roughly want to do the following:
- Evaluate N top models for that language (based on existing leaderboards, vibes, community knowledge etc.)
- For every 500 prompts generate a response from each of the candidate models being evaluated
We can then do the following:
- Use an LLM to judge the responses of each of these models (ideally we can start using an open LLM for this eventually)
- Create a new annotation task in Argilla to compare responses i.e. model 1 vs model 2
- We can then compare the human to the LLM rankings to see how/if LLMs work well as a judge
Open questions?
- Which model(s) to to use as a judge?
- How many models to use as candidates to evaluate (can be decided by each language lead/group IMO)
- How best to do the annotations. Ideally, we want to be able to compare human and LLM judgments, but if we're comparing many model outputs, it might be tricky for a human to compare them all directly.
Other ideas
Could an approach like Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models with the same SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge for Arabic?
Evaluating (open) LLMs for more languages using translated prompts
As part of MPEP, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language. We can do other things with these translated prompts, but our first next step is to use them to evaluate the performance of LLMs for a particular language.
Does LLM as a judge work outside of English?
Many people use LLMs as judges to evaluate the performance of open LLMs without requiring human input. https://github.com/tatsu-lab/alpaca_eval is a well-known example of this approach, and they have demonstrated high agreement with human rankings. However, most of these approaches are targeted towards evaluating models in English. What happens when we want to evaluate non-English models?
Ideally, it would still be compelling to leverage LLMs to judge models for non-English since this lowers the barrier to evaluating models significantly (although it doesn't remove this barrier altogether).
What we want to know is:
A possible approach
For each language with 500 translated prompts we roughly want to do the following:
We can then do the following:
Open questions?
Other ideas
Could an approach like Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models with the same SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge for Arabic?