Feature proposal

[radajakub]

Hi @lanctot, I have a feature proposal to be integrated to OpenSpiel.

For the past few months, I have been working on a benchmarking framework for LLMs, which uses OpenSpiel under the hood. The goal was to build a unified interface that would transform OpenSpiel's states and actions to something the LLMs would understand in prompts and could play by. Using this I measure how well the LLMs play simple games (tic_tac_toe, nim, connect_four) against fixed opponents (random, mcts, stochastic minimax). The results were not very good, e.g. gpt-5.4 (high) lost to a random opponent on tic_tac_toe (full results published here). To find out whether the models are truly reasoning about the states or just recalling observed strategies from training, I also introduced "obfuscations", which are alternative rule, state and action formulations that preserve the game tree but try to prevent the LLM from immediately recognizing the game.

Now, regarding the feature proposal. I published this as an installable package (GitHub repo), however then I noticed that OpenSpiel 2.0 came out alongside with the structured representations directly aimed at evaluating LLMs, thus overlapping with my package in some areas. As it's much more mature codebase than mine, I am wondering whether you'd be interested in integrating some of the features into OpenSpiel. If yes, I would be more than happy to implement them and draft a PR. Specifically, I currently have two features in mind:

1. Add stochasticity to the minimax player.
   To make the LLMs see more diverse states, I modified the minimax player to always collect all moves with the highest value and then sample from them uniformly at random. That way even when two minimax players play against each other, the games are different. I found that it sometimes exposes the LLMs performing worse on less conventional states.

2. General mechanism for the obfuscations.
   I think it could work like an additional layer between the base engine and the input for the LLMs (that's how it's in my project). I defined each obfuscation as a head prompt, where the modified rules are explained, and two transformation functions mapping states and actions to new states and actions. In my case they are mappings from a board states and numbers to strings, in OpenSpiel it could be a mapping from JSON to JSON. I think it could be useful addition for studying LLMs in games, if that is the direction OpenSpiel wants to aim to. The only downside I can think about is that by publishing the changes in a well-known project, that is probably indexed and trained on, the unknown element of the obfuscations will be lost. However, I believe it could be introduced as a general transform mechanism with the specific instances kept outside of OpenSpiel repository and left up to the user. This would ensure that it's consistent for all users and potential papers using OpenSpiel.

What do you think? Let me please know, if you'd be interested in the features I mentioned. We could also discuss other features, if relevant to OpenSpiel. I would of course maintain this part of the codebase as I don't want to add additional work for you and your team.

[lanctot]

Hi @radajakub , sorry for the delayed reply! Looping in @jhtschultz.

Sounds interesting. I think also you should check out the Kaggle Game Arena as many of the OpenSpiel changes were inspired by the use cases in there. In particular, you can maybe check out their code to see the kind of harnesses that were used for the LLMs and how they wrapped the OpenSpiel games.

Point 1 is not controversial, that would be a generally useful addition. I'd recommend making it an option (enabled by default).

Point 2 is great, having a general way to do it would seem useful, and then maybe a few examples that people could easily override to suit their use cases. I'd love John's take on that one. :)

[jhtschultz]

Hi @radajakub,

Agreed on point 1, seems like that should be default behavior.

For point 2, I'm not sure these transformations belong in OpenSpiel itself. The idea behind the JSONs was to give users an easily manipulable state. Any transformations on top of that are best applied by individual users. If there's some gap in functionality that I'm missing here that would be helpful, do let me know.

You can check out the exact prompts and harnesses used for the Kaggle Game Arena here: https://github.com/Kaggle/kaggle-environments/tree/master/kaggle_environments/envs/open_spiel_env. There's also a game-proxy pattern there which might be worth a look. The proxies were more useful before we had the JSON structs, but you might still find them helpful if you need to override some aspect of a game's default behavior.

[radajakub]

Hi @lanctot and @jhtschultz,

thank you for the responses!

I will prepare a PR for point 1 then.

Regarding point 2, I definitely agree that concrete transformations don't belong to OpenSpiel. They are not novel games and as I mentioned, publishing them might not be a good idea for keeping the potential benchmarks secret. However, I could see value in providing a general way that would enable the users to inject a mapping between the canonical state and the "obfuscated" one. It's slightly different from what the Kaggle Game Arena is doing (as far as I understand it). They allow for manipulating prompts to see the model's sensitivity to instructions. What I'm proposing is different in that it transforms only the surface level of the game, not the prompts. These I would of course keep up to the user. I will try to explain it on an example.

Let's consider standard tic tac toe on a 3x3 grid. What I do is that I provide mapping from the 3x3 board to a Magic Square (https://en.wikipedia.org/wiki/Magic_square). This transforms the game from selecting cells on a grid to picking a number from a set of numbers until one of the players has three numbers summing to 15. I do this by adding an abstraction layer between OpenSpiel and the LLM harness itself. My proposal would be to add a mechanism, that would just integrate it more tightly to the engine so that if the user chooses to supply such transformation, the methods state.observation_string(), state.legal_actions(), state.apply_action(action), etc. return the mapped versions directly.

I totally understand if you'd not want it in OpenSpiel, it is just some usecase that I found and had to implement some infrastructure for.

[jhtschultz]

Isn't this just an instance of a game transform? open_spiel/game_transforms

[radajakub]

I think that it could be done as a game transform. Resp. using the WrappedState and additionally implementing the ToJSON() methods. I thought of these as transforming the game itself rather than just modifying the representation, but this would work.

[radajakub]

Hi @lanctot, @jhtschultz sorry it took so long, I was at IJCAI and then had to catch up with other stuff. I finally managed to submit a PR regarding the point 1 (#1605). I'll be glad if you'll take a look, I'll fix any issues.

[radajakub]

Some additional tests have been run today. I found one bug and fixed it. It was due to the algorithm now returning an object instead of an integer. Now it locally passes.