Wesnoth test harness for AI development

[SeattleDad]

I'm pretty much a Newbie to Wesnoth, so I have a pretty basic question. I'm interested in testing out some recent research on Monte Carlo methods for game AI and am trying to figure out if Wesnoth would be a good fit for this. Basically what I need for this project is to be able to pit AI version 1 vs AI version 2 in batch (without sound, graphics, etc. so it will run very fast) and get a report of how often each version won. In particular, I'm going to want to have my AI play against the built-in AI (the RCA AI). Is there an architecture for doing this in Wesnoth? If so, could somebody please point me at something that could get me started? I'm not seeing how to do this at http://wiki.wesnoth.org/General_RCA_AI_Howto. Thanks.

[mattsc]

Check out the very end of WritingYourOwnAI: 'AI-specific parameters'. I think that's pretty much what you are asking about. I have not tried that myself though, so I don't know if that works with the current version.

[SeattleDad]

Thanks! That's a big help. I now see a similar hint over here: http://forums.wesnoth.org/viewtopic.php ... te#p365084

One other question. The high-level concept of a Monte Carlo method involves running a simulation of the game to explore the consequences of a decision. Is there a way, if you are on turn n, to simulate a move, get the opponent's response on turn n + 1, make a response on turn n + 2, and thus play it out a bit until turn n + k and then back up to turn n and make what you have now decided is the best move? I think if I could do that, then I'd be in business.

Thanks again.

[mattsc]

One other question. The high-level concept of a Monte Carlo method involves running a simulation of the game to explore the consequences of a decision. Is there a way, if you are on turn n, to simulate a move, get the opponent's response on turn n + 1, make a response on turn n + 2, and thus play it out a bit until turn n + k and then back up to turn n and make what you have now decided is the best move? I think if I could do that, then I'd be in business.

I assume you want this done automatically, many thousand or millions of times? If you'd only need a few trials so that you can do it manually, you just turn auto-saves on, play your k moves, and then reload. But assuming that that's not practical...

I don't think that you can save/reload automatically in pure AI vs. AI games. (Or if there is a way, I don't know about it.) You also cannot change the turn number backwards, AFAIK. What you could do, however, is save the state of the game at the beginning of turn n, play your k turns, and then restore that saved state on turn n+k. The turn number would be different, but everything else would be the same. Would that work? (Potential caveat: I don't know if there is a limit for the turn_number variable, so you might not be able to play this out indefinitely.) If so, here's an example of how you could do this (there are many options):

• Set up a 'side 1 turn' event that, at the beginning of each turn, stores all units, as well as the state of everything else that could change on your map (village ownership etc.)
• At the same time, a counter variable is initialized that is incremented each turn
• Set up a 'new turn' event that, once the counter reaches k, kills all current units, restores all stored units, and resets everything else on the map. ('new turn' is triggered before 'side 1 turn', that's why I chose those two; if you want something else, just choose a different event combination.)

This might not be very elegant, but I think it would work.

[SeattleDad]

Matt,

Thanks again for the tip and yes, the idea is to do perhaps 1000's of simulations to see which move is best.

I see where you're going with this. It sounds feasible. My main concern is on pure AI vs. AI games at the moment, but if I wanted to do AI vs. Human, wouldn't I face a problem where the user would see all the different trials simulations happening in the GUI or would there be some way to not show the simulation in the GUI while still showing the user the actual move that the Monte Carlo AI picked?

Thanks again,
Seattle

[mattsc]

if I wanted to do AI vs. Human, wouldn't I face a problem where the user would see all the different trials simulations happening in the GUI or would there be some way to not show the simulation in the GUI while still showing the user the actual move that the Monte Carlo AI picked?

Seattle: You mean during the turn on which the AI is currently working? That's not a problem. The RCA AI also has candidate actions for which it plays through a number of options and simulates the outcome, without the player ever seeing that. How exactly you do that in practice depends on how (and in which language) you're going to write your AI, but it shouldn't be a problem. You can work on a virtual map, or move the units around without updating the map, or something similar to that. Not all those options are possible in all AI languages (C++, Formula, Lua), but each of them should have at least one way to make this work.

[Boucman]

automated testing of AI has already been done once, so all the bricks are in place to do batch games in AI vs AI...

save/reload of games via AI or WML... I'm not sure if it has been done, but it might not be that complicated to add on the C++ side, and patches would probably be accepted in mainline.

a C++ patch might be simpler than a WML or AI work around