how about Wesnoth AI contest?

[silly_sad]

what steps shall be taken to make it possible and open for general public?

[Iris]

This doesn’t look like a technical support question, so... moved to Users’ Forum for now.

You might want to elaborate further.

[silly_sad]

first and foremost, the API manual must be published.
second, some flexible model of communication/interection must be proposed.

for now i have no ideas.
wanted to discuss if such contest is possible at all.

P.S.
ah, and i have posted it to the technical subforum, because the discussion will inevitably be technical.

[Turuk]

These two things seem to be at odds:

first and foremost, the API manual must be published.
second, some flexible model of communication/interection must be proposed.

for now i have no ideas.

It's certainly possible to work towards some sort of competition to create a better AI, but you would have to provide some sort of further detail on the structure of what you expect people to do.

If the discussion becomes technical it can be moved again, but right now it's all theoretical.

[silly_sad]

[quote="Turuk"]
It's certainly possible to work towards some sort of competition to create a better AI, but you would have to provide some sort of further detail on the structure of what you expect people to do.[/quote]

i want competitors to do AI that plays and wins and complies with wesnoth specs.

[Iris]

Well, before you posted this including the emboldened magic word, there was no way to be sure this was an AI-making contest proposal:

first and foremost, the API manual must be published.

Anyway, now that this point is clearer, I’m moving the topic to Coders’ Corner, which is a far more appropriate place than Technical Support if you care to read both forums’ descriptions.

Still, you should be prepared to elaborate further on your idea if you expect it to work. For example, since it’s a contest, there obviously have to be rules and some set of criteria to designate the winner(s), and reward(s) for them. And of course, somebody in charge of organizing all this — the obvious assumption is that person will be yourself. But before any of that is possible, as you said yourself, we need documentation. Have you taken a look at the source code (for C++ AI engines, especially src/ai/) and the LuaAI page in the wiki (for Lua AI scripting) to see if the existing documentation suffices? It’s also a good idea to take a look at how the existing AI engines are implemented.

[silly_sad]

Still, you should be prepared to elaborate further on your idea if you expect it to work.

i am not sure if it is possible at all, this is why i ask questions rather than answer them.
at first glance we have certain problems that "The Rules" must have solved:
-- ensure different language/technique/engine AIs running flawlessly
-- safeguard against malicious and brocken AI modules
-- (as opposed to the most obvious and natural solution to the previous problems (which is of course run the AI remotely)) ensure that it is not the human makes moves.

well, we have a contradiction here. any ideas?

[Turuk]

It's certainly possible to work towards some sort of competition to create a better AI, but you would have to provide some sort of further detail on the structure of what you expect people to do.

i want competitors to do AI that plays and wins and complies with wesnoth specs.

AI improvement is always a great goal, but as noted by shadowm, how will you contest run?

Still, you should be prepared to elaborate further on your idea if you expect it to work.

i am not sure if it is possible at all, this is why i ask questions rather than answer them.
at first glance we have certain problems that "The Rules" must have solved:
-- ensure different language/technique/engine AIs running flawlessly
-- safeguard against malicious and brocken AI modules
-- (as opposed to the most obvious and natural solution to the previous problems (which is of course run the AI remotely)) ensure that it is not the human makes moves.

well, we have a contradiction here. any ideas?

This all seems to cover the very basics, and does not help explain how the actual contest will work?

[zookeeper]

What we need first and foremost is someone who's capable and willing to organize such a contest. Speculating on the details of the contest is fine but ultimately that's not going to do much unless there is an organizer.

The way I'd see a contest like that working: we'd come up with a list of balanced multiplayer maps which would be used. Once the deadline is over, the organizers would pit the submitted AI's against each other and record the results (possibly the matches could be on the MP server so everyone could observe in real time). There's no reason to involve the AI writers in the actual running of the matches, and it's a whole lot simpler if it's all performed by the organizers. The matches would be run on a predefined version of Wesnoth and it'd be up to the AI writers to make sure their AI performs on that version. Rules would of course disallow things like conditional handling of specific maps (because we want the contest to result in good general-purpose AI's).

[Turuk]

An excellent point. I would be willing to help organize, though my only concern is whether there are enough people interested in writing an AI to make it a contest?

[iceiceice]

Some comments on the technical aspects:

Right now there are mainly two kinds of AI you would want to consider (afaik):

- Lua-based AIs. These are very easy for you -- you can just ask people to submit their lua ai folders / post add-ons containing this on the add-on server, and run them against eachother, or use the standard AI testing command line arguments to play them against eachother many times on whatever maps you like.
- C++-based AIs. This is much more difficult.

With C++ based AIs, if you want to run it locally you have to recompile the game. You have two options -- (1) take all the submitted C++ AIs, put their source code together and try to compile the whole thing (and hope that it works!) or (2) compile each C++ AI separately and use a local wesnothd so that they can play against eachother. This is perhaps workable but will take much longer and is still pretty annoying.

I thought about this some time ago when I was working on an AI project, I think actually the best thing is if we code up some kind of "AI bot" that can join the official mp server, and then respond to commands. So that other players can ask the bot to join their game and play against them, or the tournament organizer can ask two bots to play many games against eachother. I think this would be a good thing outside of tournaments, as it can make it easier for you to get people to help test your AI. But this doesn't exist yet.

Also, I want to point out an email from the dev list in February, apparently there was some kind of wesnoth ai tournament on the internet at some point (?), so you might want to see how they ran it.

https://mail.gna.org/public/wesnoth-dev ... 00081.html

[tekelili]

Just some common sense coments:

Without any limit or puntuation over computing time, contest would reward overcomplicated algorithms with little perfomance improve.

Try to forbid expert knowled of maps inside code looks impossible to me. An AI code could have hidden expert knowledge (even not done in propose) in terrain configuration evaluation. Said in other way: If AI code decide do something when some village pattern is find, and that pattern is only present in one contest map, then is like use expert knowledge.

Be aware of "dirty outputs". Such a contest would give the best AI for a limited enverioment, that could not be the "best desirable" AI. In other words, we could have as winners ultradefensive AI or AI unable to perform decently at different scale than contest playground.

Just my 2 cents.

Edit: Imho the best aim for such a contest would be do a lot of "micro-competitions" as factions match ups or single map play ground. Final task would be integrate all winnings AI to perform at such limited enverioments with the best one.

[zookeeper]

Right now there are mainly two kinds of AI you would want to consider (afaik):

- Lua-based AIs. These are very easy for you -- you can just ask people to submit their lua ai folders / post add-ons containing this on the add-on server, and run them against eachother, or use the standard AI testing command line arguments to play them against eachother many times on whatever maps you like.
- C++-based AIs. This is much more difficult.

With C++ based AIs, if you want to run it locally you have to recompile the game. You have two options -- (1) take all the submitted C++ AIs, put their source code together and try to compile the whole thing (and hope that it works!) or (2) compile each C++ AI separately and use a local wesnothd so that they can play against eachother. This is perhaps workable but will take much longer and is still pretty annoying.

I think it's a given that any contest we could have would have to be non-C++ AI's only. It'd be way too cumbersome to handle C++ AI's, and frankly, there's no one who'd write good ones anyway.

[iceiceice]

I think it's a given that any contest we could have would have to be non-C++ AI's only. It'd be way too cumbersome to handle C++ AI's, and frankly, there's no one who'd write good ones anyway.

In the near term, I think that's exactly right.

In the long term though, there was a recent post in Coder's Corner from someone working on a C++ ai, so maybe they would like to participate. Also the AI-related GSOC is in C++ iirc? Also someday I might be able to write my montecarlo AI in C++ after enough engine refactor has happened to really make that possible. That's the plan anyways

[Turuk]

Try to forbid expert knowled of maps inside code looks impossible to me. An AI code could have hidden expert knowledge (even not done in propose) in terrain configuration evaluation. Said in other way: If AI code decide do something when some village pattern is find, and that pattern is only present in one contest map, then is like use expert knowledge.

That is very true, but this would quickly be discovered when the AI is tested across multiple maps (it would perform poorly on any of the others). It would be a waste of someone's time to try to beat the contest this way, as this issue would soon be discovered, but it is of course a good rule still.

Be aware of "dirty outputs". Such a contest would give the best AI for a limited enverioment, that could not be the "best desirable" AI. In other words, we could have as winners ultradefensive AI or AI unable to perform decently at different scale than contest playground.

Agreed, but that would also be vetted through a well-designed testing process that would ensure the AI would actually play competitively and not just wait a player out.