Machine Learning Recruiter

[mattsc]

So after a lot of questions to the community, I think I have something to contribute now.

Nice. You probably have seen on the forums that the topic of the AI recruiting comes up from time to time, so it's good to see that you are working on this. You might also know that one of the approved Wesnoth Google Summer of Code projects is about improving AI recruiting. You might want to get in contact with the people working on that.

I don't have too much to say about what you posted (I don't know that much about it and I have no say in what patches should be included etc.). It looks good in general and will be interesting to see what you come up with from your ML experiments. One comment on your patch #6 though. I don't think that including the macros like that is the right thing to do here. (And I know that that was originally my suggestion...) First, it includes all the macros, while you only need CA macros. Using only {core/macros/ai_candidate_actions.cfg} (untested) should work with much less overhead. Second, ai/ais/testing_ai_default.cfg is useful for including in other places also, where you already have access to the macros, so there would be duplicate code in that case. I think another solution would be better. For example, creating another file which includes both the macros and testing_ai_default.cfg, but there are other options as well.

As a side note, I am pretty sure that you do not need to include ai/ais/testing_ai_default.cfg as AI algorithm on the command line, as this is the default.

Finally, keep working on this and don't be discouraged by the lack of replies. That is quite common with "difficult topics" like this and doesn't mean that nobody is interested. If you keep posting when you make progress, you will notice in time that quite a few people follow what you are doing.

[SeattleDad]

Thanks Matt and everyone else for your comments.

I have a high-level plan on how this will work, but it involves an open-source C++ ML package. In brief, I'd like to do most of the complicated computation in Lua to compute the inputs to the decision-making process (e.g. which types of friendly and enemy units are currently on the map) and then pass an array to the C++ ML package, which can make the final decision. I'm thinking I can pass the data in with helper.set_variable_array(), but could somebody point me to a class on the C++ side that could read that array?

Thanks,
SeattleDad

[SeattleDad]

I'm having some trouble with my AI trying to recruit onto the half hexes on the edge of the board. I identify a location as being valid as follows:

Code: Select all

        local terrain = wesnoth.get_terrain(loc[1], loc[2])
        if wesnoth.get_terrain_info(terrain).castle or wesnoth.get_terrain_info(terrain).keep then
             -- Recruit the unit

The problem is that it's identifying tiles which are half-hexes as being eligible for recruiting. Since no units can go on these hexes, I get an error when I try to recruit there.

Anybody have any ideas?

[mattsc]

There are a number of possibilities. I'd probably try to select the hexes considered (loc[1],loc[2]) so that the border is not included in the first place. But I don't know how you have that set up, so I cannot tell you how to do that here. If you don't want to or cannot do it that way, you could get the map size using wesnoth.get_map_size and add conditions that x and y coordinates need to be >0 and <= width or height. Or you could use wesnoth.find_path to see if a side unit (e.g. the leader) can get there. There are a couple other options also, but I'd probably go with the map_size option, if you cannot exclude the border locations in the first place.

[SeattleDad]

Thanks Matt. That worked. I was hoping there would be more direct way of testing whether a hex is a half-hex, but this does the job.

[SeattleDad]

So I'm working on my final checklist for posting a patch with my new machine learning-based general purpose recruiting algorithm and I had a couple of questions:

1. The machine learning models are stored in .json files which I am passing into C++ code that executes the models at run time. I've got this all working when I pass in an absolute path from the Lua to the C++, but this should be using a path relative to wesnoth/data. I guess what I need to know is how to get the absolute path of wesnoth/data and then I could append the relative location of the model file onto that.
2. Is there a way to pass in arguments from the command line directly into Lua for a multiplayer nogui game? In particular, I want to use the --parm argument. Is there some way to access that in Lua?

Thanks!
SeattleDad

[lipk]

Umm, I can't speak for the development team, but I see some oddities in this project. First and foremost, does this "learning-based" algorithm mean that the AI's behavior would change over the time? I don't think that is desired, as it could possibly break the balance of campaigns. If they could be balanced at all, regarding that all clients would have different AIs. Secondly, did you make sure that the algorithm works with potentially *any* kind of recruit list? An AI usable only with the standard MP factions wouldn't be that much of an improvement in my opinion.

On the technical side, am I right that you're about to introduce a new package dependency? I'm even less qualified to comment on this topic, but I guess you won't be allowed to do that without a *very* good reason. That is, are you sure that you couldn't use standard WML instead of json?

1. The machine learning models are stored in .json files which I am passing into C++ code that executes the models at run time. I've got this all working when I pass in an absolute path from the Lua to the C++, but this should be using a path relative to wesnoth/data. I guess what I need to know is how to get the absolute path of wesnoth/data and then I could append the relative location of the model file onto that.

There're some handy functions in src/filesystem.?pp, including one which returns the data directory's path.

2. Is there a way to pass in arguments from the command line directly into Lua for a multiplayer nogui game? In particular, I want to use the --parm argument. Is there some way to access that in Lua?

I don't think so, all the parameters are handled by C++ afaik.

EDIT: and sorry if those questions are already answered, I'm just lazy as always to read through all that text...

[SeattleDad]

Umm, I can't speak for the development team, but I see some oddities in this project. First and foremost, does this "learning-based" algorithm mean that the AI's behavior would change over time?

No, there is a distinct development cycle when the machine learning algorithm builds the model and writes that out to a .json file. In game play, the algorithm's behavior is dependent on the .json file and thus doesn't change from player to player.

Secondly, did you make sure that the algorithm works with potentially *any* kind of recruit list? An AI usable only with the standard MP factions wouldn't be that much of an improvement in my opinion.

The first patch I'm going to post will be something of a proof of concept. It will have the following limitations:
1. Only works on two-player MP games
2. Tuned to work on just four two-player maps, although I'm guessing that it would do reasonably well on other maps
3. Will work on any recruit list that is a subset of the recruit lists of one of the factions in MP

I think the architecture is reasonably clean, so my expectation is that this recruiter could eventually be used in a wide variety of situations. In particular, I think that it should be fairly straightforward for campaign designers to build recruiters customized to their scenarios and all of the above limitations are easy to fix.

On the technical side, am I right that you're about to introduce a new package dependency? I'm even less qualified to comment on this topic, but I guess you won't be allowed to do that without a *very* good reason. That is, are you sure that you couldn't use standard WML instead of json?

I'm bringing in code from an LGPL package, but I'm directly incorporating only the code needed for runtime compatibility into Wesnoth, so the user won't have to install anything else. The json is essential for compatibility with the package.

1. The machine learning models are stored in .json files which I am passing into C++ code that executes the models at run time. I've got this all working when I pass in an absolute path from the Lua to the C++, but this should be using a path relative to wesnoth/data. I guess what I need to know is how to get the absolute path of wesnoth/data and then I could append the relative location of the model file onto that.

There're some handy functions in src/filesystem.?pp, including one which returns the data directory's path.

Thanks. get_wml_location() from filesystem did the trick.

2. Is there a way to pass in arguments from the command line directly into Lua for a multiplayer nogui game? In particular, I want to use the --parm argument. Is there some way to access that in Lua?

I don't think so, all the parameters are handled by C++ afaik.

If there's no way to access this in Lua, could someone point me to the C++? I think I know enough about calling C++ from Lua now that I could make this work.

[lipk]

The first patch I'm going to post will be something of a proof of concept. It will have the following limitations:
1. Only works on two-player MP games
2. Tuned to work on just four two-player maps, although I'm guessing that it would do reasonably well on other maps
3. Will work on any recruit list that is a subset of the recruit lists of one of the factions in MP

I think the architecture is reasonably clean, so my expectation is that this recruiter could eventually be used in a wide variety of situations. In particular, I think that it should be fairly straightforward for campaign designers to build recruiters customized to their scenarios and all of the above limitations are easy to fix.

Then am I right in my guessing that every custom setup using this algorithm would require its own data pile? No matter how easy it would be to build that, this would mean that your algorithm can't be a full replacement for the standard one. I know that it's very unfortunate to ask for structural changes in such a late stage of the development, but it'd be cool if you could make it so that the algorithm could make generalizations from the results; i.e. it wouldn't think like "hey, I'm on Oni's Den playing Knalgans against Undead, then I'll need some Dwarvish Fighters" but rather "I'm on a small map playing against a faction mostly vulnerable to impact damage, so I'll need an impact capable warrior, no problem if it's slow". If it already does this way, then sorry, it somehow isn't clear to me...

If there's no way to access this in Lua, could someone point me to the C++? I think I know enough about calling C++ from Lua now that I could make this work.

Have a look at src/game.cpp.

[SkeletonCrew]

On the technical side, am I right that you're about to introduce a new package dependency? I'm even less qualified to comment on this topic, but I guess you won't be allowed to do that without a *very* good reason. That is, are you sure that you couldn't use standard WML instead of json?

I'm bringing in code from an LGPL package, but I'm directly incorporating only the code needed for runtime compatibility into Wesnoth, so the user won't have to install anything else. The json is essential for compatibility with the package.

I also wonder why JSON instead of WML. We are somewhat reluctant to add new
dependencies. Adding code of another project is only done when _really_ needed.
This has nothing to do with licensing, but with maintenance. When you copy code
you need to maintain it as well, fetch upstream fixes, new releases, bugs etc.

Wesnoth already has good support for WML, also external tools for parsing WML.
What would be the advantage for Wesnoth to use JSON in this case?

[JaMiT]

2. Is there a way to pass in arguments from the command line directly into Lua for a multiplayer nogui game? In particular, I want to use the --parm argument. Is there some way to access that in Lua?

I am not familiar with what Lua can access, but I happened to look through the handling of --parm (in the C++ source) recently. Well, more accurately, I looked through code that happens to include the handling of --parm, so I easily could have missed something key in another section of the code. If I read the code right and did not miss anything important, the --parm arguments get translated into the side definitions. Specifically, "--parm 1:village_gold:2" should have the same effect as including (or overwriting) the line "village_gold=2" in the [side] tag defining side 1 (in the multiplayer scenario's WML file). That same line also gets included in an [ai] subtag of [side]. This [ai] subtag would be appended after any earlier subtags.

For example, "--parm 1:village_gold:2 --parm 1:what:ever" should be like editing the WML definition of the scenario as follows:

Code: Select all

    [side]
        side=1

        # The side definition as it already existed, minus village_gold=

        village_gold=2
        what=ever
        [ai]
            village_gold=2
            what=ever
        [/ai]
    [/side]

So if Lua can access attributes set in either [side] or [side][ai], you should/might be able to pass arbitrary data with --parm. Unless other parts of the code trim off unrecognized attributes before Lua would have a chance to look at them.

[SeattleDad]

So if Lua can access attributes set in either [side] or [side][ai], you should/might be able to pass arbitrary data with --parm. Unless other parts of the code trim off unrecognized attributes before Lua would have a chance to look at them.

Thanks for the tip. I've given this a try and unfortunately adding:
--parm 1:seattle:hello
gives me the error message
20120709 22:44:31 warning unit: Unknown attribute 'seattle' discarded.

Which seems to be coming from this spot in unit.cpp

Code: Select all

        BOOST_FOREACH(const config::attribute &attr, input_cfg.attribute_range()) {
                if (attr.first == "do_not_list") continue;
                WRN_UT << "Unknown attribute '" << attr.first << "' discarded.\n";
        }

On the other hand, I can seem to reset all of the previously defined side attributes. For instance if I add the argument "--parm 1:user_team_name:hello" to a nogui command line call to Wesnoth, it does the right thing and I can access this in Lua as follows:
wesnoth.sides[1].user_team_name

So as a hack for doing command-line testing, I could pass in an arbitrary parameter through user_team_name since I don't think that's used for anything that would affect the outcome of a game. Still, it would be nice if I could pass in arbitrary attributes or there was even one free parameter called, for instance, "dev_parm", that could be used for this kind of thing.

[SeattleDad]

Then am I right in my guessing that every custom setup using this algorithm would require its own data pile? No matter how easy it would be to build that, this would mean that your algorithm can't be a full replacement for the standard one. I know that it's very unfortunate to ask for structural changes in such a late stage of the development, but it'd be cool if you could make it so that the algorithm could make generalizations from the results; i.e. it wouldn't think like "hey, I'm on Oni's Den playing Knalgans against Undead, then I'll need some Dwarvish Fighters" but rather "I'm on a small map playing against a faction mostly vulnerable to impact damage, so I'll need an impact capable warrior, no problem if it's slow". If it already does this way, then sorry, it somehow isn't clear to me...

So a quick preview of the experiments I've been running is that the ML Recruiting algorithm defeats the current recruiting algorithm by a wide margin in AI vs AI play when run on a large number of trials in nogui mode. The algorithm is aware of which faction it's fighting and which units it's fighting. In contrast to the current recruiter it knows nothing about the capabilities of a "Skeleton Archer" as an enemy unit or the capabilities of a "Wose" as a friendly unit, for instance. It just learns that when there are a lot of enemy Skeleton Archers, a Wose tends to perform well, so it recruits more of them. So in addition to the recruiters which work well in 2-player mode, I think we could build a recruiter which would work well against any unit used in multiplayer game by training it on maps where it is playing against multiple factions. It would then learn how to fight in a situation where it has both a Skeleton Archer and a Dwarvish Fighter as an opponent. For units it's never encountered before, we could handle it in a few different ways:
1. Use the current recruiter in this situation. Over the short to medium term, I think they'll both exist alongside each other. Campaign builders will have a choice. This will also avoid issues with play-balance, although campaigns could consider using the ML Recruiter at higher difficulty levels.
2. We get somewhat sub-optimal performance from the ML Recruiter, but still maybe better than the current recruiter
3. The campaign builder trains a recruiter that incorporates the new unit and includes it as a resource with the scenario. This sounds scary, but is pretty easy with the available tools.
4. A little trickier, but we could probably build a recruiter which models unknown enemy units by weapons and resistances as you suggest. The idea would be to train it on some scenarios where it has some unknown units mixed in with the known units so that it could train the parameters correctly.

[SeattleDad]

Wesnoth already has good support for WML, also external tools for parsing WML.
What would be the advantage for Wesnoth to use JSON in this case?

It's needed for compatibility with the ML toolkit I'm using. The toolkit reads and writes models as JSON. I guess I'm hoping that this the community will think that this patch is cool enough to consider allowing machine learning models written out as JSON into Wesnoth. I would argue that the benefits to opening up Wesnoth to ML approaches could be really big. It could open up the game to work by a lot of machine learning researchers like myself who could hopefully do a lot of interesting work on the AI.

[lipk]

--parm 1:seattle:hello
gives me the error message
20120709 22:44:31 warning unit: Unknown attribute 'seattle' discarded.

I guess "arbitrary" in this case means "anything you could put in a side definition in WML". "Seattle" is not something like that.

So a quick preview of the experiments I've been running is that the ML Recruiting algorithm defeats the current recruiting algorithm by a wide margin in AI vs AI play when run on a large number of trials in nogui mode.

The problem is that AIs are usually meant to play against humans, and the success rate may greatly depend on your enemy's style. For example, you mentioned that even an AI recruiting solely Thunderers would beat the current one; I doubt however that it would be also more effective against a human. I don't say that your recruiter isn't better than the default, just that its performance versus an AI doesn't necessarily indicate its overall quality.

So in addition to the recruiters which work well in 2-player mode, I think we could build a recruiter which would work well against any unit used in multiplayer game by training it on maps where it is playing against multiple factions.

3. The campaign builder trains a recruiter that incorporates the new unit and includes it as a resource with the scenario. This sounds scary, but is pretty easy with the available tools.

This is pretty much what I'm concerned about; you need to build new recruiters for virtually every scenario. Whatever fancy tools you have, simulating ~100 matches for each of your 20 twenty scenarios in a campaign just to have a better AI doesn't sound like a good deal. Especially not if you have to re-generate stuff whenever you change a scenario. In my opinion, it's essential for the AI to be able to work well without having to run hundreds of simulations beforehand.

Don't misunderstand me, this whole thing is really awesome stuff, I just have doubts that it'd fit well into the main code. Wesnoth is a rather big project, and they have to take many aspects into consideration when adding new features. Performance might not be the most important one of them.