Machine Learning Recruiter

[Crab]

This label seems fine and the general approach seems fine, although couldn't we use it as the default on two-player multiplayer games on which we had tested it?

yes, we can set it default for 2p mp maps, but only for default era (non-default eras can have different unit sets).
P.S. <joke> if we name it "RCA MLR AI", we'll surely confuse the translators ) </joke>

So, the plan would be to announce a new dependency on Waffles in wesnoth-dev mailing list, wait for a while, then, if no one objects too heavily, import the source into Wesnoth (as I understand, unfortunately, we can't get Waffles via apt-get so we need to compile it into Wesnoth to save everyone the trouble of having to install it); then, enable it not as default and make sure that there's support in project files for regenerating the caches for standard maps. btw, what's the size of the caches?

[SeattleDad]

So, the plan would be to announce a new dependency on Waffles in wesnoth-dev mailing list, wait for a while, then, if no one objects too heavily, import the source into Wesnoth (as I understand, unfortunately, we can't get Waffles via apt-get so we need to compile it into Wesnoth to save everyone the trouble of having to install it); then, enable it not as default and make sure that there's support in project files for regenerating the caches for standard maps.

This all sounds good.

btw, what's the size of the caches?

The .json files which represent the ML models are about 65K each and there is currently one for each faction. I would like to boil this down to a single model file that works across all factions, but hopefully 6 * 65K is not to big a deal.

Also, note that the patch I submitted contains all the modules necessary to generate the model files. I'm working on documentation on how to do this.

[SeattleDad]

I'm on Linux, building with gcc. ... The problem at hand is the compile error. It fails to resolve an invoke to a function (apparently affiliated with file I/O).

As per Mike Gashler (the Waffles maintainer), please try putting the following at the top of GTokenizer.cpp:

Code: Select all

#ifdef WINDOWS
#include <direct.h>
#else
#include <unistd.h>
#endif

Neither he nor I can repro this error, but we're hoping this does the trick.

[SeattleDad]

I just wanted to report that my 10-year-old son and I have played three games against the "Recommended" version of the ML Recruiter on two-player maps and have been beaten all three times. Games were:
Us------------ML Recruiter ("Recommended")
Undead--------Northerners
Rebels---------Undead
Rebels---------Knalgan Alliance

The third game was particularly interesting. The neural net started out recruiting a balanced force of Dwarvish Fighters, Thieves, Thunderers, and Poachers (but no Gryphon Riders, unlike the RCA AI) and got an advantage. We were then completely unable to recover because its later recruits were over 50% Ulfserkers, which attritioned us down to nothing.

Our strategy was far from perfect--some of these games we put ourselves on a time limit and some of these games were played 50% by my son (who is pretty good, but 10), however we're finding this harder to beat than the RCA AI, which we were pretty much always able to beat on multiplayer maps. Also note that we're seeing a comparable variety of units as we saw with the RCA AI. For instance, the ML Recruiter recruits Gryphon Riders only 2.5% of the time, but it recruits Ulfserkers 5.1% of the time overall (but a lot more, apparently, when it has an advantage against a Rebel force who are deploying a lot of Wose), whereas the RCA AI never recruited Ulfserkers on multiplayer maps.

[lipk]

Now it compiles, but when it's the AI's turn in a game, I get the following Lua error:

Code: Select all

blahblah/ml_ai_general.lua:199: bad argumen #-1 to 'get_ml_recruitment_score' (number expected, got nil)

[Alarantalara]

I just played a game using the recommended recruiter. Rebels vs. Rebels on Den of Onis.
I was very disappointed. I fought an endless succession of Woses and won easily.
That is: there was no variety at all in the recruit and the choice of unit was terrible.

Edit: I've taken a look at the console log for the game and it shows that there was a period during the middle where it wanted to recruit other units, but never had enough gold to do so, and it picked Wose when it could.

Second Edit:
The ML AI seems to have no problem playing against mattsc's rush AI. Perhaps seeing what the difference in training results when fighting it are may be useful.

[SeattleDad]

I just played a game using the recommended recruiter. Rebels vs. Rebels on Den of Onis.
I was very disappointed. I fought an endless succession of Woses and won easily.
That is: there was no variety at all in the recruit and the choice of unit was terrible.

Thanks for trying this out and it's good to hear that someone else has got the patch to work. I think you hit a bit of a corner case in that I have never run across a case where the ML Recruiter recruited only one unit with the recommended AI. From the log you can see that it had an 84.5% to 98.3% chance of picking a Wose as it's first five units. I've never seen it this biased towards one unit at the beginning of the game, although I did reproduce your results myself when I played Rebels vs. Rebels on Den of Onis myself.

I just posted ML_Recruiter_0.1.1.patch at https://gna.org/patch/index.php?3479. This is essentially the same as the last patch, except that I trained it on games played with the 0.1 patch. I then retried your game and saw that the probability of picking a Wose at the beginning of your Den of Onis Rebels vs. Rebels scenario (with the same choice of units that you picked) had fallen to 61 - 68% at the begining, so you should see a lot more variety.

Could you give this another try?

Thanks.

Edit: I just looked at the stats for ML Recruiter 0.1.1 and saw that when playing against the RCA AI, ML Recruiter wins 89.1% of the time in Rebels vs. Rebels games and (against all factions), it wins 70.7% of the time on Den of Onis, which is slightly higher than its win percentage on the other three maps.

[SeattleDad]

I just posted a new 0.2 version of the ML Recruiter patch at https://gna.org/patch/index.php?3479. The major focus of this release was on getting system integration issues correct, so the major improvements are:
--Logging messages changed from print statements to using lg::log_domain. So now, by default, nothing is printed out. Use "--log-info=ai/testing,ai/ml" at the command line to view how the neural net assesses its options
--Now have an explicit debug mode by running with --log-debug=ai/ml.
--Use local ai, which allows ML Recruiter to play against itself. Previously could only have ML Recruiter on one side.
--Some work on ML recruiting model (i.e. the core logic). Experimented with different training strategies, but features unchanged from version 0.1.

Any feedback would be appreciated. Documentation is at http://wiki.wesnoth.org/Machine_Learning_Recruiter.

[Sapient]

Your goodness metric seems poorly chosen.

The experience point component lumps three different things together: attack count, defend count, and kill count, with emphasis on striking the killing blow.

Rather than counting up experience points for attacks and kills, however, wouldn't it make more sense to track damage inflicted? Something like $($target_unit_cost * ($damage_inflicted/$target_max_hp)) would give a good insight into the offensive output of a unit. To refine it further, bonus damage due to leadership or a similar ability should be credited to the supporting unit, and poison damage should be credited to the unit that first poisoned the target.

Of course defense is still important. But is counting experience gained while defending really the best way to measure this? Probably not.

Then there is the village capture metric... it is odd to simply add this score to the other one. Initially scouts will get the captures, or whatever units happened to be recruited first. But as the fight progresses village captures will go to random units: whichever lucky one happens to go next after a kill, for example. The ability for a unit to capture a village however has nothing to do with composition of enemy troops, so I don't see how this fits with your algorithm. Maybe a village being captured is just a good thing that reflects well on your entire army, and not just that particular unit who captured it.

[SeattleDad]

Your goodness metric seems poorly chosen.

I'm not in love with the current goodness metric. It was mostly chosen because it was very easy to implement. Wesnoth handles all the bookkeeping of keeping track of experience points and I can use the "capture" event to keep track of village captures.

Even though it's not perfect, the current metric is working pretty well, IMHO. Win ratio on the 0.2 release is around 68 - 70%, which is a big improvement given that the only change is to the recruiting algorithm.

The experience point component lumps three different things together: attack count, defend count, and kill count, with emphasis on striking the killing blow.

Rather than counting up experience points for attacks and kills, however, wouldn't it make more sense to track damage inflicted? Something like $($target_unit_cost * ($damage_inflicted/$target_max_hp)) would give a good insight into the offensive output of a unit. To refine it further, bonus damage due to leadership or a similar ability should be credited to the supporting unit, and poison damage should be credited to the unit that first poisoned the target.

In defense of the current goodness metric, I think you will tend to see similar results to what you're proposing over a large number of games. Units which do more damage will get more kills. Units which have better defense will survive more battles. Also, the experience-point based goodness metric optimizes towards getting units which are most likely to get a promotion and promotions are a good thing.

Then there is the village capture metric... it is odd to simply add this score to the other one. Initially scouts will get the captures, or whatever units happened to be recruited first. But as the fight progresses village captures will go to random units: whichever lucky one happens to go next after a kill, for example. The ability for a unit to capture a village however has nothing to do with composition of enemy troops, so I don't see how this fits with your algorithm.

The idea of the village capture metric is to give some additional credit to fast units, which tend to be more likely to capture villages. I think fast units tend to be more likely to capture villages all through the game, although this is certainly more true at the beginning than the end. I agree that the likelihood to capture villages isn't much influenced by the composition of enemy troops, but it is influenced by the units' speed.

Interestingly, recruiting of scouting units is an area that the current ML Recruiter is good at. It strongly emphasizes fast units when it is winning, because slower units will be unlikely to reach the battle in time in order to get experience points/village captures.

However, again, I think that you do make some good points. Right now, I'm looking at two lines of experiments:
1. New unit-based goodness metrics. Try out something along the lines of your damage-based metric. I'm also concerned that the current metric is blind to healing, slowing, poison, and leadership. I suppose that healing, poison, and leadership could all be folded into a damage/unit cost metric. Slowing doesn't quite fit, but could be blended in somehow. Note that some of these are a little tricky to implement. For instance, there's no event that signals a heal or a use of leadership. You'd have to work this out yourself from unit adjacency during a battle, for instance.
2. Directly predicting win or loss as opposed to a unit-specific metric.

Thanks for your input. My take is that a strong point of the ML Recruiter is that it allows you to experiment with different features and goodness metrics to see what works best.

[SeattleDad]

So I'm working on writing a new "goodness metric" (http://wiki.wesnoth.org/Machine_Learnin ... iter_works) along the lines of what Sapient suggested above. This requires considerably more bookkeeping than the experience-point-based metric that I had been using previously, so I need a place to store this data. The most elegant solution seems to me to hold the information in the unit table. I've done some experimentation and it seems that the variables table does the trick (see http://wiki.wesnoth.org/LuaWML:Units#wesnoth.get_unit). For instance, the following would record who poisoned a unit:

Code: Select all

            local attacker = wesnoth.get_units{x=ev.x1, y=ev.y1}[1]
            local defender = wesnoth.get_units{x=ev.x2, y=ev.y2}[1]
            --assuming that we determine that attacker poisoned defender
            defender.variables.poisoned_by = attacker.id
            defender.variables.is_poisoned = true
            --Now I can see all the variables stored under defender.variables:
            for key,value in pairs(defender.variables.__cfg) do
                print ("defender.variables.__cfg",key,value)
            end
            -- I can also directly access the variable:
            print("I was poisoned by", defender.variables.poisoned_by)

Two questions:
1. Does anybody see anything wrong with this general approach? I am thinking of prefixing all of my variables with ml_ (e.g. ml_is_poisoned) so as to avoid variable collisions.
2. Is there any way to set up a unit filter (http://wiki.wesnoth.org/StandardUnitFilter), which can filter based on the contents of "variables". For instance, how do I set up a filter to select for defender.variables.is_poisoned == true?

Thanks!

[Sapient]

To filter on a unit variable named "is_poisoned"

Code: Select all

[filter_wml]
  [variables]
    is_poisoned=true
  [/variables]
[/filter_wml]

See also the wiki example here.

To filter on a status you would write [status] instead of [variables].

Although I am surprised that filtering directly on status=poisoned isn't supported... It probably should be.

FYI, if you decide to go with predicting win/loss, you may be interested in the Advantage calculator. Discussion thread here. I think all that came out of it so far was AI's [unit_worth] tag, though.

[SeattleDad]

To filter on a unit variable named "is_poisoned"

Code: Select all

[filter_wml]
  [variables]
    is_poisoned=true
  [/variables]
[/filter_wml]

Thanks, but what I really need is an example of how to do this kind of thing in Lua. For instance, the following doesn't work:

Code: Select all

wesnoth.get_units{status = {poisoned=true}}

Could someone point me to some code that has a nested filter in Lua?

FYI, if you decide to go with predicting win/loss, you may be interested in the Advantage calculator.

I did an experiment where I tried using win/loss as the goodness metric (win = 1, loss = 0). This sort of worked. The predictions of win/loss were reasonable in so far as it made reasonably good predictions of whether it was winning or losing, but the problem was that it didn't differentiate much between the probability of winning with it's different choices of units to recruit (e.g., recruiting an Orcish Archer vs. recruiting an Orcish Grunt). The result was a roughly 59% winning percentage vs. the 68 - 71% winning percentage I'm seeing with the XP + village capture metric. I'm currently focusing on a version of the metric you suggested to see if I can push the win percentage higher.

[mattsc]

Thanks, but what I really need is an example of how to do this kind of thing in Lua. For instance, the following doesn't work:

Code: Select all

wesnoth.get_units{status = {poisoned=true}}

No, that wouldn't work in WML either. What Sapient meant is that the filter for poison needs to be inside a [filter_wml] also, such as this:

Code: Select all

local poisoned_units = wesnoth.get_units {
    { "filter_wml", {
        { "status", { poisoned = true } }
    } }
}

There are other ways to do this. I've attached a few code snippets that test for both units with poison weapons and poisoned units below, just in case you're interested:

Code: Select all

Lua way of testing whether a unit (stored in a table) is poisoned:

                local status = H.get_child(defender.__cfg, "status")
                local cant_poison = status.poisoned or status.not_living

2 Ways of testing for poison weapons:
(Note: AH.get_live_units() is almost the same as wesnoth.get_units, just that it exclude petrified units)

            -- Keep this for reference how it's done, but don't use now
            -- local poisoners = AH.get_live_units { side = wesnoth.current.side,
            --    formula = '$this_unit.attacks_left > 0', canrecruit = 'no',
            --    { "filter_wml", {
            --        { "attack", {
            --            { "specials", {
            --                { "poison", { } }
            --            } }
            --        } }
            --    } }
            --}

            local attackers = AH.get_live_units { side = wesnoth.current.side,
                formula = '$this_unit.attacks_left > 0', canrecruit = 'no' }

            local poisoners, others = {}, {}
            for i,a in ipairs(attackers) do
                local is_poisoner = false
                local weapon_number = 0
                for att in H.child_range(a.__cfg, 'attack') do
                    weapon_number = weapon_number + 1
                    for sp in H.child_range(att, 'specials') do
                        if H.get_child(sp, 'poison') then
                            --print('is_poinsoner:', a.id, ' weapon number: ', weapon_number)
                            is_poisoner = true
                        end
                    end
                end

                if is_poisoner then
                    table.insert(poisoners, a)
                else
                    table.insert(others, a)
                end
            end

[SeattleDad]

Code: Select all

                local status = H.get_child(defender.__cfg, "status")
                local cant_poison = status.poisoned or status.not_living

Thanks, that all seems to be working. However, wouldn't it be easier to write the above code snippet as:

Code: Select all

local cant_poison = defender.status.poisoned or defender.status.not_living