Machine Learning Recruiter

[SeattleDad]

I'd like to experiment with writing a Lua stage that would completely replace the RCA AI recruiting routing (AI_CA_RECRUITMENT). Are there any examples out there of completely or partially overriding AI_CA_RECRUITMENT with Lua? I think I just need a template and then I'd be ready to go.

Thanks!

[mattsc]

There are several examples using Formula AI, but I am not aware of anything in Lua AI. So I wrote one really quickly:

Code: Select all

    [side]
        side=1
        controller=ai
        name=Vanak
        id=Vanak
        type=Orcish Ruler
        side=1

        canrecruit=yes
        recruit=Orcish Grunt,Orcish Archer,Goblin Spearman
        gold=200

        {ai/aliases/stable_singleplayer.cfg}
        [ai]
            version=10710
            [engine]
                name="lua"
                code= <<
--! ==============================================================
ai = ...
local ai_stdlib = wesnoth.require('ai/lua/stdlib.lua');
ai_stdlib.init(ai)

local recruit_ai = {}

function recruit_ai:recruit_evaluation()

    local leader_on_keep = wesnoth.get_units { side = wesnoth.current.side, x = 28, y = 6, canrecruit = yes }[1]
    local unit_in_way = wesnoth.get_unit(27, 7)

    if leader_on_keep and (not unit_in_way) then
        return 180010
    else
        return 0
    end
end

function recruit_ai:recruit_execution()
    ai.recruit("Goblin Spearman", 27, 7)
end

return recruit_ai
--! ==============================================================
>>
            [/engine]
            {MODIFY_AI_ADD_CANDIDATE_ACTION 1 main_loop (
                [candidate_action]
                    engine=lua
                    name=lurker_moves
                    evaluation="return (...):recruit_evaluation()"
                    execution="(...):recruit_execution()"
                [/candidate_action]
            )}
        [/ai]
    [/side]

This is a very simple example, and since it's meant as a template only I hardcoded everything (and I don't even check for gold). It recruits a goblin spearman on hex 27,7 if that hex is empty and the leader is on the keep. It has a score just higher than the standard recruitment candidate action, which takes care of recruiting on other hexes. If you want to replace normal recruiting entirely, you can just remove that candidate action.

See this wiki page and links therein for more information.

PS: In practice, for longer code, you probably want to put everything between the '--! ======' lines into a separate file and include it here using wesnoth.require().

[SeattleDad]

Matt: Thanks!

I have a Lua-based AI which is a little more sophisticated in that it does a more sophisticated check of whether the leader is in the keep and check of whether there is an open castle or keep hex for recruiting a unit. Note that this AI assumes that there is only one leader and that the current side is the Knalgan Alliance. I'd be interested in any feedback on what I have so far.

Code: Select all

#textdomain wesnoth

[ai]
    id=seattle_ai_random
    description=_"Multiplayer_AI^Seattle AI" 
    version=10710
    [engine]
        name="lua"
        code = <<
--! =======================================================================
ai = ...
local ai_stdlib = wesnoth.require('ai/lua/stdlib.lua');
ai_stdlib.init(ai)
local recruit_ai = {}
print("hello world, in seattle_ai")
function recruit_ai:recruit_evaluation()
    local leaders_on_current_side = wesnoth.get_units { side = wesnoth.current.side, canrecruit = true }
    leader_on_keep = false
    local leader = leaders_on_current_side[1]
    --print("Leader is on " .. leader.x .. " " .. leader.y)
    if wesnoth.get_terrain_info(wesnoth.get_terrain(leader.x, leader.y)).keep then
        leader_on_keep = true
    end
    
    local unit_in_way = true
    for _,loc in ipairs(wesnoth.get_locations { x=leader.x, y= leader.y, radius=2 }) do
        print("Found a candidate spot at " .. loc[1] .. " " .. loc[2])
        local terrain = wesnoth.get_terrain(loc[1], loc[2])
        if wesnoth.get_terrain_info(terrain).castle or wesnoth.get_terrain_info(terrain).keep then
            if (wesnoth.get_unit(loc[1],loc[2]) == nil) then
                recruit_location = loc
                unit_in_way = false
                --print("Found a spot at " .. recruit_location[1] .. " " .. recruit_location[2])
            end
        end
    end
    
    if leader_on_keep and (not unit_in_way) then
        return 180010
    else
        return 0
    end
end
function recruit_ai:recruit_execution()
    ai.recruit("Dwarvish Thunderer", recruit_location[1], recruit_location[2])
end

return recruit_ai
--! =======================================================================
>>
    [/engine]
    [stage]
        id=main_loop
        name=testing_ai_default::candidate_action_evaluation_loop
        {AI_CA_GOTO}
        # {AI_CA_RECRUITMENT}
        [candidate_action]
            engine=lua
            name=seattle_recruiter
            evaluation="return (...):recruit_evaluation()"
            execution="(...):recruit_execution()"
        [/candidate_action]
        {AI_CA_MOVE_LEADER_TO_GOALS}
        {AI_CA_MOVE_LEADER_TO_KEEP}
        {AI_CA_COMBAT}
        {AI_CA_HEALING}
        {AI_CA_VILLAGES}
        {AI_CA_RETREAT}
        {AI_CA_MOVE_TO_TARGETS}
        {AI_CA_PASSIVE_LEADER_SHARES_KEEP}
    [/stage]
[/ai]

The AI always recruits Dwarvish Thunderers as long as the leader is in the keep and there is an open hex. More sophistication to come down the road, although interestingly, this AI seems to be outperforming the RCA AI for the Knalgan Alliance on the two-player Freelands map based on my limited experiments.

The biggest problem I'm having, though, is that I can't figure out how to run this from the command line. The above code runs fine in the GUI when I put this file (called "seattle_ai_default.cfg") in Resources/data/ai/ais/ along with testing_ai_default.cfg, it is selectable as "Seattle AI" in the pick list, but I can't figure out how to select it from the command line. The following, for instance, doesn't work:

MacOS/Wesnoth --nogui --nodelay -m --controller 1:ai --controller 2:ai --side 1:"Knalgan Alliance" --side 2:Northerners --ai-config 1:Resources/data/ai/ais/seattle_ai_default.cfg

Could anyone give me an example command line of how ai-config is supposed to work?

Thanks,
SeattleDad

[mattsc]

SeattleDad: for the most part, your code looks good. The biggest problem I see is how you find castle tiles. Using a SLF with radius=2 will fail when the castle is larger than that (or elongated); and when there are two castles next to each other with one hex separation. Neither of that is very common, but I have seen both. You might want to loop over helper.adjacent_tiles() to catch all, but only contiguous castle tiles. (As a side note, if you want to do it the SLF way, you can do the entire idenfying of unoccupied castle/keep hexes in a single SLF, rather than adding the Lua loop. And, btw, you can find all leaders on keeps with a single SUF.)

Personally, I also prefer to use the data variable to get information from the eval to the exec function, rather than global variables, but I don't know if there's actually any advantage to doing one or the other.

Finally, just a comment (you probably know this, but just in case): there's no need to list the CAs in the order in which you want them executed. It will work in any order, but it helps, of course, with intuition to list them in that order.

although interestingly, this AI seems to be outperforming the RCA AI for the Knalgan Alliance on the two-player Freelands map based on my limited experiments.

That's probably just a conincidence for that particular map and matchup.

I've never done the command-line no-gui thing, so unfortunately I cannot help with that.

[Sapient]

If you want to use a SLF to filter for recruitable castle tiles, I wrote one in the WML Workshop recently:
http://forums.wesnoth.org/viewtopic.php ... 35#p514335

FormulaAI has the 'castle_locs' function

[SeattleDad]

So after a lot of questions to the community, I think I have something to contribute now. The attached patch accomplishes two things:

1. data/ai/dev/seattle_ai_general.lua: Provides a recruiting framework, built on top of mattsc's sample recruiter which manages whether it is possible to recruit any units based on whether the leader is in the keep, whether there are free hexes in a castle, and whether you have enough gold. It also includes two sample recruiting logics: thunderer_recruiter() and random_recruiter(). The former always recruits Dwarvish Thunderers, while the latter randomly selects any unit eligible to recruit.
2. data/ai/ais/seattle_ai_thunderer.cfg: ai config file which invokes the thunderer_recruiter. Note that we assume that the faction for this AI is the Knalgan Alliance.
3. data/ai/ais/seattle_ai_random.cfg: ai config file which invokes the random_recruiter. Works for any faction
4. Patch to utils/ai_test/ai_test2.py: Fixes a number of incompatibilities between this script and the current version of Wesnoth. Also writes out the winning faction, losing faction, and winning AI to the log file. This makes the log file slightly easier to work with and manipulate.
5. Patch to utils/ai_test/ai_test2.cfg. Fixes incompatibilities and strips out stuff not relevant to ai_test2.py
6. data/ai/ais/testing_ai_default_with_macro.cfg. I'm not sure if this is necessary. I created it because the standard data/ai/ais/testing_ai_default.cfg doesn't work as a parameter to --ai-config in nogui mode without {core/macros} added as the first line. If patching testing_ai_default.cfg is okay, then we should just patch that file.

1, 2, and 3 are just prototypes so I'd like feedback, but am not proposing they be added to the 1.11 build, but I think 4,5, and 6 are almost ready to go. I'd be interested in feedback on all of them.

Also, I have some interesting experiments to share that I ran with this toolset, which seem to show that there's substantial room for improvement in the current RCA_AI recruiter.

400 iterations across 4 maps of random pairs of opponents with Random AI:

$ cat ai_test__Thursday_26_April_2012_04-30AM.log | cut -d, -f 18 | sort | uniq -c
163 "seattle_ai_random
237 "testing_ai_default_with_macro
1 "winner_ai"
--Random AI beats the RCA_AI recruiter 40.8% of the time
------------------

400 iterations across 4 maps of Knalgan vs. random opponents with Thunderer AI:

$ cat ai_test__Thursday_26_April_2012_08-45PM.log | grep -v '"Knalgan Alliance", "Knalgan Alliance"' | cut -d, -f 16 | sort | uniq -c
24 "Drakes"
206 "Knalgan Alliance"
10 "Loyalists"
35 "Northerners"
15 "Rebels"
57 "Undead"
1 "winner_faction"

This is 206 wins for the Knalgan Alliance and 141 wins for their opponents (excluding Knalgan vs. Knalgan games). 59.4% winning percentage for Knalgan Alliance
----------------------

400 iterations across 4 maps of Knalgen Alliance with RCA AI on both sides:

$ cat ai_test__Thursday_26_April_2012_11-43PM.log | grep -v '"Knalgan Alliance", "Knalgan Alliance"' | cut -d, -f 16 | sort | uniq -c
31 "Drakes"
160 "Knalgan Alliance"
18 "Loyalists"
48 "Northerners"
18 "Rebels"
45 "Undead"
1 "winner_faction"

The above sums to exactly 160 wins for Knalgan and 160 wins for their opponents, as you would expect, so the game is not balanced for the Knalgan's
-------------------

400 iterations across 4 maps of Knalgan Alliance with Random AI against default:

$ cat ai_test__Friday_27_April_2012_04-59AM.log | grep -v '"Knalgan Alliance", "Knalgan Alliance"' | cut -d, -f 16 | sort | uniq -c
34 "Drakes"
155 "Knalgan Alliance"
20 "Loyalists"
53 "Northerners"
24 "Rebels"
54 "Undead"
1 "winner_faction"

This sums to 155 wins for Knalgan Alliance and 185 wins for their opponents (disregarding Knalgan vs. Knalgan)
45.6% winning percentage for Knalgan Alliance using Random AI
---------------
Summary
So the above experiment suggests that, at least for the Knalgan's on multiplayer, there is room for improvement for the RCA AI recruiter. The score is:
59.4% winning percentage: Thunderer only (note that RCA AI recruits thunderers infrequently). Note: I totally get that an AI that recruited only Thunderers wouldn't be much fun from a game-play perspective. I'll be aiming for an "optimal" recruiter first and then we can figure out how to add the "fun" element back in later, probably by blending in some randomness.
50% winning percentage: RCA AI with Knalgan's
45.6% winning percentage: Random AI (choose a unit at random every time you have a recruiting opportunity). Although we do worse, we don't do that much worse.

I'd be interested in any feedback on all of this. For the next stage of my project, I'd like to do some intensive work on a new recruiting AI which is built around machine learning. Basically, I'll use an open source ML toolkit which will try to automatically learn an optimal recruiting strategy by seeing what works best across large numbers of games. The result of this project should be a new recruiter or set of recruiters which is generated by the ML algorithm. The run-time executable, if things go according to my current plan, will be a Lua executable which will be called by the same mechanism as my thunderer and random recruiters.

Thanks,
SeattleDad

[Civhai]

Wow, that is very interesting. I planned to do something similar for my own era because I think he does particularly bad there and I recently discussed with a friend about using machine learning for an improvement for the AI. My statement was that it was too complex to come up with anything useful, but we thought about improving the whole ai while you only want to improve the recruitment, which is much more realistic, I think. Our main problem was that you have extremely many parameters and trying to simplify it is very hard because the optimal recruitment strategy really depends on the map, the opponents, the strategies of the opponents and even the allies, so you really have to optimize a function in many variables and it is not clear that this function is even convex or anything, that is why we gave up the idea of using machine learning. But of course, if you only look at recruitments, you might eliminate at least some conditions.

[SeattleDad]

Thanks! I'm thinking the recruiting is a relatively simple task since, as I see it, the current recruiting algorithm (the class ai_default_recruitment_stage, which is defined in ai.hpp and ai.cpp) just depends on the following variables, as I understand it:
--Distribution of friendly units
--Distribution of enemy units
--Number of neutral villages (if there are more neutral villages, it builds more scouts)
--Distribution of terrain types on the map

My hypothesis is that I can pass these features into an ML algorithm and get a better decision function than what we currently have in ai_default_recruitment_stage.

[SeattleDad]

How do I trigger an action at the end of the scenario? I'm finding that I really need to write out some statistics that I'm collecting in global variables at the end of a scenario and don't see how to do this.

If someone could just show me how to write some Lua that will write out "hello world" at the end of a scenario, that would be all I'd need. I'm suspecting this could be done through wesnoth.game_events, but don't see exactly how.

Thanks!

[Civhai]

I don't know lua, but I guess you could define something like

[event]
name=victory
# do something
[/event]
[event]
name=defeat
# do something
[/event]
[event]
name=time over
# do something
[/event]

[Sapient]

A similar discussion about output:
http://forums.wesnoth.org/viewtopic.php ... 79#p511779

Normally Lua in Wesnoth runs under a safety mode which prevents use of file input/output libraries.

[SeattleDad]

Thanks Civhai, but I'm afraid I'm still a little confused. I've tried putting some code along the lines of what you suggest at the bottom of an AI config and it's not working. I think I'm missing something about how you are supposed to register something like this. Here's what I did:

Code: Select all

#textdomain wesnoth
{core/macros}
[ai]
    id=testing_ai_default_with_macro
    description=_"Multiplayer_AI^RCA AI" # RCA := Register Candidate Action; more info at http://forums.wesnoth.org/viewtopic.php?p=419625#p419625
    version=10710
    [stage]
        id=main_loop
        name=testing_ai_default::candidate_action_evaluation_loop
        {AI_CA_GOTO}
        {AI_CA_RECRUITMENT}
        {AI_CA_MOVE_LEADER_TO_GOALS}
        {AI_CA_MOVE_LEADER_TO_KEEP}
        {AI_CA_COMBAT}
        {AI_CA_HEALING}
        {AI_CA_VILLAGES}
        {AI_CA_RETREAT}
        {AI_CA_MOVE_TO_TARGETS}
        {AI_CA_PASSIVE_LEADER_SHARES_KEEP}
    [/stage]
[/ai]

[event]
    name=victory
    first_time_only=no
    # [allow_undo][/allow_undo]
    [filter]
        side=1
    [/filter]

    [lua]
        code = <<
            print ("Hello world from victory event")
        >>
    [/lua]
[/event]

The only thing new here is what's between "event" and "/event", but it's just not getting triggered at the end of a game when side 1 wins.

I also feel like I should be doing this through wesnoth.game_events as per http://wiki.wesnoth.org/LuaWML:Events#w ... ame_events, but I'm unclear on exactly how to do this.

[mattsc]

SeattleDad: I believe (based on a couple empirical tests I just did) that if you play a scenario with only AI sides, then no victory event gets triggered at the end of the scenario. So that's one reason why your code didn't work. The following event does work in an AI-only scenario (tested):

Code: Select all

[event]
    name=die
    first_time_only=no
    [filter]
        side=2
    [/filter]
    [if]
        [have_unit]
            side=2
            count=0
        [/have_unit]
        [then]
            {MESSAGE narrator "" "" _"Hello, World!"}
        [/then]
    [/if]
[/event]

This triggers when the last of the Side 2 units dies (and of course you can use a Lua print command instead of the message).

There might also be an issue in that you have that event included in the AI config file. It might not work from there. I don't know that for sure, but I don't have time to test this right now, so why don't you give that a try.

I do have a couple comments on your longer post from a few days ago (although I am not the right person to address some of that), but I've been traveling all week and haven't had time for it yet. I'll be back later today.

[SeattleDad]

Matt,

Thanks so much for the example. One very basic question: I've tried putting this code inside the AI Config file, but as you suspected, it's not working there. Where is this code supposed to go?

Thanks!
SeattleDad

[mattsc]

I've tried putting this code inside the AI Config file, but as you suspected, it's not working there. Where is this code supposed to go?

It's supposed to go inside the [scenario] or [multiplayer] tags. (There are a few other tags it can go in also, see EventWML.) Is that acceptable for your purposes, to modify the scenarios/maps you are testing?

If you need a solution that works from within the AI config file, umm, to be honest I am not sure what the best solution is. You could probably set up a candidate action that sets up such an event (after checking whether it exists already). That seems somewhat of a misuse of a CA, but I think it should be possible. Let me know if that is what you need and I'll think about how this could be done some more.