Help with testing Fred - Freelands MP Custom AI v0.14.15

[Elder2]

Wow, I just found out about this project, an ai designed to be so good to have a chance against a player with equal resources? Very interesting.

I played it once, with undeads dark sorcerer, I even tried to play good like I would againt a real player because I did not know what to expect. I used my classical recruit as if there was fog of war. And then applied my cheese strategy against orc... Fred is extremely defensive, which is fine since I did not give it any chances for an attack, and with his not necessarily best build army againt my faction there was little chance for a successfull attack. However I feel like there were some opportunities to try to overwhelm me in a few turns of trade. Also the ai didnt try to outmaneuver me which makes matters easier. I am not sure if it would have worked that well with a higher income, but I just camped, probably with a higher income it could do much better but I think the camping strategy would still win. So I made a perfect very strong army and finally marched on the ai and basically killed everything.

The ai may be a bit too defensive, though at some point in the end I felt that its defensive evaluation algorithm kind of broke. The ai may scale well with higher income, but I am not willing to camp so much with ud again against higher income fred, but I do not think like 2 or 3 units more would have made a big difference if I set the income to be higher by about 2.
The main problem against a camping strategy at least as ud is that there were issues with ai sending units into my corner so I could pick them off, feeding me more free kills early game. I do not understand these moves at all, I would assume its an issue with the algorithm.

[mattsc]

I don't have time for anything else right now, but I wanted to clear up one misconception right away:

an ai designed to be so good to have a chance against a player with equal resources?

Uh, I don't know how you came up with that, I've never claimed anything like that. As I say on the first page of this thread:

My aim for Fred is mostly to provide an AI that is fun to play against and provides an entertaining alternative to the Default AI on at least the Freelands map. I'm not trying to (and am definitely not capable of) making him play as well as a human player.

That's really all, just an amateurish attempt at having some fun.

[Elder2]

Hm well, its very campy so maybe not that fun to play xD but fine.

Btw I have watched cold steel's replay with loyalists and even against lawful faction like loyalists its unreal how campy it is, this is probably the most campy play I have ever seen, scared of loyalists at day, scared of loyalists at night and it never actually attacks to deal a significant damage but rather focuses on forming pointless defensive lines while getting destroyed.

[singalen]

1.13 is a bit less forgiving with sloppy Lua code, so it is not at all unlikely that there might be some error messages popping up in 1.13. If you encounter any of those, please post the error message and, ideally, a save from which this can be reproduced.

Thanks a lot for the update!
I can make a pull request to the core code. Anything that can make 1.13 as or more comfortable as 1.12 to program?

[name]

Some more factions at higher income levels:

rebels_vs_14.7_income+5.gz

(27.94 KiB) Downloaded 359 times

This battle hinges heavily on turn 16. It makes me wonder what would happen if Fred was a bit more fearful during dawn and more aggressive during dusk. I might have to hack him and test this at some point.

northerners_vs_14.7_income+5.gz

(35.21 KiB) Downloaded 438 times

I found out here that chaotic factions have a harder time against Fred. I started massing goblin heavy armies to keep upkeep down but later shifted to mostly trolls with assassin support, attacking at day when Fred is defensive. By the mid twenty turns I ended up going all in trying to take his western villages and did so at a high cost. I was able to hold them just long enough for reinforcements to cement their control or else I might have lost the game due to force depletion.

undead_vs_14.7_income+6.gz

(42.17 KiB) Downloaded 403 times

With the also chaotic undead and an additional 6 income for Fred, this was a challenge. Working in my favor though was that Fred went light on trolls and attacked at night when undead have the greater advantage over northerners. For some reason I found myself recruiting a sizeable defensive corpse army to keep upkeep down and then banking heavily before risking rolling out too many level one units. As a result the battle did not really begin in earnest until almost turn 40 with a peak of no less than 70 units on the field.

You're right, the defensive positioning is not perfect, but it's probably about as good as I can expect it to be.

Well, as before, we'll have to see if it makes a noticeable difference. I think the main improvements will come from not exposing small groups of units too much (as Fred did, for example, in your replay in the southeast; although I do like the unkillable troll whelp!) and coordinating actions some more.

Yeah it looks to me like his tactical positioning (grouping units, forming lines, taking good terrain) is very strong. When I can separate some of his forces and defeat them in detail, it feels like this is because of strategic positioning (how many units deployed to a region and when they attack or retreat as a group with regards to time of day). All operations layer stuff in other words.

That being said, at some point in the future, you might consider writing up a map editor feature request for an AI "game plan" tool, to make positioning a content-side issue that takes advantage of human preprocessing power.

Oops, just found out this is already a feature. There' is an "areas" menu in the map editor (though not yet active in 12.x) which lets you custom tag areas of the map for AI scripts to read. So no need for a feature request then.

[name]

Btw I have watched cold steel's replay with loyalists and even against lawful faction like loyalists its unreal how campy it is, this is probably the most campy play I have ever seen, scared of loyalists at day, scared of loyalists at night and it never actually attacks to deal a significant damage but rather focuses on forming pointless defensive lines while getting destroyed.

You have to keep in mind that defensive play has been the main focus of development for this AI up until now, because getting an AI to understand how to even form a proper defensive line with this game's complex terrain and faction balance is difficult. Little help is provided by white papers and such, because they mostly focus on path-finding for common FPS or RTS genres.

But you should give this AI another play after the next version or two is released and the "operations layer" is complete, as that could give this AI the strategical level planning it needs to launch formidable offensives, in addition to holding ground as well as it already does or better (due to better unit allocation).

[Elder2]

While I believe it is hard, probably even very hard to program an AI to do that, I bet it is even harder to make AI know when to do that, maybe even impossible as it has not been yet shown. Its worthless if AI knows how to form defensive lines if it does it all the time, this is definitely not something you can call a proper or working defense. There were many situations in the replay where normal standard wesnoth AI would have done much better, teaching AI one trick doesn't automatically makes it superior if it does it all the time regardless of its utility.

[mattsc]

Hi All. Thanks for all the comments and replays. I'm traveling at the moment with little free time, so it will take me a while to catch up with everything and extract test cases from the replays. I will do so eventually, but it will likely take me even longer than usual. Just a few quick comments for now:

Zap-Zarap and Cold Steel: Thanks for yet more testing and repays. I really appreciate your repeated efforts of sticking with Fred. I'm sure I'll have some more specific comments after digesting all the replays.

singalen: Thanks for the offer, but there's really no problem with 1.13 and Fred should work just as well as in 1.12. It's just that 1.13 has stricter error handling than 1.12 (which is good) and I might not have encountered all situations yet in my tests. (Well, there is one problem: I don't seem to be able to save the game from a replay at the moment in 1.13. Still looking into why that is the case and then I'll submit a bug report if it's not because of my setup.)

ElderofZion: Well, frankly, if you don't enjoy it, then don't play against Fred. Pretty much the only thing I have gotten out of what you have written is that Fred is too defensive—which isn't exactly news. And as Cold Steel said, the switching between aggressive and defensive play is something I am looking into for the next release (that's one of the things the operations layer does). I make no promises that it will "fix" things, this is all a work in progress with uncertain outcome. But thanks for the replay anyway. I will study it and see what I can extract from it.

[name]

Two more to complete the set:

knalgans_vs_14.7_income+5.gz

(35.57 KiB) Downloaded 440 times

drakes_vs_14.7_income+5.gz

(28.92 KiB) Downloaded 374 times

One thing I did notice with the drakes is that Fred was more aggressive with them than other factions, which I suspect might be because they concentrate the same power into fewer units. So by a raw unit count or level count drakes look less intimidating. Valuing an opposing force by its total gold value might prove a more accurate way to estimate its threat level (but this would need to be tested of course).

[mattsc]

As always, thank you very much, Cold Steel. As I said, it's going to be a while until I get to work through all of these replays, but I will do so eventually.

Concerning your comment, that's interesting and I will look into its cause, but nothing in the evaluation is based on the unit number ratio. The metrics are generally either gold value or expected damage the units can do to each other, all of that modified by hitpoints and/or experience and/or time of day etc. However, these metrics are certainly not all well tuned yet and doing so is on my todo list. That's what
- Clean up which type of power/value/etc. parameters to use in each situation
in Issue 15.

So that's just my long-winded way of saying that I agree with you!

[singalen]

I played a couple of games, one with +5 gold to AI, one with +3, and Fred beat me to a pulp. I normally beat the default AI on these settings.
Great job!
I don't think you are interested in replays?

[mattsc]

Thanks, singalen, great to hear that it worked! If you don't mind sharing, I am always interested in the replays, whether Fred beats you to a pulp or the other way around doesn't matter. I always learn something about what could be improved from them.

[singalen]

OK, sadly I only saved last one. I'll make more.
edit: +a mirror match that I finally managed to win, with just +1 gold advantage for Fred.

[mattsc]

Thank you.

[Elder2]

Spoiler:

Fred+8 vs only dark adepts (and 2 bats)

On lower income I have also tested only goblins working.

I also tried only adepts on +18, didn't go so well. But I saw how many units ai gets and I am pretty sure I would be able to win at +18 income, its just that with Fred it would probably take 50 turns, so its not very likely I will ever try it seriously. Ai can still be manipulated to play how you want, though its not that easy.