Help with testing Fred - Freelands MP Custom AI v0.14.15
Moderator: Forum Moderators
Re: Help with testing Fred - Freelands MP Custom AI v0.14.3
For your curiosity: For commenting one needs a Github account, but not any special settings for your repository.mattsc wrote:In principle people could even add comments there, but I have no idea if one needs a github account for that and whether I need to set up the permissions differently but, quite honestly, I am way too lazy to try to find out ...
Re: Help with testing Fred - Freelands MP Custom AI v0.14.3
Thanks. The former I thought was the case, the latter I was not sure about.pauxlo wrote:For your curiosity: For commenting one needs a Github account, but not any special settings for your repository.
In the meantime, I am making slow progress on Fred. I think I have the mechanics of protecting the leader down, but it needs some fine-tuning. In general, I am finding yet again that the whole defensive unit positioning continues to be the hardest part for me to get to an acceptable level. I probably should shelf working on the details of that for now and give it some dedicated time for a later release.
Also, just for kicks* I did a bunch of batch testing of Fred (100 gold) vs. the default AI playing drakes (with 160 gold; that's important, at the same gold the default AI just gets trounced). That's the match-up with which Fred usually has the most problems. For previous version, he averaged around 50% win rate. Currently he's winning between 70 and 80%. I know that doesn't necessarily mean that he'll also do better against a human opponent, but at least I don't seem to be making things worse.
*Well, not just for kicks. The other purpose of this is to run through a lot of games to see if there are any errors. I did find a couple that way ...
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.3
What do you think is mainly responsible for his improvement against drakes?
Re: Help with testing Fred - Freelands MP Custom AI v0.14.3
I don't think that there's a single change that is the main reason for this, but the overall effect of many gradual changes is that Fred does better defending now, and in particular protecting his leader, when he's up against a stronger force. The main factors contributing to this are defensive unit positioning in the tactics layer (while I am saying that that's still very hard for me to figure out, it is better than it used to be) and better overall orders and unit distribution in the operations layer.Cold Steel wrote:What do you think is mainly responsible for his improvement against drakes?
So, I have thought about this some more and put together a rough road map for the next few releases in Issue #12 on the AI-demos site. I think splitting it up like that rather than trying to tackle it all at once will help the overall process.mattsc wrote:I probably should shelf working on the details of that for now and give it some dedicated time for a later release.
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
Fred / AI-demos v0.14.4 is now on the Wesnoth 1.12 add-ons server
This version addresses several of the issues raised by ForPeace. See
Issue #12: Fred: Tasks for the 0.14.4 release
Issue #13: Fred: Tasks for the 0.14.5 release
for a summary of what was done and a tentative plan for the next couple releases. If you tested Fred v0.14.3, the differences might or might not be noticeable, I'm not sure. Also see the known issues listed below.
v0.14.4 complete changelog
Known or expected issues
Interesting batch testing result
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
I think it is great that the AI is being developed further! I am wondering though from the batch result does this mean that Fred as the Northerners lost 3 games against the Northerners using Default AI, but won every other game?
Also if i download the Fred AI can i play against it on other maps beside freelands?
Thank you!
Also if i download the Fred AI can i play against it on other maps beside freelands?
Thank you!
Creator of: The Reign of The Lords Era,The Gnats Franken Dungeon.
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
Yes, that's what that means. That's nothing new though, he's been achieving these kinds of win ratios for several years. See for example here.The_Gnat wrote:I am wondering though from the batch result does this mean that Fred as the Northerners lost 3 games against the Northerners using Default AI, but won every other game?
Not at this time, there is quite a bit of map-specific code. When I get Fred to the point that's "good enough" on Freelands — if that ever happens — then I'll work on adapting this to other maps.The_Gnat wrote:Also if i download the Fred AI can i play against it on other maps beside freelands?
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
Okay. Hopefully that will be able to happen sooner rather than later!mattsc wrote:Not at this time, there is quite a bit of map-specific code. When I get Fred to the point that's "good enough" on Freelands — if that ever happens — then I'll work on adapting this to other maps.
As for the impressive win ratio why do you think Fred lost at all to the northerners?
Also is Fred able to play any other teams?
Creator of: The Reign of The Lords Era,The Gnats Franken Dungeon.
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
No promises.The_Gnat wrote:Okay. Hopefully that will be able to happen sooner rather than later!
I really don't know. I'd have to look into this, but there's not really a point in doing so until I have worked on some of the obvious current weaknesses as outlined in previous posts.The_Gnat wrote:As for the impressive win ratio why do you think Fred lost at all to the northerners?
In principle yes, but it's currently disabled. It's pretty easy to hack around that though. Just remove the first conditional in line 62 here. He won't do as well with other factions though as he has no idea how to use certain abilities (like healing or leadership).The_Gnat wrote:Also is Fred able to play any other teams?
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
Okay thank you!
Creator of: The Reign of The Lords Era,The Gnats Franken Dungeon.
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
I just downloaded the add-on to look at it and i am impressed! I thought it would be simpler but it is certainly complicated and i respect that it will take a long time to turn this into a full usable AI!
Creator of: The Reign of The Lords Era,The Gnats Franken Dungeon.
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
It has been some time since I played a match against Fred, so I may have forgotten how good he was in previous versions, but it felt like he was more responsive in switching between overall attack and defend and his defensive line forming was more apparent. I thought he did quite well given no gold advantage and played very humanly overall. As such, I'll only list the (possible) mistakes he made because listing the things he did right would be too verbose.
Bigger:
His biggest mistake may have been not retreating his western forces further north and into more defensive terrain on turns 19-20. This might be considered the turning point of the battle.
Smaller:
Orcish Archer left in open at day on turn 7 at (15,13) and then again at dawn on turn 18 at (14,14). On turn 19 at day, does something similar with orcish grunt at (24,10).
On turn 21 at (22,8) an assassin is left in the open, although in this case he was pushed out there by an archer that needed his hex to make a kill shot on one of the drakes.
On turn 27, the king moves off keep to less favorable terrain.
The only real evaluation slow down also took place on turn 27.
Fun:
At turn 23 the adventurous orcish grunt has single-handedly taken the enemy castle at (19,20), but witnessing from its battlements the return of the dragon horde, has a sudden crisis of courage.
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
Thanks, Cold Steel, for the replay and the useful comments. And for playtesting in the first place!
I watched the replay first and then read your comments (and then I watched parts of it again). I thought it was fun to watch those back and forth dynamics play out with the ToD. Fred's not perfect at it, but he's "making a good effort" by now. I was pleased to see that. And I'm pretty sure that you're right and he wouldn't have been able to do this in previous versions to the same extent. Maybe to a lesser degree in 0.14.3, but not before that.
I had also noticed Fred several times leaving individual units stranded too far in enemy territory, as you point out. That's a "known problem" and caused by a mix of several issues, some in the unit positioning and some in the operations layer (overall coordination). As you can see from the current AI-demos roadmap, I am planning to deal with those for 0.14.6 and 0.14.7. I am going to come back to your replay then, it provides several really good test cases for that!
For 0.14.5, I have started to dig into the attack evaluation. I'm not sure if these will make an significant difference overall, or only in specific cases, but there's a whole bunch of problems with that that I've always pushed off dealing with.
I watched the replay first and then read your comments (and then I watched parts of it again). I thought it was fun to watch those back and forth dynamics play out with the ToD. Fred's not perfect at it, but he's "making a good effort" by now. I was pleased to see that. And I'm pretty sure that you're right and he wouldn't have been able to do this in previous versions to the same extent. Maybe to a lesser degree in 0.14.3, but not before that.
I had also noticed Fred several times leaving individual units stranded too far in enemy territory, as you point out. That's a "known problem" and caused by a mix of several issues, some in the unit positioning and some in the operations layer (overall coordination). As you can see from the current AI-demos roadmap, I am planning to deal with those for 0.14.6 and 0.14.7. I am going to come back to your replay then, it provides several really good test cases for that!
For 0.14.5, I have started to dig into the attack evaluation. I'm not sure if these will make an significant difference overall, or only in specific cases, but there's a whole bunch of problems with that that I've always pushed off dealing with.
Ha, yeah, that is pretty funny!Cold Steel wrote:Fun:
At turn 23 the adventurous orcish grunt has single-handedly taken the enemy castle at (19,20), but witnessing from its battlements the return of the dragon horde, has a sudden crisis of courage.
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
No problem, playing against Fred is a fun change of pace from the zombie rushing of RCA and the 5 minute waits between turns in multiplayer. As more of the community finds this out for themselves you should have quite a few more testers.
I did notice some things that might hopefully be relevant to attack evaluation issues in this next replay:
Notes:
Force deployment (beginning on turns 5-6) focused a bit too heavily on the center theater. These forces may have done more good threatening/rushing the knalgans in the eastern theater, because defending the center requires very minimal resources. I think he may have gotten spooked by forces I seemed to be amassing in the center, though in reality they were only passing through to reinforce the western theater.
The retreat in the western theater at turn 8 may not have been far enough and ended up adjacent to castle terrain for the enemy to attack from.
It is interesting that in this match up he often puts out defensive screens of assassins rather than trolls on open ground. He might be evaluating the assassin’s superior terrain defense without fully taking into account the resistance, cost and/or regeneration advantages of trolls. There are good examples / test cases of Fred’s assassin lines from turn 10 onwards in the western theater.
He maybe could have counter attacked in the west on turns 17-19. Though the dwarfish line was formidable, its center was on poor terrain and it was dusk, plus Fred’s castle was close by for quick reinforcements. This may be a difficult thing to improve because around the this same time and place in the drake match, Fred suffered for not being defensive enough and here against dwarfs he suffers for not being offensive enough.
On turn 21, the slaying of the dwarf at (13,5) was well executed by Fred, first attacking with the king from strong terrain and then well protecting his flanks with heavies.
Yeah, I saw that under the known issues list and almost didn't report instances of it, but decided it was better to err on the side of being thorough. Glad it turned out to be useful rather than annoying.mattsc wrote: I had also noticed Fred several times leaving individual units stranded too far in enemy territory, as you point out. That's a "known problem" and caused by a mix of several issues, some in the unit positioning and some in the operations layer (overall coordination). As you can see from the current AI-demos roadmap, I am planning to deal with those for 0.14.6 and 0.14.7. I am going to come back to your replay then, it provides several really good test cases for that!
Indeed, that is quite a list.mattsc wrote: For 0.14.5, I have started to dig into the attack evaluation. I'm not sure if these will make an significant difference overall, or only in specific cases, but there's a whole bunch of problems with that that I've always pushed off dealing with.
I did notice some things that might hopefully be relevant to attack evaluation issues in this next replay:
Notes:
Force deployment (beginning on turns 5-6) focused a bit too heavily on the center theater. These forces may have done more good threatening/rushing the knalgans in the eastern theater, because defending the center requires very minimal resources. I think he may have gotten spooked by forces I seemed to be amassing in the center, though in reality they were only passing through to reinforce the western theater.
The retreat in the western theater at turn 8 may not have been far enough and ended up adjacent to castle terrain for the enemy to attack from.
It is interesting that in this match up he often puts out defensive screens of assassins rather than trolls on open ground. He might be evaluating the assassin’s superior terrain defense without fully taking into account the resistance, cost and/or regeneration advantages of trolls. There are good examples / test cases of Fred’s assassin lines from turn 10 onwards in the western theater.
He maybe could have counter attacked in the west on turns 17-19. Though the dwarfish line was formidable, its center was on poor terrain and it was dusk, plus Fred’s castle was close by for quick reinforcements. This may be a difficult thing to improve because around the this same time and place in the drake match, Fred suffered for not being defensive enough and here against dwarfs he suffers for not being offensive enough.
On turn 21, the slaying of the dwarf at (13,5) was well executed by Fred, first attacking with the king from strong terrain and then well protecting his flanks with heavies.
Re: Help with testing Fred - Freelands MP Custom AI v0.14.4
That's great to hear, thanks.Cold Steel wrote:No problem, playing against Fred is a fun change of pace from the zombie rushing of RCA and the 5 minute waits between turns in multiplayer. As more of the community finds this out for themselves you should have quite a few more testers.
Absolutely! Please, never hold back on any comments you have. You may notice something I didn't, you probably have a different perspective or different priorities. And even if you "only" say exactly what I am thinking, it will be good to get the confirmation from somebody who is much better at this than I am. I'm not that good a player in the first place (in MP at least, I seem to be doing semi-ok in SP*).Cold Steel wrote:Yeah, I saw that under the known issues list and almost didn't report instances of it, but decided it was better to err on the side of being thorough. Glad it turned out to be useful rather than annoying.
Well, I've done a bit of work on the attack code by now. [ You could, in principle, check the commits on github, but I really don't know why anybody would do that to themselves. ;P ] However, so far it's all been just internal refactoring. I am partly cleaning up the code that has "grown" over several years, and partly I am checking that the approximations I am making in the attack evaluations are good enough. Hopefully I will get to things that actually affect behavior soon(ish).Cold Steel wrote:Indeed, that is quite a list.
Perfect, thanks! It really is fun for me to watch these replays, seeing that Fred more or less does the things I am trying to teach him to do. With mistakes, sure, but overall about as good as I could hope for at this stage. He does like to put assassins on the front line, doesn't he? I'm sure you're right that he is likely putting too much emphasis on the terrain defense.Cold Steel wrote:I did notice some things that might hopefully be relevant to attack evaluation issues in this next replay:
I'll leave it at that for comments for now. As you say, these provide some great test cases. I'll report back on this when I get to dealing with them. Thank you again!!
* As a side note, coming from my SP background: one of my far-in-the-future goals once I start generalizing Fred for other maps/factions is to set up an AI that can play through and win HttT on medium. Revealing my hubris here, I've decided that doing so on easy is too easy.
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on