[micro_ai] slow

The place to post your WML questions and answers.

Moderator: Forum Moderators

Forum rules
  • Please use [code] BBCode tags in your posts for embedding WML snippets.
  • To keep your code readable so that others can easily help you, make sure to indent it following our conventions.
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

I really don't have time to read all this right now, but even with all this it doesn't look like I have all the information needed to answer the question. The problem with the Goto MAI was not the fact that the filter needed to be evaluated and matched many units/hexes, but that there was pathfinding to be done for each unit for each hex. In this case, it is likely the attack evaluation itself that takes a lot of the time, but it's impossible to say without more information. The first thing to do:
enclave wrote: May 8th, 2018, 3:35 pmNow question is only if its simple_attack micro_ai completely bugged. Or it happens because I have large list of weapon specials...
Would many weapon specials make a difference with freezing it for 10 sek on a basic filter with exact x,y coordinates?
It might and that should be easy to test for you yourself. Replace the unit in question in debug mode by one that does not have a lot of specials and see what happens.
enclave wrote: May 8th, 2018, 3:35 pmThis would bloody go x1000 faster if there was a micro_ai tag called "goto_attack" which would use goto filters and then just temporarily set AI aggression to 1000, until next unit.. cheeses...
Well, feel free to write one. It's all Lua, so anybody can do it. :)
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: [micro_ai] slow

Post by enclave »

mattsc wrote: May 8th, 2018, 8:46 pm Well, feel free to write one. It's all Lua, so anybody can do it. :)
haha ;)
Where would I find the source code for all micro_ai? if you happen to know...
I will try to use a unit without weapon specials, see if it make a difference...
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

enclave wrote: May 8th, 2018, 8:55 pm Where would I find the source code for all micro_ai? if you happen to know...
I do. It's here. (That's specifically for the simple attack, but the rest is in the same directory.)

So, the first thing I'd do is that test with a unit with no or fewer specials, to see if that causes the problem. Then I'd go into that code and add some time_stamp print commands to figure out whether the individual attack evaluation takes a long time, or whether it takes so long because it does too many different evaluations. I could do that for you (but not right now and possibly not today any more), but I'd need a save to try it myself and you'll probably get to it sooner anyway.

After that, well, then it depends on what you find... It is quite possible that it is indeed the attack evaluation, which is done in Lua and therefore slower than the default AI's combat CA (written in C++).

Btw, depending on what you want to do, you might be able to do this with the attacks aspect, which might (or might not) be faster than the Simple Attacks MAI.
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: [micro_ai] slow

Post by enclave »

mattsc wrote: May 8th, 2018, 9:04 pm I do. It's here. (That's specifically for the simple attack, but the rest is in the same directory.)

So, the first thing I'd do is that test with a unit with no or fewer specials, to see if that causes the problem. Then I'd go into that code and add some time_stamp print commands to figure out whether the individual attack evaluation takes a long time, or whether it takes so long because it does too many different evaluations. I could do that for you (but not right now and possibly not today any more), but I'd need a save to try it myself and you'll probably get to it sooner anyway.

After that, well, then it depends on what you find... It is quite possible that it is indeed the attack evaluation, which is done in Lua and therefore slower than the default AI's combat CA (written in C++).

Btw, depending on what you want to do, you might be able to do this with the attacks aspect, which might (or might not) be faster than the Simple Attacks MAI.
Thank you very much mattsc, also for reading all my stuff :D even part of it ;))
I already spent half a day on it today, so I'm done for today, I just updated my mod with ai with aggression=1, works similar to what I wanted.. (almost)
I would like to find the reason of problem eventually... so I could leave it until you have more free time (I'm happy with result for now, more or less, [micro_ai] would suit my needs better, but without it its bearable)

Then when I get time I will post a save here, so you could try it maybe... see if your PC gives same delay.
I will also post the result with no weapon specials.
And you could tell me exactly where to put timing for all these processes you mentioned, because for me its a bit like a dark forest, will take time to figure out what is what.
After that we will see where it leads...
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

Sounds like a plan. As long as there's no rush, I am sure we can figure it out. Send me that save when you're ready and I'll have a look. After that I can tell you what I did to test things.
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: [micro_ai] slow

Post by enclave »

Sorry for not writting for a while, I gave up on attack micro_ai because I could get same/similar functionality with goto and aggression=1
And even with goto it's still very slow so I didn't see the point to investigate into attack micro ai... sorry..
The more goto I add the slower it becomes... so I can't develop my AI any more, it has reached the limit... too smart and too slow :D

But... I have still questions to ask... (just not about attack micro_ai for now.. but something much much more simple..)
I have some micro ais, all of them goto... the filter checks for unit types or abilities..
After all units already moved the AI side hangs for some time and then ends turn (The more micro AIs, the longer it hangs)
Even if there are 0 units of filtered types and 0 units with filtered abilities on whole map each goto micro_ai is adding more lag time to that hang.

(Please don't read any codes unless you are interested in these details for some reason, just read my text to save time.. You may scroll to the very bottom end now for my main questions)

That's an example save file with following micro_ais in it:
ZZ_report_savefile_micro_ai.gz
(260.66 KiB) Downloaded 327 times
(there are no ships, no ranged units, no peasants workers.. but AI spends a huge amount of time calculating something on end of turn)
Spoiler:
And the next one has different micro_ais in it, the affected units move quite quickly... there is no problem with that...
But this lag in the end of turn is so bad :(
for unknown reason after all units have already moved the AI spends a lot of time calculating something... what is AI calculating?
ZZ_Report2_savefile_micro_ai.gz
(271.99 KiB) Downloaded 334 times
This is an example of the same map with exactly same macros and events, but without any micro ai (nearly no lag at end of turn):
ZZ_Report2_NO_micro_ai.gz
(267.43 KiB) Downloaded 300 times
And this one is with 2 micro ais, for settlers and for leader
ZZ_Report4_micro_ai.gz
(270.15 KiB) Downloaded 278 times
Spoiler:
And this one has only micro ai for settlers units.. (they move very fast, but at end of turn AI hangs for some time..)
(I left last player as HUMAN side if it helps test something, use this file)
ZZ_Report5_settlers_micro_ai.gz
(261.43 KiB) Downloaded 287 times
Spoiler:
I timed the start at moveto event and end on side turn event (because side turn end till side turn would give no much lag time.. around 60 ms)

My main questions:
1) Why is there a massive delay/lag/hang/freeze just before the end of AI turn which depends on amount of micro_ais (the more micro_ai code has, the longer is the freeze) while these micro ais contain filters for unit types that are not even present on map. Why AI calculates/thinks about something if filter returns NO result???
2) Would I be able to speed up the AI moves somehow (NOT the end turn delay) if I used some kind of lua functions instead of WML filters?
Thanks...

No rush.
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

I do not get a "massive delay" at the end of the turn. The values displayed are usually a little less than second, caused by the MAI evaluation taking 200 - 300 ms each a few times. That does not sound so unreasonable to me. The cases when the value you display on the screen is a few seconds happen when it also appears to include an attack animation (which makes sense if you starting counting from a moveto event). But I have to play through several full turns of 10 sides all moving a lot of units before this even happens the first time. Which brings up this:

These saves are really not how you should create a test case for a problem if you want help from others. There's a lot going on once you load on of those files, it took me quite some time to figure out what you're even talking about - and as you see, I still don't really know what you mean. [In addition, the output (stdout/stderr) is flooded with errors, so I have to sift through all of that for any debug output I produce there.] I was very close to saying that this isn't worth my time. You should reduce this to a situation that consist of only the minimal that is required to reproduce what you are trying to demonstrate. Please provide a save that you can load and that immediately (as in, directly following loading the save, and ideally ending the human side's turn) shows the problem as clearly as possible. And BTW, I do get the feeling that you are trying (so this isn't meant to sound harsh), but this is not it yet.
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: [micro_ai] slow

Post by enclave »

mattsc wrote: May 22nd, 2018, 4:09 pm I do not get a "massive delay" at the end of the turn. The values displayed are usually a little less than second, caused by the MAI evaluation taking 200 - 300 ms each a few times. That does not sound so unreasonable to me. The cases when the value you display on the screen is a few seconds happen when it also appears to include an attack animation (which makes sense if you starting counting from a moveto event). But I have to play through several full turns of 10 sides all moving a lot of units before this even happens the first time. Which brings up this:

These saves are really not how you should create a test case for a problem if you want help from others. There's a lot going on once you load on of those files, it took me quite some time to figure out what you're even talking about - and as you see, I still don't really know what you mean. [In addition, the output (stdout/stderr) is flooded with errors, so I have to sift through all of that for any debug output I produce there.] I was very close to saying that this isn't worth my time. You should reduce this to a situation that consist of only the minimal that is required to reproduce what you are trying to demonstrate. Please provide a save that you can load and that immediately (as in, directly following loading the save, and ideally ending the human side's turn) shows the problem as clearly as possible. And BTW, I do get the feeling that you are trying (so this isn't meant to sound harsh), but this is not it yet.
Thanks very much for trying to figure out the problem!
But well if you are running the savefile on a Intel Xeon Platinum 8173M server then unfortunately I won't be able to represent the lag no matter what kind of savefile I provide (unless I create a neveredning loop maybe?).. My PC is not the best, it's just 6 Core AMD fx-6100, but clearly not the worst among all the PCs that average wesnoth player is using for wesnoth. It may be a phone or even single core laptop from somewhere around year 2005 (because Wesnoth is not too demanding and you dont need to have latest video card and latest intel processor to play it). The savefile I provided was best way for me to test the lags, identify the most demanding micro_ais to modify or exclude them, so I thought it would be the easiest way to represent the problem to you. I can not do any better, and not really willing or have tome to, unfortunately. I assumed my questions very simple and straightforward but you haven't even tried to give me a clue of what it could potentially be and not even tried to answer my question #2 (it's easier to just ignore it I assume..), which brings us to the point that this problem is going nowhere.
To finalise it let's just assume that the output (stdout/stderr) is flooded with errors hence it makes a delay due to my bugged code (very likely possible that this is the problem). Thanks for trying to help! Sorry for taking your time ;)
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

Umm ... I am trying to help you. I used the saves you provided and took quite some time to sift through a lot of things that look like clutter to me, with no result. I cannot reproduce "massive delays". That is on a 3-year old Macbook Air which is not exactly the slowest computer in existence, but not anywhere close to what you mention. So I either do not have the information I need, or I do not understand where it is and how to access it sufficiently easily.

As for your second question, I was going to come back to that. I currently do not have enough information to answer that either, but I figured I take it one step at a time.

EDIT: Oh, and the reason why I did not "give [you] a clue of what it could potentially be" is because I have no idea what it could potentially be. I need to be able to reproduce and understand the problem first.
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: [micro_ai] slow

Post by enclave »

mattsc wrote: May 22nd, 2018, 5:49 pm Umm ... I am trying to help you. I used the saves you provided and took quite some time to sift through a lot of things that look like clutter to me, with no result. I cannot reproduce "massive delays". That is on a 3-year old Macbook Air which is not exactly the slowest computer in existence, but not anywhere close to what you mention. So I either do not have the information I need, or I do not understand where it is and how to access it sufficiently easily.

As for your second question, I was going to come back to that. I currently do not have enough information to answer that either, but I figured I take it one step at a time.

EDIT: Oh, and the reason why I did not "give [you] a clue of what it could potentially be" is because I have no idea what it could potentially be. I need to be able to reproduce and understand the problem first.
Yeah I'm sorry, I didn't mean to be nasty or anything, I really appreciate your time, I just don't think we can figure out the problem..
(I was hoping that the details I provided was enough, but unfortunately like you said you don't see any lag, which means that I need to take much more actions than I have time or desire for.. if provided details was enough then you would help me, but in our case looks like we stuck. Please don't take it personal or anything similar. I just hoped for a quicker and more simple solution, but there is none according to our conversation. I understand that ideally I would have to create a separate add-on with ONLY micro_ai in it, test for lags again.. see if its my add-on causing lag or it's realy micro_ai.. it makes sense to me, I understand it perfectly fine.. maybe some day we could come back to that conversation, but for now I can't do any more. Thanks for everything again ;))
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

enclave wrote: May 22nd, 2018, 6:45 pm Yeah I'm sorry, I didn't mean to be nasty or anything
Good, neither did I. :) And I am sorry if I came across that way.

I wouldn't quite give up yet though. So here's the kind of save that you would ideally provide:
  • When you first load it, the human player is in control. This is important because sometimes in order to figure out what exactly is happening with an AI, you need to change things up a bit on the map (add/remove units, change hitpoints, etc.) before the AI takes over. This can be done using debug mode if the player is in control, which is much easier than hacking into the save.
  • All that's needed to reproduce the error from there is ending the human player's turn. The AI with the problem should take over directly, and the situation should be set up so that it is obvious (as in, choose a case that gives you a really long delay, rather than a sort-of kind-of long one.
  • Those should be the only two sides active, so that control goes back to the player immediately after.
  • Ideally you would also remove everything else that does not affect the test case (to eliminate other potential causes and to reduce the number of errors printed to stderr), but I understand that that might not be 100% possible.
That's the ideal case. If not all of those points are 100% true, that's fine, but the closer we get to that, the easier it will be to zero in on the problem.

You should also tell me how long it actually takes on your system. That way, if mine were a factor 2, or even 4, faster, we're likely looking at the same issue. If it's a lot more than that, it might point to a different problem, possibly even independent of the AI.

Does that make sense? In any case, as you said yourself, there's no rush.


Edit: Just saw that you added something while I was typing. I don't think you need to create a separate add-on. But you could disable all but one player and one AI side, and get me a save from right before you experience a really big delay. That would already be much better. But if you don't have time for that, or don't want to do it, no worries also. :)
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: [micro_ai] slow

Post by enclave »

mattsc wrote: May 22nd, 2018, 7:06 pm [*]When you first load it, the human player is in control. This is important because sometimes in order to figure out what exactly is happening with an AI, you need to change things up a bit on the map (add/remove units, change hitpoints, etc.) before the AI takes over. This can be done using debug mode if the player is in control, which is much easier than hacking into the save.
This could easily be hackable if you reload this game in multiplayer server, replace the ai side of interest with human, save it and then load it in local game to use debug. I found it easier to see the problem if ai was moving without any interrupts, but I understand that for your tests it may be the exactly opposite of "helpful".
mattsc wrote: May 22nd, 2018, 7:06 pm [*]All that's needed to reproduce the error from there is ending the human player's turn. The AI with the problem should take over directly, and the situation should be set up so that it is obvious (as in, choose a case that gives you a really long delay, rather than a sort-of kind-of long one.
In this scenario I was completely unable to test things because delays were depending on too many things and only average values of 5+ players could really tell anything.. again I understand that for your tests it may be the exactly opposite of "helpful" :D
mattsc wrote: May 22nd, 2018, 7:06 pm [*]Those should be the only two sides active, so that control goes back to the player immediately after.
[*]Ideally you would also remove everything else that does not affect the test case (to eliminate other potential causes and to reduce the number of errors printed to stderr), but I understand that that might not be 100% possible.
I can see now that you are interested in this, so I will do my best and test things in a "pure" empty add-on with only micro_ais, I'm curious myself if it would make any difference..
mattsc wrote: May 22nd, 2018, 7:06 pm You should also tell me how long it actually takes on your system. That way, if mine were a factor 2, or even 4, faster, we're likely looking at the same issue. If it's a lot more than that, it might point to a different problem, possibly even independent of the AI.
Yeah I realized that I didn't give any comparison and word "massive delay" may be misunderstood... Will provide all the numbers for previously provided savefiles. Because for me the delays are not so bad, but other players happen to suffer a lot so I had to create options to decrease the micro_ai involved and host was able to decide what level of intelligence his CPU is happy to support... The big difference also happens when there is a lof of AI sides, so usually people with just 1 AI could happily wait their turn, but when there is 8 of AI then it may become really annoying and every milisecond on every micro_ai matters a lot... because they all total up and become seconds or even minutes..
Just in case, the add-on is available on add-on server (Q Civ or qciv) and it could be tested on any map with any sides and with any micro AI settings.. (they all enabled by default, but can be individually enabled), if somehow this helps you better.. it will still be full of stdrr errors so I will create a new one with default era.. at least it would clear things out a lot when it either gives same lag time or not..
mattsc wrote: May 22nd, 2018, 7:06 pm In any case, as you said yourself, there's no rush.
I will try to do it today, but I'm not sure.. so possibly all of the above will be ready in some hours time.. Thank you very much for your interest, just to let you know the [micro_ai] really made a huge difference to my add-on, it's a great tool and it works very well. I wish it was faster and more customizable, but it's a great help even as it is.

PS. Ok to start with something.. (I just have some other project in my mind, so I'm afraid that if I start it I might not want to get back to this for a while..) The values I had on my computer (Windows 7) for the savefiles previously provided:
ZZ_report_savefile_micro_ai.gz (1398,1246,1233,1242,1307,1270,1359) average 1300
ZZ_Report2_savefile_micro_ai.gz (1905,1707,1867,2017,1850) average 1900
ZZ_Report2_NO_micro_ai.gz (316,363,285,455,402,205,313) average with NO micro ai 300
ZZ_Report4_micro_ai.gz (1500,1328,1281,1135,1141,1208) average 1200
ZZ_Report5_settlers_micro_ai.gz (1062,920,1086,1000,1241,1016,917) average 1000
*remove 70ms from my timings for end turn event, side turn end event takes about 70ms on my PC
This is timed from the last moveto event until side turn event (so after all units already moved, but before next player starts to move) So it represents the strange timeframe where AI is thinking about something unknown ... and it takes up to 4 seconds with all micro_ais enabled (savefile not included), the most annoying and questionable part is that it spends a lot of time for nothing really.. and it grows when I add more micro_ai.
PS. I'm quite sure that I also tried to test from side turn end event until side turn event and amount of time was around 80ms on my system..
Yeah just double checked. Timed from last moveto event until side turn end event with all micro_ai enabled this test shown 2900 average. With all disabled 300. So it can not be the time that AI spends to think about next turn because he hasn't yet ended his own turn..
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

Hey enclave — thanks for all the explanations. I understand much better now what's going on and why this is a problem for your add-on. I don't really have time to look into it any more today, and I won't have any time at all the next two or three days, but I had another really quick look at it and found some clues (but no final answers yet) as to what is going on.

It looks like slightly more than half of those delay times you show (which are similar to what I get) are used on evaluating the SLF and SUF in the Micro AI. I have ideas and suggestion for dealing with those, but need to do some tests that I do not have time for until later this week. Of course, there's also the question what the other ~40% of the time are used for.

Also, it looks like the longer delays are "not real". Look at this:
Screen Shot 2018-05-22 at 19.17.18.png
Screen Shot 2018-05-22 at 19.17.18.png (62.16 KiB) Viewed 5405 times
Notice how the start time is the same for both? I assume that means there's no move happening for the latter side or something, and thus you get a wrong estimate of how long this took.

As I said, I know what to do next, I just need to find the time to actually do this. In other words, no need for you to do anything else right now until I get back to this. I'll also have some more replies to what you wrote. But for now I gotta run ...
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: [micro_ai] slow

Post by enclave »

mattsc wrote: May 23rd, 2018, 2:39 am Also, it looks like the longer delays are "not real". Look at this:

Screen Shot 2018-05-22 at 19.17.18.png

Notice how the start time is the same for both? I assume that means there's no move happening for the latter side or something, and thus you get a wrong estimate of how long this took.
Yeah indeed, you have a good eye on details ;)) I noticed that the number wasnt the amount of time I waited sometimes, but never realized the reason behind it :)
mattsc wrote: May 23rd, 2018, 2:39 am As I said, I know what to do next, I just need to find the time to actually do this. In other words, no need for you to do anything else right now until I get back to this. I'll also have some more replies to what you wrote. But for now I gotta run ...
Thanks so much!!! No rush, I will do other things meanwhile ;) Thanks again! Even if we cut 10% of waiting it will be fantastic ;)
mattsc
Inactive Developer
Posts: 1217
Joined: October 13th, 2010, 6:14 pm

Re: [micro_ai] slow

Post by mattsc »

I still haven't had any time to actually work on this, but I do have a few minutes right now to make some general comments. Taking things slightly out of order:
enclave wrote: May 22nd, 2018, 7:33 pm Thank you very much for your interest, just to let you know the [micro_ai] really made a huge difference to my add-on, it's a great tool and it works very well. I wish it was faster and more customizable, but it's a great help even as it is.
That's great to hear. As for more customizable, we're always open to suggestions. There's even a feedback thread, although that's been dead for a long time now. That does not necessarily mean that we will implement everything that gets suggested into mainline (there's often a trade-off between number of options, speed, ease-of-use, etc.), but the good thing about the MAIs is that they are all written in Lua. Which means they can easily be imported into and customized for any specific add-on. Also, note that significant changes won't be made during a stable release series, so they'd be in 1.15/1.16 at the earliest.
enclave wrote: May 22nd, 2018, 7:33 pm Yeah I realized that I didn't give any comparison and word "massive delay" may be misunderstood...
Indeed. In the settings I usually deal with, SP or small 1vs1 MP maps, those values you give certainly wouldn't be much of a cause for concern. Thanks for writing them down though. Those are comparable to what I get, so I know what I need to be looking for now.

As I mentioned, my first (very quick and superficial) look showed that more than half of the time is needed simply for evaluating the filters. And that is set up to be repeated after every move of an AI unit, because both the units and the locations might change due to the previous move. And:
enclave wrote: May 22nd, 2018, 7:33 pm So it represents the strange timeframe where AI is thinking about something unknown ...
No, it is not unknown at all what it does (and all AI candidate actions do this). The AI does not know up front that there is nothing left to do, it needs to figure out first if there are units/locations left to consider. Now, it does sound like that is not done very efficiently. I'll have a look into that.

I see two ways of dealing with the long filter eval times:
  1. Reduce the time it takes to evaluate the filters (which might include moving parts into Lua). I don't know much about that, but it is probably possible. Well, I am quite certain about the part I already said in the previous paragraph, that the final evaluation, when all units have moved, can almost certainly be streamlined.
  2. If your scenario/map is such that it does not require units and/or locations to be (re)found after each AI unit move, it could be set up to be done only once at the beginning of the turn. That would obviously save a lot of evaluation time. That will not be ported to mainline though, as it would break other uses of this MAI. [Well, maybe we could add an option reevaluate_after_move=true/false or something, that defaults to the old behavior (true).]
I think that's the general comments I have. I'll do some more careful tests sometime in the next few days to see if there are any other time sinks and get back to you afterward.
Post Reply