Optimisation Discussions Advises Experiences Ideas Etc

Discussion of all aspects of the game engine, including development of new and existing features.

Moderators: Forum Moderators, Developers

User avatar
Posts: 1603
Joined: January 29th, 2012, 12:49 am
Location: Estonia

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Ravana » September 4th, 2017, 5:49 pm

Before it took wmlunits between 20 sec and 60 sec to create unit trees for ageless. So while it had timeout of 60 sec, it worked, but this summer timeout was lowered to 20 sec, so I had to make it load faster.

With current ageless you can see ageless_4_18.preprocessed.cfg which is conditionally loaded instead of usual folder structure(which is still included in addon as zookeeper asked it be so). Such file is generated with wesnoth -p, followed by some regex replaces to make it valid unprocessed wml ("" to <<>> and >><< to ").

Some references
wesnoth -p viewtopic.php?p=613516#p613516
irc https://www.wesnoth.org/irclogs/2017/06 ... -06-03.log

Posts: 553
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » September 22nd, 2017, 7:47 pm

Comparing wesnoth.get_locations vs wesnoth.match_location vs [store_locations]
In previous posts match_locations worked much much much faster than get_locations, when there were complicated filters used.
This time I will try to compare simple filters and speeds..

Code: Select all

local all_villages = wesnoth.get_locations  { terrain="*^V*"}

Code: Select all

So this gave me a count of villages on map in about 1-3 miliseconds on Isars Cross... or around 20-50 on Valeria.

Code: Select all

code = <<
local ww,hh,bb = wesnoth.get_map_size()
local counter_66 = 0
	local ybegin = 0
    while (ybegin<=hh+1)
			local xbegin = 0
			while (xbegin<=ww+1)
				local bool ns_check_if_village = wesnoth.match_location(xbegin, ybegin, { terrain="*^V*"})
if ns_check_if_village == true then 
counter_66 = counter_66 + 1 
				xbegin = xbegin + 1
	ybegin = ybegin + 1
This gave me a count of villages on Isar in about 9-30 miliseconds.. and around 210-230 on Valeria

So with very very basic filter it worked nearly 10 times faster to use:
local all_villages = wesnoth.get_locations { terrain="*^V*"}

Code: Select all

This is actually giving similar values as lua.. 1-5 on isar and 40-50 on Valeria

NEXT: I will try to add more terrains to filter and all the testing will now be on 180x90 Valeria map.
wesnoth.match_location(xbegin, ybegin, { terrain="*^V*"}) 210-230 miliseconds
wesnoth.get_locations { terrain="*^V*"} 20-50 miliseconds
[store_locations]terrain="*^V*" 40-50 miliseconds

wesnoth.match_location(xbegin, ybegin, { terrain="*^V*,*^F*"}) 240-280 miliseconds
wesnoth.get_locations { terrain="*^V*,*^F*"} 20-60 miliseconds
[store_locations]terrain="*^V*,*^F*" 60-100 miliseconds

wesnoth.match_location(xbegin, ybegin, { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*"}) 390-430 miliseconds
wesnoth.get_locations { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*"} 30-60 miliseconds
[store_locations]terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*" 80-120 miliseconds

wesnoth.match_location(xbegin, ybegin, { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*", {"not", {terrain="H*,H*^*,G*,G*^*"}}})
600-630 ms
wesnoth.get_locations { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*", {"not", {terrain="H*,H*^*,G*,G*^*"}}}
80-110ms (funny.. max is less than test3.. i guess my PC was under less load during test4)

Ok.. so from these examples get_locations or store_locations working much quicker than iterating over each hex of the map with match_location
Lets try to filter just a 20 square hex area with same filters as in TEST4
wesnoth.match_location 20-40 ms
wesnoth.get_locations { x="20-40", y="20-40", terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*", {"not", {terrain="H*,H*^*,G*,G*^*"}}}
[store_locations]x=20-40 y=20-40 terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*"[not]terrain="H*,H*^*,G*,G*^*"[/not]

This time iterating over each hex of x,y from 20 to 40 with match_location was 2 times quicker than using get_locations or store_locations... while lua get_locations has same speed as WML store_locations.

In next examples I will try to compare whole map match_locations vs ipairs of get_locations to change each hex of filter matching part of map into something.. For example remove all forest only from hills.. or remove all villages except from water.. or something similar..
To be continued...

Posts: 553
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » September 22nd, 2017, 8:58 pm

Remove all villages from map, 3 ways.. comparison [terrain] vs wesnoth.get_locations (wesnoth.set_terrain) vs wesnoth.match_location (wesnoth.set_terrain) vs wesnoth.get_villages (wesnoth.set_terrain)

[terrain] 130-150ms
wesnoth.match_location & wesnoth.set_terrain 210-240ms
wesnoth.get_locations { terrain="*^V*"} & wesnoth.set_terrain 20-30ms
wesnoth.get_villages() & wesnoth.set_terrain 2ms
Somehowwesnoth.get_villages() working much faster than any other lua filter for villages, might be because of stars used in *^V* filter.. idk.. why..

Nice comparison.. Conclusions:
If you working with villages, definitly better use wesnoth.get_villages(),
if filtering smaller area, match_location probably would fit better and may be quickest..
filtering whole map is much better with get_locations..
Might make sense to consider using match_location after narrowing the area by get_locations, depends..
And of course match_location is a more advanced tool than get_locations, if you need to check adjacent hexes or radius for a specific place on map.

Posts: 553
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » April 12th, 2018, 12:16 am

5k times {REPEAT} testing some basic filters for speed (being said that [filter_wml] is slowest filter so I decided to test if it's true..) wesnoth 1.12
1)local units_with_upkeep = #wesnoth.get_units { side = 1, canrecruit="no", {"not",{ability="hc_build_new_city"}} } 854 868 880 877 876
2)local units_with_upkeep = #wesnoth.get_units { side = 1, canrecruit="no", {"filter_wml",{upkeep=1}} } 190 188 177 184 192
3)local units_with_upkeep = #wesnoth.get_units { side = 1, canrecruit="no", ability="hc_build_new_city"} 739 742 764 742 758
4)local units_with_upkeep = #wesnoth.get_units { side = 1 } 776 770 774 784 774
Wtf... I dont understand the results... I will do same test in WML, its confusing me.. filter_wml was even faster than basic "side=1" filter..

So exactly same tests in WML ([store_unit]):
1) 757 746 752
2) 742 741 744
3) 674 672 666
4) 3034 2999 3030
LOL that is funny... the more simple filter is the slower it seems to be... or maybe it depends on how much data it needs to write inside the WML variable... idk.. In this case the speed of store_unit would depend least on efficiency of the filter... most on efficiency of speed of creation of array.

These above ideally need more experiments, but I cant be bothered at this point of my life sorry..
Unfortunately the tests were carried with non-existant abilities and there was only 9 sides with 1 unit leader on field. So the accuracy is questionable, although the results are not what I expected to see at all. (The filters itself are 99% sure working fine, I tested them on another era, but without timing)

NOW just 20 times {REPEAT} testing some basic store locations filter. (50k too long)
terrain=Ch*,Kh* 89 88 85 72 76
2) terrain=C*,K* 89 76 71 86 75
3) terrain=Ch,Kh,Chw 69 77 83 76 82
4) terrain=C*^*,K*^* 80 109 87 75 78
No difference at all with all the stars or without... I mean the difference is nearly non-existant..
then I wonder why lua wesnoth.get_villages is much much more faster than normal terrain filter..

Post Reply