Optimisation Discussions Advises Experiences Ideas Etc

Discussion of all aspects of the game engine, including development of new and existing features.

Moderators: Forum Moderators, Developers

User avatar
Ravana
Moderator
Posts: 1815
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Ravana » September 4th, 2017, 5:49 pm

Before it took wmlunits between 20 sec and 60 sec to create unit trees for ageless. So while it had timeout of 60 sec, it worked, but this summer timeout was lowered to 20 sec, so I had to make it load faster.

With current ageless you can see ageless_4_18.preprocessed.cfg which is conditionally loaded instead of usual folder structure(which is still included in addon as zookeeper asked it be so). Such file is generated with wesnoth -p, followed by some regex replaces to make it valid unprocessed wml ("" to <<>> and >><< to ").


Some references
wesnoth -p viewtopic.php?p=613516#p613516
irc https://www.wesnoth.org/irclogs/2017/06 ... -06-03.log

enclave
Posts: 822
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » September 22nd, 2017, 7:47 pm

Comparing wesnoth.get_locations vs wesnoth.match_location vs [store_locations]
In previous posts match_locations worked much much much faster than get_locations, when there were complicated filters used.
This time I will try to compare simple filters and speeds..

Code: Select all

local all_villages = wesnoth.get_locations  { terrain="*^V*"}
wesnoth.set_variable("counter_66",#all_villages)

Code: Select all

[message]
message="$counter_66"
[/message]
So this gave me a count of villages on map in about 1-3 miliseconds on Isars Cross... or around 20-50 on Valeria.

Code: Select all

[lua]
code = <<
local ww,hh,bb = wesnoth.get_map_size()
local counter_66 = 0
	local ybegin = 0
    while (ybegin<=hh+1)
	do
			local xbegin = 0
			while (xbegin<=ww+1)
			do
				local bool ns_check_if_village = wesnoth.match_location(xbegin, ybegin, { terrain="*^V*"})
if ns_check_if_village == true then 
counter_66 = counter_66 + 1 
end
				xbegin = xbegin + 1
			end
	ybegin = ybegin + 1
	end
wesnoth.set_variable("counter_66",counter_66)
>>
[/lua]
[message]
message="$counter_66"
[/message]
This gave me a count of villages on Isar in about 9-30 miliseconds.. and around 210-230 on Valeria

So with very very basic filter it worked nearly 10 times faster to use:
local all_villages = wesnoth.get_locations { terrain="*^V*"}

Code: Select all

[store_locations]
terrain=*^V*
variable=counter_66
[/store_locations]
[message]
message="$counter_66.length"
[/message]
This is actually giving similar values as lua.. 1-5 on isar and 40-50 on Valeria

NEXT: I will try to add more terrains to filter and all the testing will now be on 180x90 Valeria map.
SO..
TEST1
wesnoth.match_location(xbegin, ybegin, { terrain="*^V*"}) 210-230 miliseconds
wesnoth.get_locations { terrain="*^V*"} 20-50 miliseconds
[store_locations]terrain="*^V*" 40-50 miliseconds

TEST2
wesnoth.match_location(xbegin, ybegin, { terrain="*^V*,*^F*"}) 240-280 miliseconds
wesnoth.get_locations { terrain="*^V*,*^F*"} 20-60 miliseconds
[store_locations]terrain="*^V*,*^F*" 60-100 miliseconds

TEST3
wesnoth.match_location(xbegin, ybegin, { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*"}) 390-430 miliseconds
wesnoth.get_locations { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*"} 30-60 miliseconds
[store_locations]terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*" 80-120 miliseconds

TEST4
wesnoth.match_location(xbegin, ybegin, { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*", {"not", {terrain="H*,H*^*,G*,G*^*"}}})
600-630 ms
wesnoth.get_locations { terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*", {"not", {terrain="H*,H*^*,G*,G*^*"}}}
70-100ms
[store_locations]terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*"[not]terrain="H*,H*^*,G*,G*^*"[/not]
80-110ms (funny.. max is less than test3.. i guess my PC was under less load during test4)

Ok.. so from these examples get_locations or store_locations working much quicker than iterating over each hex of the map with match_location
Lets try to filter just a 20 square hex area with same filters as in TEST4
So
TEST5
wesnoth.match_location 20-40 ms
Spoiler:
wesnoth.get_locations { x="20-40", y="20-40", terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*", {"not", {terrain="H*,H*^*,G*,G*^*"}}}
40-70ms
[store_locations]x=20-40 y=20-40 terrain="*^V*,*^F*,*^Q*,*^G*,*^E*,*^W*,*^D*"[not]terrain="H*,H*^*,G*,G*^*"[/not]
40-70ms

This time iterating over each hex of x,y from 20 to 40 with match_location was 2 times quicker than using get_locations or store_locations... while lua get_locations has same speed as WML store_locations.


In next examples I will try to compare whole map match_locations vs ipairs of get_locations to change each hex of filter matching part of map into something.. For example remove all forest only from hills.. or remove all villages except from water.. or something similar..
To be continued...

enclave
Posts: 822
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » September 22nd, 2017, 8:58 pm

Remove all villages from map, 3 ways.. comparison [terrain] vs wesnoth.get_locations (wesnoth.set_terrain) vs wesnoth.match_location (wesnoth.set_terrain) vs wesnoth.get_villages (wesnoth.set_terrain)

[terrain] 130-150ms
Spoiler:
wesnoth.match_location & wesnoth.set_terrain 210-240ms
Spoiler:
wesnoth.get_locations { terrain="*^V*"} & wesnoth.set_terrain 20-30ms
Spoiler:
wesnoth.get_villages() & wesnoth.set_terrain 2ms
Spoiler:
Somehowwesnoth.get_villages() working much faster than any other lua filter for villages, might be because of stars used in *^V* filter.. idk.. why..

Nice comparison.. Conclusions:
If you working with villages, definitly better use wesnoth.get_villages(),
if filtering smaller area, match_location probably would fit better and may be quickest..
filtering whole map is much better with get_locations..
Might make sense to consider using match_location after narrowing the area by get_locations, depends..
And of course match_location is a more advanced tool than get_locations, if you need to check adjacent hexes or radius for a specific place on map.

enclave
Posts: 822
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » April 12th, 2018, 12:16 am

5k times {REPEAT} testing some basic filters for speed (being said that [filter_wml] is slowest filter so I decided to test if it's true..) wesnoth 1.12
-------
1)local units_with_upkeep = #wesnoth.get_units { side = 1, canrecruit="no", {"not",{ability="hc_build_new_city"}} } 854 868 880 877 876
2)local units_with_upkeep = #wesnoth.get_units { side = 1, canrecruit="no", {"filter_wml",{upkeep=1}} } 190 188 177 184 192
3)local units_with_upkeep = #wesnoth.get_units { side = 1, canrecruit="no", ability="hc_build_new_city"} 739 742 764 742 758
4)local units_with_upkeep = #wesnoth.get_units { side = 1 } 776 770 774 784 774
Wth... I dont understand the results... I will do same test in WML, its confusing me.. filter_wml was even faster than basic "side=1" filter..

So exactly same tests in WML ([store_unit]):
1) 757 746 752
Spoiler:
2) 742 741 744
Spoiler:
3) 674 672 666
Spoiler:
4) 3034 2999 3030
Spoiler:
LOL that is funny... the more simple filter is the slower it seems to be... or maybe it depends on how much data it needs to write inside the WML variable... idk.. In this case the speed of store_unit would depend least on efficiency of the filter... most on efficiency of speed of creation of array.

These above ideally need more experiments, but I cant be bothered at this point of my life sorry..
Unfortunately the tests were carried with non-existant abilities and there was only 9 sides with 1 unit leader on field. So the accuracy is questionable, although the results are not what I expected to see at all. (The filters itself are 99% sure working fine, I tested them on another era, but without timing)


NOW just 20 times {REPEAT} testing some basic store locations filter. (50k too long)
--------
1)
Spoiler:
terrain=Ch*,Kh* 89 88 85 72 76
2) terrain=C*,K* 89 76 71 86 75
3) terrain=Ch,Kh,Chw 69 77 83 76 82
4) terrain=C*^*,K*^* 80 109 87 75 78
No difference at all with all the stars or without... I mean the difference is nearly non-existant..
then I wonder why lua wesnoth.get_villages is much much more faster than normal terrain filter..
Last edited by enclave on May 27th, 2018, 1:37 pm, edited 1 time in total.

enclave
Posts: 822
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » April 28th, 2018, 8:19 pm

A funny find on 1.12:
The code starts on select event:

Code: Select all

	[store_locations]	
	[not]
	find_in=qc_visible_villages
	radius=$qc_system_variables.city_min_distance
	[/not]
	[not]
	terrain=$hc_system_variables.illegal_city_building_terrain
	[/not]
	[filter_vision]
	visible=yes
	respect_fog=no
	side=$side_number
	[/filter_vision]
	variable=qc_allowed_build
	[/store_locations]
This will be timed and will take 100 mileseconds or whatever units the time stamp is using...

While if I just change $side_number into $unit.side the code will start to take 700-900 miliseconds... almost 10 times slower.
Without any massive change... like so:
[store_locations]
[not]
find_in=qc_visible_villages
radius=$qc_system_variables.city_min_distance
[/not]
[not]
terrain=$hc_system_variables.illegal_city_building_terrain
[/not]
[filter_vision]
visible=yes
respect_fog=no
side=$unit.side
[/filter_vision]
variable=qc_allowed_build
[/store_locations]

Also just to be more helpful if I set {VARIABLE temp_side $unit.side} before storing locations and using $unit.side in filter, and then use side=$temp_side in filter then speed will again be only 100 ms or whatever time stamp units.
Magic, but that is how it is...

gfgtdf
Developer
Posts: 1026
Joined: February 10th, 2013, 2:25 pm

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by gfgtdf » April 28th, 2018, 8:23 pm

well this is somehow expected, the first time that $unit is used it needs to be created first, that is, the game silently performs a [store_unit] for the primary unit that is invonved in that event, and that takes some time.
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.

enclave
Posts: 822
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave » April 29th, 2018, 12:33 pm

thanks gfgtdf... thats interesting.. Ill keep it in mind using unit variables in future...
Is it same thing for $x1 $y1? or it's only about $unit $secondary_unit etc?

gfgtdf
Developer
Posts: 1026
Joined: February 10th, 2013, 2:25 pm

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by gfgtdf » April 29th, 2018, 4:50 pm

this is only about unit, it has some unxpected side effects becasue if you change the primary unit (the original unit, for example via [modify_unit], not using $unit) the content of $unit might be different depending onn when your event used $unit for the first time: if you used $unit before your change the $unit variable will contain the state of the original unit, even when you access it again after your change, but you did not use $unit at all before your change it will contain the 'new' state of the unit.
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.

Post Reply