Optimisation Discussions Advises Experiences Ideas Etc

Discussion of all aspects of the game engine, including development of new and existing features.

Moderator: Forum Moderators

enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave »

Some of us have created eras or modifications. They work.

But there may be not obvious problems.. The code is working fine! But...
Maybe you can not reload your saved game because save file code is taking too much space..
Maybe your random map generator takes ages to generate map and most users leave before map was even loaded. Maybe something works just too slow.. Why?

There are at least 3 ways to do same things when you code the era or mod.
1) Using Macros
2) Using Fire Events
3) Using Lua

In my opinion they all have problems.. Lua might be problem-free if you know it at expert level, which I don't even after spending couple years coding some wesnoth eras/mods. And I still would not use lua alone to code anything..

So In here I will throw my experiences and comparison when trying to optimise something using these 3 methods.. Real speed difference.. real megabite per savefile difference.. etc
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave »

MACRO vs FIRE_EVENT in random map generation cycle.. version 1.12 of course..

TIMING MACRO MMBE_RANDOMIZE_CURRENT_HEX

Code: Select all

[set_variable]
   name=check_time_before
   time=stamp
[/set_variable]
	{FOREACH mmbe_hex i}
	   {MMBE_RANDOMIZE_CURRENT_HEX}
	{NEXT i}
[set_variable]
   name=check_time_after
   time=stamp
[/set_variable]
When hosting a game, the difference between check time before and after was 3112, 3058 and it felt quick..

TIMING FIRE EVENT mmbe_randomize_current_hex

Code: Select all

[set_variable]
   name=check_time_before
   time=stamp
[/set_variable]
	{FOREACH mmbe_hex i}
	   {FIRE_EVENT mmbe_randomize_current_hex}
	{NEXT i}
[set_variable]
   name=check_time_after
   time=stamp
[/set_variable]
When hosting a game, the difference between check time before and after was 23563, 25496 and it felt SLOW..

So hosting 2 times Isar Cross with macro and fire_event has shown that same code with macro is nearly 10 times quicker than same code with fire_event when used in cycle.. FOREACH, where mmbe hex is every hex of Isar cross with some exclusions.. (for example hexes with castles or villages not getting randomized so they are not included into mmbe_hex array.. as per settings)

And that's the code for fire_event {FIRE_EVENT mmbe_randomize_current_hex} or {MMBE_RANDOMIZE_CURRENT_HEX} macro. If you having difficulties to read it due to bad coding culture, please don't, I code bad, this is a long macro.. and i will spend a lot of time changing it to your preference. While it really doesn't matter... the only thing that matters in my opionion is that you should use fire_events everywhere you can EXCEPT when you doing a HUGE calculation that requires FAST SPEED.
Spoiler:
If anyone has questions, for example would like me to provide all macros I used, feel free to ask. For now it seems to be obvious that macro wins speed race vs fire_event at least in certain circumstances (I think most of the time macro is faster, specially in case if there are not too many macros used and if the whole code of whole era is not too big). I will modify my codes into lua some time later and we will see how much quicker it would do same thing in lua.. But I already seen how same things may take 1 sek in lua, while macros would last at least 5 sek.. so I expect to see a big speed gain, but let's not talk about it until it's tested..
User avatar
Ravana
Forum Moderator
Posts: 2949
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Ravana »

Main benefit of fire_event is not duplicating your code to all places where you call it. If you only use the code from one place, then macro doesnt hurt you.
User avatar
Pentarctagon
Project Manager
Posts: 5527
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Pentarctagon »

Ravana wrote:Main benefit of fire_event is not duplicating your code to all places where you call it. If you only use the code from one place, then macro doesnt hurt you.
More specifically: not duplicating your code after preprocessing is done. If all you care about is not having duplicate code for the same functionality everywhere, then macros are fine. The risk then becomes that your WML ends up taking up MBs of space after preprocessing is done, but if you don't care about that, then it isn't much of a risk.

What would be somewhat interesting though, is if there was a way to load a particular chunk of WML from the cache, rather than keeping all WML in the save file. That would at least prevent the save files themselves from taking up a huge amount of space.

-edit-
Also, my understanding as far as lua goes, is that it generally isn't any more performant than WML. It's strength is that it can do a lot more than WML in certain cases (things involving string manipulation, for example), and that it can be used to create new WML tags.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
gfgtdf
Developer
Posts: 1432
Joined: February 10th, 2013, 2:25 pm

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by gfgtdf »

Pentarctagon wrote: Also, my understanding as far as lua goes, is that it generally isn't any more performant than WML.
This greatly depnds on what you are trying to do, exspecialyl very basic tags ([set_variable], [while] etc.) are much slower than their lua conterparts, you can see this easily since thoe tags are implemntted in lua, for example [set_varialbe] name,add=a,5 [/set_varialbe] is equivalent to the lua code

Code: Select all


function wesnoth.wml_actions.set_variable(cfg)
	local name = cfg.name or helper.wml_error "trying to set a variable with an empty name"

	if cfg.value ~= nil then -- check for nil because user may try to set a variable as false
		wesnoth.set_variable(name, cfg.value)
	end

	if cfg.literal ~= nil then
		wesnoth.set_variable(name, helper.shallow_literal(cfg).literal)
	end

	if cfg.to_variable then
		wesnoth.set_variable(name, wesnoth.get_variable(cfg.to_variable))
	end

	if cfg.add then
		wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) + (tonumber(cfg.add) or 0))
	end

	if cfg.sub then
		wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) - (tonumber(cfg.sub) or 0))
	end

	if cfg.multiply then
		wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) * (tonumber(cfg.multiply) or 0))
	end

	if cfg.divide then
		local divide = tonumber(cfg.divide) or 0
		if divide == 0 then helper.wml_error("division by zero on variable " .. name) end
		wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) / divide)
	end

	if cfg.modulo then
		local modulo = tonumber(cfg.modulo) or 0
		if modulo == 0 then helper.wml_error("division by zero on variable " .. name) end
		wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) % modulo)
	end

	if cfg.abs then
		wesnoth.set_variable(name, math.abs(tonumber(wesnoth.get_variable(name)) or 0))
	end

	if cfg.root then
		if cfg.root == "square" then
			local radicand = tonumber(wesnoth.get_variable(name)) or 0
			if radicand < 0 then helper.wml_error("square root of negative number on variable " .. name) end
			wesnoth.set_variable(name, math.sqrt(radicand))
		end
	end

	if cfg.power then
		wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) ^ (tonumber(cfg.power) or 0))
	end

	if cfg.round then
		local var = tonumber(wesnoth.get_variable(name) or 0)
		local round_val = cfg.round
		if round_val == "ceil" then
			wesnoth.set_variable(name, math.ceil(var))
		elseif round_val == "floor" then
			wesnoth.set_variable(name, math.floor(var))
		else
			local decimals, discarded = math.modf(tonumber(round_val) or 0)
			local value = var * (10 ^ decimals)
			value = helper.round(value)
			value = value * (10 ^ -decimals)
			wesnoth.set_variable(name, value)
		end
	end

	-- unlike the other math operations, ipart and fpart do not act on
	-- the value already contained in the variable
	-- but on the value assigned to the respective key
	if cfg.ipart then
		local ivalue, fvalue = math.modf(tonumber(cfg.ipart) or 0)
		wesnoth.set_variable(name, ivalue)
	end

	if cfg.fpart then
		local ivalue, fvalue = math.modf(tonumber(cfg.fpart) or 0)
		wesnoth.set_variable(name, fvalue)
	end

	if cfg.string_length ~= nil then
		wesnoth.set_variable(name, string.len(tostring(cfg.string_length)))
	end

	if cfg.time then
		if cfg.time == "stamp" then
			wesnoth.set_variable(name, wesnoth.get_time_stamp())
		end
	end

	if cfg.rand then
		wesnoth.set_variable(name, helper.rand(tostring(cfg.rand)))
	end

	local join_child = helper.get_child(cfg, "join")
	if join_child then
		local array_name = join_child.variable or helper.wml_error "missing variable= attribute in [join]"
		local separator = join_child.separator
		local key_name = join_child.key or "value"
		local remove_empty = join_child.remove_empty

		local string_to_join = ''

		for i, element in ipairs(helper.get_variable_array(array_name)) do
			if element[key_name] ~= nil or (not remove_empty) then
				if #string_to_join > 0 then
					string_to_join = string_to_join .. separator
				end
				string_to_join = string_to_join .. element[key_name]
			end
		end

		wesnoth.set_variable(name, string_to_join)
	end
end

wesnoth.wml_actions.set_variable {name="a", add=1}
ofc this most of these ifs evalualte to false, but its still much slower than the lua equivlent of just a = a + 5


Another case is when you are changing units stats, in wml this often requires storing and unstoring the unit which is a very slow process since all the attributes need to be first saved and then the units is complteley rebuild, even if you just wanted to change one attribute (like hitpoints). In lua you can changw those attributes directly witout havng to unstore/store anything.


A case where it won't match that much are tags that are more or less quivelent to their internal lua function, for example [store_locations], whether lua or wml, the filter will usually be applied to all locations on the initial set and the effecieness of the filter is much more important then the potential overhead of using wml here.

EDIT: It would be nice to have a [wmlcode] phpbb tag that would bhave like

Code: Select all

 but automatically run wmlindent on it.
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.
User avatar
zookeeper
WML Wizard
Posts: 9742
Joined: September 11th, 2004, 10:40 pm
Location: Finland

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by zookeeper »

enclave wrote:MACRO vs FIRE_EVENT in random map generation cycle.. version 1.12 of course..
That isn't about "MACRO vs FIRE_EVENT". You're using a macro in both cases in the first place.

As was already said, almost always the point of using [fire_event] "instead of a macro" is to avoid code duplication. In your example there is no code duplication. Your example is a demonstration only of how [fire_event] does have an inherent performance cost, meaning that it might not be a good idea to use it in a loop. Which is certainly a good thing to be aware of when writing your code, but it doesn't have anything to do with macros.
User avatar
Pentarctagon
Project Manager
Posts: 5527
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Pentarctagon »

gfgtdf wrote:...words...
It sounds like what would be a more interesting test then, would be the difference between using the VARIABLE macro vs inline lua using wesnoth.set_variable.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Ravana
Forum Moderator
Posts: 2949
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Ravana »

Or would it make sense to define VARIABLE to use lua in the first place.
gfgtdf
Developer
Posts: 1432
Joined: February 10th, 2013, 2:25 pm

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by gfgtdf »

Pentarctagon wrote:
gfgtdf wrote:...words...
It sounds like what would be a more interesting test then, would be the difference between using the VARIABLE macro vs inline lua using wesnoth.set_variable.

well inline [lua] would need to compile the lua at runtime first, so it's unclear whether [lua] woudl be faster than [set_variable] for that, but note that you (or rather i) usually don't replace single tags with lua but the whole functions/event . That's also why i compared it to a simple a = a + 5 since it's quite likleley that a would become a local variable if you port the whole event/function to lua.
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.
User avatar
Pentarctagon
Project Manager
Posts: 5527
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Pentarctagon »

gfgtdf wrote:well inline [lua] would need to compile the lua at runtime first, so it's unclear whether [lua] woudl be faster than [set_variable] for that, but note that you (or rather i) usually don't replace single tags with lua but the whole functions/event . That's also why i compared it to a simple a = a + 5 since it's quite likleley that a would become a local variable if you port the whole event/function to lua.
Which is why it would make an interesting test. So, something like:
  • VARIABLE
  • inline lua
  • some sort of WML tag like [wesnoth_set_variable] that only calls wesnoth.set_variable rather than doing all the other checks like [set_variable] does
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave »

wow thats a big conversation here ;) anyone would like to share results? :) I wouldnt be able to do many lua tests, because i'm too bad at them. I will only write my particular MACRO vs FIRE_EVENT code with a bit of lua soon when I get a time for it. If somebody could share your example would be really cool :)
zookeeper wrote: That isn't about "MACRO vs FIRE_EVENT". You're using a macro in both cases in the first place.
Your example is a demonstration only of how [fire_event] does have an inherent performance cost, meaning that it might not be a good idea to use it in a loop. Which is certainly a good thing to be aware of when writing your code, but it doesn't have anything to do with macros.
Sorry I can't understand your text.. or point behind your text.
I think I demonstrated quite good a particular case when macro and fire_event are used to represent the same code as alternative to a function/procedure. Obviously they all contain macros.. and I think it's even more obvious that if I replaced every single macro of that code with [fire_event] then performance would be decreased progressively. I think you agreed with that in your text.. But I still don't understand what you were trying to say..
User avatar
Pentarctagon
Project Manager
Posts: 5527
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Pentarctagon »

enclave wrote:wow thats a big conversation here ;) anyone would like to share results? :) I wouldnt be able to do many lua tests, because i'm too bad at them. I will only write my particular MACRO vs FIRE_EVENT code with a bit of lua soon when I get a time for it. If somebody could share your example would be really cool :)
zookeeper wrote: That isn't about "MACRO vs FIRE_EVENT". You're using a macro in both cases in the first place.
Your example is a demonstration only of how [fire_event] does have an inherent performance cost, meaning that it might not be a good idea to use it in a loop. Which is certainly a good thing to be aware of when writing your code, but it doesn't have anything to do with macros.
Sorry I can't understand your text.. or point behind your text.
I think I demonstrated quite good a particular case when macro and fire_event are used to represent the same code as alternative to a function/procedure. Obviously they all contain macros.. and I think it's even more obvious that if I replaced every single macro of that code with [fire_event] then performance would be decreased progressively. I think you agreed with that in your text.. But I still don't understand what you were trying to say..
It's also probably worth mentioning that there's no runtime performance penalty to using macros vs just writing out the WML yourself. The preprocessor does all the macro substitution beforehand and caches it. This is why, for example, the WML I've written for my add-on is ~9.8k lines, while the cached WML from my add-on after the preprocessor runs is over 63k lines.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave »

I had serious optimization problems with my add-on written without fire events.. People could not load the savefile because it was exceeding 40mb maximum allowed size.. So basically I had to convert a lot of my macros into fire events to try to reduce the savefile size. After reading about optimization in wiki it felt like fire_event is the answer to everything.. but turns out that if you need speed, then fire_event plays against you in certain circumstances and it was a big surprise for me to find it out.. therefore I think it's very important to talk about it :D
User avatar
Pentarctagon
Project Manager
Posts: 5527
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by Pentarctagon »

enclave wrote:I had serious optimization problems with my add-on written without fire events.. People could not load the savefile because it was exceeding 40mb maximum allowed size.. So basically I had to convert a lot of my macros into fire events to try to reduce the savefile size. After reading about optimization in wiki it felt like fire_event is the answer to everything.. but turns out that if you need speed, then fire_event plays against you in certain circumstances and it was a big surprise for me to find it out.. therefore I think it's very important to talk about it :D
In which case, it'd be good to specify which type of optimization you're talking about - speed or size. Macros are better in terms of speed, but can take up additional space. [fire_event] reduces the amount of space needed for save files, but has a performance penalty.

-edit-
Did a quick test:

Code: Select all

[set_variable]
  name=start
  time=stamp
[/set_variable]

[for]
  start=0
  end=500000
  step=1
  [do]
    
    {VARIABLE test 5}
    
  [/do]
[/for]

[set_variable]
  name=end
  time=stamp
[/set_variable]
[message]
  message="$start - $end"
[/message]
gave a start of 7017 and an end of 11502, for a difference of 4485.

Code: Select all

[set_variable]
  name=start
  time=stamp
[/set_variable]

[for]
  start=0
  end=500000
  step=1
  [do]
    
    [lua]
      code=<<
        wesnoth.set_variable("test", 5)
      >>
    [/lua]
    
  [/do]
[/for]

[set_variable]
  name=end
  time=stamp
[/set_variable]
[message]
  message="$start - $end"
[/message]
gave a start of 6609 and an end of 10308, for a difference of 3699.

Code: Select all

function wesnoth.wml_actions.set_test_variable(var)
  wesnoth.set_variable(var.variable, 5)
end
and

Code: Select all

[set_variable]
  name=start
  time=stamp
[/set_variable]

[for]
  start=0
  end=500000
  step=1
  [do]
    
    [set_test_variable]
      variable=test
    [/set_test_variable]
    
  [/do]
[/for]

[set_variable]
  name=end
  time=stamp
[/set_variable]
[message]
  message="$start - $end"
[/message]
gave a start of 6765 and an end of 9171, for a difference of 2406.

So directly calling wesnoth.set_variable via a simpler WML tag was close to 2x faster, but given that it took 500,000 calls to create a difference of ~2 seconds, actually writing code this way seems beyond even micro-optimization in most cases.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
enclave
Posts: 936
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Post by enclave »

Nice :)
Have you tried multiple times? Because there is always a chance that your computer was performing some other tasks at the background.. and at the point of testing the performance of wesnoth could be decreased by tiny bit, enough to change the results. I will actually copy your code and test the same, just curiosity of :)
Post Reply