Optimisation Discussions Advises Experiences Ideas Etc

Discussion of all aspects of the game engine, including development of new and existing features.

Moderators: Forum Moderators, Developers

Optimisation Discussions Advises Experiences Ideas Etc

Postby enclave » June 8th, 2017, 9:35 pm

Some of us have created eras or modifications. They work.

But there may be not obvious problems.. The code is working fine! But...
Maybe you can not reload your saved game because save file code is taking too much space..
Maybe your random map generator takes ages to generate map and most users leave before map was even loaded. Maybe something works just too slow.. Why?

There are at least 3 ways to do same things when you code the era or mod.
1) Using Macros
2) Using Fire Events
3) Using Lua

In my opinion they all have problems.. Lua might be problem-free if you know it at expert level, which I don't even after spending couple years coding some wesnoth eras/mods. And I still would not use lua alone to code anything..

So In here I will throw my experiences and comparison when trying to optimise something using these 3 methods.. Real speed difference.. real megabite per savefile difference.. etc
enclave
 
Posts: 533
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby enclave » June 8th, 2017, 10:10 pm

MACRO vs FIRE_EVENT in random map generation cycle.. version 1.12 of course..

TIMING MACRO MMBE_RANDOMIZE_CURRENT_HEX
Code: Select all
[set_variable]
   name=check_time_before
   time=stamp
[/set_variable]
   {FOREACH mmbe_hex i}
      {MMBE_RANDOMIZE_CURRENT_HEX}
   {NEXT i}
[set_variable]
   name=check_time_after
   time=stamp
[/set_variable]

When hosting a game, the difference between check time before and after was 3112, 3058 and it felt quick..

TIMING FIRE EVENT mmbe_randomize_current_hex
Code: Select all
[set_variable]
   name=check_time_before
   time=stamp
[/set_variable]
   {FOREACH mmbe_hex i}
      {FIRE_EVENT mmbe_randomize_current_hex}
   {NEXT i}
[set_variable]
   name=check_time_after
   time=stamp
[/set_variable]

When hosting a game, the difference between check time before and after was 23563, 25496 and it felt SLOW..

So hosting 2 times Isar Cross with macro and fire_event has shown that same code with macro is nearly 10 times quicker than same code with fire_event when used in cycle.. FOREACH, where mmbe hex is every hex of Isar cross with some exclusions.. (for example hexes with castles or villages not getting randomized so they are not included into mmbe_hex array.. as per settings)

And that's the code for fire_event {FIRE_EVENT mmbe_randomize_current_hex} or {MMBE_RANDOMIZE_CURRENT_HEX} macro. If you having difficulties to read it due to bad coding culture, please don't, I code bad, this is a long macro.. and i will spend a lot of time changing it to your preference. While it really doesn't matter... the only thing that matters in my opionion is that you should use fire_events everywhere you can EXCEPT when you doing a HUGE calculation that requires FAST SPEED.
Spoiler:


If anyone has questions, for example would like me to provide all macros I used, feel free to ask. For now it seems to be obvious that macro wins speed race vs fire_event at least in certain circumstances (I think most of the time macro is faster, specially in case if there are not too many macros used and if the whole code of whole era is not too big). I will modify my codes into lua some time later and we will see how much quicker it would do same thing in lua.. But I already seen how same things may take 1 sek in lua, while macros would last at least 5 sek.. so I expect to see a big speed gain, but let's not talk about it until it's tested..
enclave
 
Posts: 533
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby Ravana » June 8th, 2017, 10:20 pm

Main benefit of fire_event is not duplicating your code to all places where you call it. If you only use the code from one place, then macro doesnt hurt you.
User avatar
Ravana
Moderator
 
Posts: 1460
Joined: January 29th, 2012, 12:49 am
Location: Estonia

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby Pentarctagon » June 9th, 2017, 2:18 pm

Ravana wrote:Main benefit of fire_event is not duplicating your code to all places where you call it. If you only use the code from one place, then macro doesnt hurt you.


More specifically: not duplicating your code after preprocessing is done. If all you care about is not having duplicate code for the same functionality everywhere, then macros are fine. The risk then becomes that your WML ends up taking up MBs of space after preprocessing is done, but if you don't care about that, then it isn't much of a risk.

What would be somewhat interesting though, is if there was a way to load a particular chunk of WML from the cache, rather than keeping all WML in the save file. That would at least prevent the save files themselves from taking up a huge amount of space.

-edit-
Also, my understanding as far as lua goes, is that it generally isn't any more performant than WML. It's strength is that it can do a lot more than WML in certain cases (things involving string manipulation, for example), and that it can be used to create new WML tags.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Pentarctagon
Forum Administrator
 
Posts: 2949
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby gfgtdf » June 9th, 2017, 5:48 pm

Pentarctagon wrote:Also, my understanding as far as lua goes, is that it generally isn't any more performant than WML.

This greatly depnds on what you are trying to do, exspecialyl very basic tags ([set_variable], [while] etc.) are much slower than their lua conterparts, you can see this easily since thoe tags are implemntted in lua, for example [set_varialbe] name,add=a,5 [/set_varialbe] is equivalent to the lua code

Code: Select all

function wesnoth.wml_actions.set_variable(cfg)
   local name = cfg.name or helper.wml_error "trying to set a variable with an empty name"

   if cfg.value ~= nil then -- check for nil because user may try to set a variable as false
      wesnoth.set_variable(name, cfg.value)
   end

   if cfg.literal ~= nil then
      wesnoth.set_variable(name, helper.shallow_literal(cfg).literal)
   end

   if cfg.to_variable then
      wesnoth.set_variable(name, wesnoth.get_variable(cfg.to_variable))
   end

   if cfg.add then
      wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) + (tonumber(cfg.add) or 0))
   end

   if cfg.sub then
      wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) - (tonumber(cfg.sub) or 0))
   end

   if cfg.multiply then
      wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) * (tonumber(cfg.multiply) or 0))
   end

   if cfg.divide then
      local divide = tonumber(cfg.divide) or 0
      if divide == 0 then helper.wml_error("division by zero on variable " .. name) end
      wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) / divide)
   end

   if cfg.modulo then
      local modulo = tonumber(cfg.modulo) or 0
      if modulo == 0 then helper.wml_error("division by zero on variable " .. name) end
      wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) % modulo)
   end

   if cfg.abs then
      wesnoth.set_variable(name, math.abs(tonumber(wesnoth.get_variable(name)) or 0))
   end

   if cfg.root then
      if cfg.root == "square" then
         local radicand = tonumber(wesnoth.get_variable(name)) or 0
         if radicand < 0 then helper.wml_error("square root of negative number on variable " .. name) end
         wesnoth.set_variable(name, math.sqrt(radicand))
      end
   end

   if cfg.power then
      wesnoth.set_variable(name, (tonumber(wesnoth.get_variable(name)) or 0) ^ (tonumber(cfg.power) or 0))
   end

   if cfg.round then
      local var = tonumber(wesnoth.get_variable(name) or 0)
      local round_val = cfg.round
      if round_val == "ceil" then
         wesnoth.set_variable(name, math.ceil(var))
      elseif round_val == "floor" then
         wesnoth.set_variable(name, math.floor(var))
      else
         local decimals, discarded = math.modf(tonumber(round_val) or 0)
         local value = var * (10 ^ decimals)
         value = helper.round(value)
         value = value * (10 ^ -decimals)
         wesnoth.set_variable(name, value)
      end
   end

   -- unlike the other math operations, ipart and fpart do not act on
   -- the value already contained in the variable
   -- but on the value assigned to the respective key
   if cfg.ipart then
      local ivalue, fvalue = math.modf(tonumber(cfg.ipart) or 0)
      wesnoth.set_variable(name, ivalue)
   end

   if cfg.fpart then
      local ivalue, fvalue = math.modf(tonumber(cfg.fpart) or 0)
      wesnoth.set_variable(name, fvalue)
   end

   if cfg.string_length ~= nil then
      wesnoth.set_variable(name, string.len(tostring(cfg.string_length)))
   end

   if cfg.time then
      if cfg.time == "stamp" then
         wesnoth.set_variable(name, wesnoth.get_time_stamp())
      end
   end

   if cfg.rand then
      wesnoth.set_variable(name, helper.rand(tostring(cfg.rand)))
   end

   local join_child = helper.get_child(cfg, "join")
   if join_child then
      local array_name = join_child.variable or helper.wml_error "missing variable= attribute in [join]"
      local separator = join_child.separator
      local key_name = join_child.key or "value"
      local remove_empty = join_child.remove_empty

      local string_to_join = ''

      for i, element in ipairs(helper.get_variable_array(array_name)) do
         if element[key_name] ~= nil or (not remove_empty) then
            if #string_to_join > 0 then
               string_to_join = string_to_join .. separator
            end
            string_to_join = string_to_join .. element[key_name]
         end
      end

      wesnoth.set_variable(name, string_to_join)
   end
end

wesnoth.wml_actions.set_variable {name="a", add=1}

ofc this most of these ifs evalualte to false, but its still much slower than the lua equivlent of just a = a + 5


Another case is when you are changing units stats, in wml this often requires storing and unstoring the unit which is a very slow process since all the attributes need to be first saved and then the units is complteley rebuild, even if you just wanted to change one attribute (like hitpoints). In lua you can changw those attributes directly witout havng to unstore/store anything.


A case where it won't match that much are tags that are more or less quivelent to their internal lua function, for example [store_locations], whether lua or wml, the filter will usually be applied to all locations on the initial set and the effecieness of the filter is much more important then the potential overhead of using wml here.

EDIT: It would be nice to have a [wmlcode] phpbb tag that would bhave like [code] but automatically run wmlindent on it.
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.
gfgtdf
Developer
 
Posts: 910
Joined: February 10th, 2013, 2:25 pm

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby zookeeper » June 9th, 2017, 6:43 pm

enclave wrote:MACRO vs FIRE_EVENT in random map generation cycle.. version 1.12 of course..

That isn't about "MACRO vs FIRE_EVENT". You're using a macro in both cases in the first place.

As was already said, almost always the point of using [fire_event] "instead of a macro" is to avoid code duplication. In your example there is no code duplication. Your example is a demonstration only of how [fire_event] does have an inherent performance cost, meaning that it might not be a good idea to use it in a loop. Which is certainly a good thing to be aware of when writing your code, but it doesn't have anything to do with macros.
User avatar
zookeeper
WML Wizard
 
Posts: 9610
Joined: September 11th, 2004, 10:40 pm
Location: Finland

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby Pentarctagon » June 9th, 2017, 7:22 pm

gfgtdf wrote:...words...


It sounds like what would be a more interesting test then, would be the difference between using the VARIABLE macro vs inline lua using wesnoth.set_variable.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Pentarctagon
Forum Administrator
 
Posts: 2949
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby Ravana » June 9th, 2017, 7:26 pm

Or would it make sense to define VARIABLE to use lua in the first place.
User avatar
Ravana
Moderator
 
Posts: 1460
Joined: January 29th, 2012, 12:49 am
Location: Estonia

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby gfgtdf » June 9th, 2017, 7:32 pm

Pentarctagon wrote:
gfgtdf wrote:...words...


It sounds like what would be a more interesting test then, would be the difference between using the VARIABLE macro vs inline lua using wesnoth.set_variable.



well inline [lua] would need to compile the lua at runtime first, so it's unclear whether [lua] woudl be faster than [set_variable] for that, but note that you (or rather i) usually don't replace single tags with lua but the whole functions/event . That's also why i compared it to a simple a = a + 5 since it's quite likleley that a would become a local variable if you port the whole event/function to lua.
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.
gfgtdf
Developer
 
Posts: 910
Joined: February 10th, 2013, 2:25 pm

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby Pentarctagon » June 9th, 2017, 7:46 pm

gfgtdf wrote:well inline [lua] would need to compile the lua at runtime first, so it's unclear whether [lua] woudl be faster than [set_variable] for that, but note that you (or rather i) usually don't replace single tags with lua but the whole functions/event . That's also why i compared it to a simple a = a + 5 since it's quite likleley that a would become a local variable if you port the whole event/function to lua.


Which is why it would make an interesting test. So, something like:
  • VARIABLE
  • inline lua
  • some sort of WML tag like [wesnoth_set_variable] that only calls wesnoth.set_variable rather than doing all the other checks like [set_variable] does
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Pentarctagon
Forum Administrator
 
Posts: 2949
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby enclave » June 9th, 2017, 8:28 pm

wow thats a big conversation here ;) anyone would like to share results? :) I wouldnt be able to do many lua tests, because i'm too bad at them. I will only write my particular MACRO vs FIRE_EVENT code with a bit of lua soon when I get a time for it. If somebody could share your example would be really cool :)

zookeeper wrote:That isn't about "MACRO vs FIRE_EVENT". You're using a macro in both cases in the first place.
Your example is a demonstration only of how [fire_event] does have an inherent performance cost, meaning that it might not be a good idea to use it in a loop. Which is certainly a good thing to be aware of when writing your code, but it doesn't have anything to do with macros.

Sorry I can't understand your text.. or point behind your text.
I think I demonstrated quite good a particular case when macro and fire_event are used to represent the same code as alternative to a function/procedure. Obviously they all contain macros.. and I think it's even more obvious that if I replaced every single macro of that code with [fire_event] then performance would be decreased progressively. I think you agreed with that in your text.. But I still don't understand what you were trying to say..
enclave
 
Posts: 533
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby Pentarctagon » June 9th, 2017, 8:38 pm

enclave wrote:wow thats a big conversation here ;) anyone would like to share results? :) I wouldnt be able to do many lua tests, because i'm too bad at them. I will only write my particular MACRO vs FIRE_EVENT code with a bit of lua soon when I get a time for it. If somebody could share your example would be really cool :)

zookeeper wrote:That isn't about "MACRO vs FIRE_EVENT". You're using a macro in both cases in the first place.
Your example is a demonstration only of how [fire_event] does have an inherent performance cost, meaning that it might not be a good idea to use it in a loop. Which is certainly a good thing to be aware of when writing your code, but it doesn't have anything to do with macros.

Sorry I can't understand your text.. or point behind your text.
I think I demonstrated quite good a particular case when macro and fire_event are used to represent the same code as alternative to a function/procedure. Obviously they all contain macros.. and I think it's even more obvious that if I replaced every single macro of that code with [fire_event] then performance would be decreased progressively. I think you agreed with that in your text.. But I still don't understand what you were trying to say..


It's also probably worth mentioning that there's no runtime performance penalty to using macros vs just writing out the WML yourself. The preprocessor does all the macro substitution beforehand and caches it. This is why, for example, the WML I've written for my add-on is ~9.8k lines, while the cached WML from my add-on after the preprocessor runs is over 63k lines.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Pentarctagon
Forum Administrator
 
Posts: 2949
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby enclave » June 9th, 2017, 8:54 pm

I had serious optimization problems with my add-on written without fire events.. People could not load the savefile because it was exceeding 40mb maximum allowed size.. So basically I had to convert a lot of my macros into fire events to try to reduce the savefile size. After reading about optimization in wiki it felt like fire_event is the answer to everything.. but turns out that if you need speed, then fire_event plays against you in certain circumstances and it was a big surprise for me to find it out.. therefore I think it's very important to talk about it :D
enclave
 
Posts: 533
Joined: December 15th, 2007, 8:52 am

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby Pentarctagon » June 9th, 2017, 9:39 pm

enclave wrote:I had serious optimization problems with my add-on written without fire events.. People could not load the savefile because it was exceeding 40mb maximum allowed size.. So basically I had to convert a lot of my macros into fire events to try to reduce the savefile size. After reading about optimization in wiki it felt like fire_event is the answer to everything.. but turns out that if you need speed, then fire_event plays against you in certain circumstances and it was a big surprise for me to find it out.. therefore I think it's very important to talk about it :D


In which case, it'd be good to specify which type of optimization you're talking about - speed or size. Macros are better in terms of speed, but can take up additional space. [fire_event] reduces the amount of space needed for save files, but has a performance penalty.

-edit-
Did a quick test:
Code: Select all
[set_variable]
  name=start
  time=stamp
[/set_variable]

[for]
  start=0
  end=500000
  step=1
  [do]
   
    {VARIABLE test 5}
   
  [/do]
[/for]

[set_variable]
  name=end
  time=stamp
[/set_variable]
[message]
  message="$start - $end"
[/message]

gave a start of 7017 and an end of 11502, for a difference of 4485.

Code: Select all
[set_variable]
  name=start
  time=stamp
[/set_variable]

[for]
  start=0
  end=500000
  step=1
  [do]
   
    [lua]
      code=<<
        wesnoth.set_variable("test", 5)
      >>
    [/lua]
   
  [/do]
[/for]

[set_variable]
  name=end
  time=stamp
[/set_variable]
[message]
  message="$start - $end"
[/message]

gave a start of 6609 and an end of 10308, for a difference of 3699.

Code: Select all
function wesnoth.wml_actions.set_test_variable(var)
  wesnoth.set_variable(var.variable, 5)
end

and
Code: Select all
[set_variable]
  name=start
  time=stamp
[/set_variable]

[for]
  start=0
  end=500000
  step=1
  [do]
   
    [set_test_variable]
      variable=test
    [/set_test_variable]
   
  [/do]
[/for]

[set_variable]
  name=end
  time=stamp
[/set_variable]
[message]
  message="$start - $end"
[/message]

gave a start of 6765 and an end of 9171, for a difference of 2406.

So directly calling wesnoth.set_variable via a simpler WML tag was close to 2x faster, but given that it took 500,000 calls to create a difference of ~2 seconds, actually writing code this way seems beyond even micro-optimization in most cases.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Pentarctagon
Forum Administrator
 
Posts: 2949
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Optimisation Discussions Advises Experiences Ideas Etc

Postby enclave » June 10th, 2017, 4:10 pm

Nice :)
Have you tried multiple times? Because there is always a chance that your computer was performing some other tasks at the background.. and at the point of testing the performance of wesnoth could be decreased by tiny bit, enough to change the results. I will actually copy your code and test the same, just curiosity of :)
enclave
 
Posts: 533
Joined: December 15th, 2007, 8:52 am

Next

Return to Coder’s Corner

Who is online

Users browsing this forum: No registered users and 1 guest