very slow wesnoth_addon_manager

Having trouble with the game? Report issues and get help here. Read this first!

Moderator: Forum Moderators

Forum rules
Before reporting issues in this section, you must read the following topic:
Soliton
Site Administrator
Posts: 1679
Joined: April 5th, 2005, 3:25 pm
Location: #wesnoth-mp

Re: very slow wesnoth_addon_manager

Post by Soliton »

Looks like this was not updated to python 3 yet. Replace file(name, "w").write(mythread.data) with with open(name, "wb") as f: f.write(mythread.data)
"If gameplay requires it, they can be made to live on Venus." -- scott
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

Re: very slow wesnoth_addon_manager

Post by vghetto »

Thank you Soliton, that fixed --raw_download.

I tried doing the unpacking with --unpack and it threw a similar error.

Code: Select all

$ ../tools/wesnoth_addon_manager -U Wild_Frontiers -c add-ons
Opening socket to add-ons.wesnoth.org:15014 for 1.14.x
Connected as 42.
Traceback (most recent call last):
  File "../tools/wesnoth_addon_manager", line 291, in <module>
    data = file(args.unpack).read()
NameError: name 'file' is not defined
Closing socket.
User avatar
Elvish_Hunter
Posts: 1575
Joined: September 4th, 2009, 2:39 pm
Location: Lintanir Forest...

Re: very slow wesnoth_addon_manager

Post by Elvish_Hunter »

Good catch. I just pushed a fix for these errors both in master (c72ea20) and 1.14 (fab37ca979a1c9625eace6e25b9cb60bddf70a36).
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
gnombat
Posts: 671
Joined: June 10th, 2010, 8:49 pm

Re: very slow wesnoth_addon_manager

Post by gnombat »

gnombat wrote: April 11th, 2021, 4:04 pm I suspect that the problem is that wesnoth_addon_manager is written in Python, which is abominably slow
Actually, upon further investigation, I think that the problem is that the parser uses bytes objects everywhere, which causes the parse time to increase quadratically with respect to add-on size. Using bytearray instead would probably improve performance for large add-ons.
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

Re: very slow wesnoth_addon_manager

Post by vghetto »

gnombat wrote: April 13th, 2021, 1:38 am Actually, upon further investigation, I think that the problem is that the parser uses bytes objects everywhere, which causes the parse time to increase quadratically with respect to add-on size. Using bytearray instead would probably improve performance for large add-ons.
Is it as simple as find/replace bytes into bytearray or is it more than that?
Because I can try doing that and report back the results if it is so.
User avatar
Elvish_Hunter
Posts: 1575
Joined: September 4th, 2009, 2:39 pm
Location: Lintanir Forest...

Re: very slow wesnoth_addon_manager

Post by Elvish_Hunter »

There's more. For example, to create an empty bytearray you can't just use b"", but you have to use bytearray() (and there are other differences when constructing such objects). You can create a bytearray from bytes with bytearray(b"My sequence of bytes"). Bytearrays also support mutable object methods, like the ones available for lists, which bytes object don't support. Some functions expect bytes objects as arguments, so you'll have to cast bytearrays to bytes with bytes(my_bytearray).
Other than that, all methods and operators supported by bytes seem to be supported by bytearrays as well, so you can give it a try and let us know.
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
gnombat
Posts: 671
Joined: June 10th, 2010, 8:49 pm

Re: very slow wesnoth_addon_manager

Post by gnombat »

I'm not very familiar with programming in Python, so I'm not sure exactly what needs to be changed, but the basic idea is to do something like this:

Code: Select all

--- data/tools/wesnoth/wmlparser3.py
+++ data/tools/wesnoth/wmlparser3.py
@@ -448,7 +448,7 @@ class Parser:
                 self.in_arrows = False
                 self.temp_string += line[:arrows]
                 self.temp_string_node = StringNode(self.temp_string)
-                self.temp_string = b""
+                self.temp_string = bytearray()
                 self.temp_key_nodes[self.commas].value.append(
                     self.temp_string_node)
                 self.in_arrows = False
@@ -477,7 +477,7 @@ class Parser:
                 if self.translatable:
                     self.temp_string_node.textdomain = self.textdomain
                     self.translatable = False
-                self.temp_string = b""
+                self.temp_string = bytearray()
                 if not self.temp_key_nodes:
                     raise WMLError(self, "Unexpected string value.")

@@ -587,7 +587,7 @@ class Parser:
             for subsegment in segment.split(b",", maxsplit):
                 self.temp_string += subsegment.strip()
                 self.temp_string_node = StringNode(self.temp_string)
-                self.temp_string = b""
+                self.temp_string = bytearray()
                 self.temp_key_nodes[self.commas].value.append(
                     self.temp_string_node)
                 if self.commas < n - 1:
@@ -605,7 +605,7 @@ class Parser:
         """

         # parsing state
-        self.temp_string = b""
+        self.temp_string = bytearray()
         self.temp_string_node = None
         self.commas = 0
         self.temp_key_nodes = []
Note that this patch is not quite correct, because bytearray is not the same type as bytes (in particular it means that the type signature for the StringNode class is not right any more). However, it does appear to work at least with the wesnoth_addon_manager program.
Post Reply