very slow wesnoth_addon_manager

Having trouble with the game? Report issues and get help here. Read this first!

Moderator: Forum Moderators

Forum rules
Before reporting issues in this section, you must read the following topic:
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

very slow wesnoth_addon_manager

Post by vghetto »

Hi,

When a big mod (50+ MB) is downloaded with wesnoth_addon_manager, it takes much much longer to process the files than when the mod is downloaded from within wesnoth itself.
I'm not talking about the download speed here. I'm referring to when the transfer has finished and it says Received xxx bytes, and before it lists all the files included in the mod.

I'm not sure what addon_manager is doing during that time, but my cpu goes sky high and it takes up to 5 minutes finishing doing whatever it is supposed to be doing.

Anyone knows what's going on here?
User avatar
Lord-Knightmare
Discord Moderator
Posts: 2337
Joined: May 24th, 2010, 5:26 pm
Location: Somewhere in the depths of Irdya, gathering my army to eventually destroy the known world.
Contact:

Re: very slow wesnoth_addon_manager

Post by Lord-Knightmare »

Hi, vghetto!

I actually noticed the same problem today. I just assumed that my internet was being a jerk to me as usual but since you're having the same issue, maybe this is something else?

The add-on client just disconnects after a while and refuses to download.

Also, this happened for the 1.14 add-on server (it has too many add-ons in it anyway).
Creator of "War of Legends"
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
gnombat
Posts: 671
Joined: June 10th, 2010, 8:49 pm

Re: very slow wesnoth_addon_manager

Post by gnombat »

If you're just looking for a way to download add-ons from the command line, it's probably simpler to download over HTTP(S):

Code: Select all

wget https://files.wesnoth.org/addons/1.14/Wild_Frontiers.tar.bz2
tar xvf Wild_Frontiers.tar.bz2
Of course, that will work only for downloads - if you want to upload an add-on from the command line, you will have to use wesnoth_addon_manager (but then you might run into the same issue if the add-on is large).
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

Re: very slow wesnoth_addon_manager

Post by vghetto »

Thanks gombat,
Again this isn't an internet/download issue. It's the unpacking phase of the mod that takes forever to finish.
When I download the mod from within wesnoth->add-ons I don't experience any delays.

I have all the mods from 1.12, 1.14, and 1.15 and each of those versions' wesnoth_addon_manager suffer from the same problem.

As you might be aware recently there are a some really big mods that are being updated almost daily, and they are taking up to 5 mins each to unpack. The_Demon_of_our_Ourselves, A_Song_of_Fire as examples.

It has to do with either the mod size or number of files within that mod, for example Legend_of_the_Invincibles_beta is kinda big but doesn't take as long unpack relatively to the other two.

This is the command that I use to update all add-ons

Code: Select all

wesnoth_addon_manager -V -d '.*'
User avatar
Lord-Knightmare
Discord Moderator
Posts: 2337
Joined: May 24th, 2010, 5:26 pm
Location: Somewhere in the depths of Irdya, gathering my army to eventually destroy the known world.
Contact:

Re: very slow wesnoth_addon_manager

Post by Lord-Knightmare »

As you might be aware recently there are a some really big mods that are being updated almost daily, and they are taking up to 5 mins each to unpack. The_Demon_of_our_Ourselves, A_Song_of_Fire as examples.
5 minutes? It takes like 20 minutes to unpack in my case. I eventually just gave up on them. It takes like an hour to download each so I decided that I am better off. I did install TDOOO though. Will finish it.
Creator of "War of Legends"
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
User avatar
Elvish_Hunter
Posts: 1575
Joined: September 4th, 2009, 2:39 pm
Location: Lintanir Forest...

Re: very slow wesnoth_addon_manager

Post by Elvish_Hunter »

wesnoth_addon_manager internally uses the gzip and zlib libraries supplied with Python's standard library. If in verbose mode (which I see you used) it gets stuck on GZIP compression found... or on ZLIB compression found... there isn't much I can do.
However, if the script gets stuck in another point, the situation might be different.
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
gnombat
Posts: 671
Joined: June 10th, 2010, 8:49 pm

Re: very slow wesnoth_addon_manager

Post by gnombat »

vghetto wrote: April 11th, 2021, 3:03 pm It's the unpacking phase of the mod that takes forever to finish.
When I download the mod from within wesnoth->add-ons I don't experience any delays.
I suspect that the problem is that wesnoth_addon_manager is written in Python, which is abominably slow (and probably especially so for 50 MB add-ons).

I would be inclined to use wesnoth_addon_manager to check for updates only, and use something else (like wget) to perform the actual download:

Code: Select all

#!/bin/sh

for i in `wesnoth_addon_manager --status PATH/TO/ADD-ONS | grep '^ \* ' | awk '{ print $2 }'`
do
  wget https://files.wesnoth.org/addons/1.14/$i.tar.bz2
  ...
done
User avatar
Lord-Knightmare
Discord Moderator
Posts: 2337
Joined: May 24th, 2010, 5:26 pm
Location: Somewhere in the depths of Irdya, gathering my army to eventually destroy the known world.
Contact:

Re: very slow wesnoth_addon_manager

Post by Lord-Knightmare »

suspect that the problem is that wesnoth_addon_manager is written in Python, which is abominably slow (and probably especially so for 50 MB add-ons).
The 1.15 add-on server seems to be a lot faster when updating add-ons. So, I guess it's an improvement.
Creator of "War of Legends"
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

Re: very slow wesnoth_addon_manager

Post by vghetto »

Elvish_Hunter wrote: April 11th, 2021, 3:27 pm wesnoth_addon_manager internally uses the gzip and zlib libraries supplied with Python's standard library. If in verbose mode (which I see you used) it gets stuck on GZIP compression found... or on ZLIB compression found... there isn't much I can do.
However, if the script gets stuck in another point, the situation might be different.
I don't remember seeing either one of those messages. Here's an example of the output for illustration (it wasn't slow in this example, but I'm pointing out where the hangup usually occurs)

Code: Select all

Opening socket to add-ons.wesnoth.org:15015 for 1.15.x
Connected as 42.
Receiving 298110 bytes.
Received 298110 bytes.
Not downloading pay_to_restart because it is already up-to-date.
......
Not downloading A_Vision_Blinded because it is already up-to-date.
Receiving 1246193 bytes.
The_Undead_Rising: 92366/1246193
The_Undead_Rising: 428046/1246193
The_Undead_Rising: 783618/1246193
The_Undead_Rising: 1106322/1246193
Received 1246193 bytes.
The_Undead_Rising: 1246193/1246193 <---- After this point
Unpacking The_Undead_Rising...     <---- Or this point. sorry I'm not sure
The_Undead_Rising
  COPYING.txt (18011)
  _main.cfg (2335)
  changelog.txt (22)
  images
    maps
      turmap.png (779488)
    portraits
      malin_lich-ancient.png (303992)
      malin_lich.png (129407)
    units
      mal-keshar-soul.png (3494)
      owaec-lord.png (2062)
      possessed-horse-lord.png (2062)
      icons
        icon-lich.png (1390)
  maps
    1_The_Beginning.map (5677)
    2_Parthyn.map (7703)
  scenarios
    1_The_Beginning.cfg (16132)
    2_Parthyn.cfg (11567)
  units
    Horse_Lord.cfg (1961)
    Mal_Keshar.cfg (21006)
    Mal_Keshar_Soul.cfg (4399)
    Possessed_Horse_Lord.cfg (2012)
  utils
    abilities.cfg (28018)
    bigmap.cfg (1762)
    characters.cfg (1860)
Not downloading Water_Era because it is already up-to-date.
....
@gombat Thank you that will come in handy :)
@Lord-Knightmare, yeah, I might have had a 20 minute delay a couple of times. I'll run date before and after the update next time to get a more accurate number.
User avatar
Elvish_Hunter
Posts: 1575
Joined: September 4th, 2009, 2:39 pm
Location: Lintanir Forest...

Re: very slow wesnoth_addon_manager

Post by Elvish_Hunter »

gnombat wrote: April 11th, 2021, 4:04 pm I suspect that the problem is that wesnoth_addon_manager is written in Python, which is abominably slow (and probably especially so for 50 MB add-ons).
Not quite. I just tested by downloading one of the heaviest add-ons (The_Demon_of_our_Ourselves) and it took about five minutes to unpack it. What I noticed is that it froze between these two lines:

Code: Select all

The_Demon_of_our_Ourselves: 94774941/94774941
Unpacking The_Demon_of_our_Ourselves...
Now, what code is there between this output?

Code: Select all

        while not mythread.event.isSet():
            mythread.event.wait(1)
            if pcounter != cs.counter:
                print("%s: %d/%d" % (name, cs.counter, cs.length))
                pcounter = cs.counter

        if args.raw_download:
            file(name, "w").write(mythread.data)
        else:
            decoded = cs.decode(mythread.data)
            dirname = os.path.join(cdir, name)
            oldcfg_path = os.path.join(cdir, name + ".cfg")

            # Try to remove old campaign in case it exists.
            shutil.rmtree(dirname, True)
            try: os.remove(oldcfg_path)
            except OSError: pass

            print("Unpacking %s..." % name)
The only suspicious line here is decoded = cs.decode(mythread.data), which is defined in wesnoth/campaignserver_client.py and calls the decode method, which in turn calls the decode_WML method, which in turn creates an instance of wmlparser.Parser. This one is actually wmlparser3, which relies on the main Wesnoth executable (launched as a subprocess).
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
User avatar
Pentarctagon
Project Manager
Posts: 5496
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: very slow wesnoth_addon_manager

Post by Pentarctagon »

The in-game add-ons manager in 1.15 also gained the ability to do incremental uploads and downloads, which I don't believe the wam script is able to do. So for 1.15 at least, comparing how fast they can update an add-on wouldn't be an equivalent comparison.

Though to be honest I'm also surprised that people are using the wam script for regular uploads and downloads at all instead of using the in-game client.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
gnombat
Posts: 671
Joined: June 10th, 2010, 8:49 pm

Re: very slow wesnoth_addon_manager

Post by gnombat »

Elvish_Hunter wrote: April 11th, 2021, 4:47 pm This one is actually wmlparser3, which relies on the main Wesnoth executable (launched as a subprocess).
I think that the external wesnoth executable is used only to run WML through the preprocessor; I don't think it's actually used by the wesnoth_addon_manager command.
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

Re: very slow wesnoth_addon_manager

Post by vghetto »

Pentarctagon wrote: April 11th, 2021, 4:55 pm Though to be honest I'm also surprised that people are using the wam script for regular uploads and downloads at all instead of using the in-game client.
I use the in-game client to upload my own mods. In terms of downloading, my use case is slightly different from regular players.

I download the mods for code grepping purposes rather than playing. I tend to watch replays if a mod is marginally interesting to me.
Of all the mods, what I have actually played might not amount to 1% of what is available.

The directory where I download them isn't the same as my wesnoth add-ons directory. I could do a symbolic link and then do an ingame update, but that has its own problems.

A lot of the mods especially on 1.14 are seriously broken, so loading up wesnoth with them installed would start popping up errors right and left and some of those mods break other things, especially with the campaign menu.

Using "wam" is more convenient to me because it keeps them all up to date without having to load any one of them even once. And, I don't even need to know the name of the mod, '.*' takes care of everything.When a new mod gets released, I'll get it automatically. presto :)

please don't deprecate wam :(
User avatar
Pentarctagon
Project Manager
Posts: 5496
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: very slow wesnoth_addon_manager

Post by Pentarctagon »

wam is used by some of the wesnoth.org services if I'm remembering correctly, so it's not going anywhere :)
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

Re: very slow wesnoth_addon_manager

Post by vghetto »

I tried passing --raw_download so the decode step gets skipped and I got this error.

Code: Select all

$ /home/amnesia/local/games/wesnoth/1.14/share/wesnoth/data/tools/wesnoth_addon_manager -V -R -d '.*'
Opening socket to add-ons.wesnoth.org:15014 for 1.14.x
Connected as 42.
Receiving 687648 bytes.
Received 687648 bytes.
Receiving 17510 bytes.
Received 17510 bytes.
PYR_No_Preperation_Turn: 17510/17510
Traceback (most recent call last):
  File "/home/amnesia/local/games/wesnoth/1.14/share/wesnoth/data/tools/wesnoth_addon_manager", line 279, in <module>
    get(name, title, version, type, uploads, dependencies, args.campaigns_dir)
  File "/home/amnesia/local/games/wesnoth/1.14/share/wesnoth/data/tools/wesnoth_addon_manager", line 133, in get
    file(name, "w").write(mythread.data)
NameError: name 'file' is not defined
Closing socket.
Post Reply