very slow wesnoth_addon_manager
Moderator: Forum Moderators
Forum rules
Before reporting issues in this section, you must read the following topic:
Before reporting issues in this section, you must read the following topic:
very slow wesnoth_addon_manager
Hi,
When a big mod (50+ MB) is downloaded with wesnoth_addon_manager, it takes much much longer to process the files than when the mod is downloaded from within wesnoth itself.
I'm not talking about the download speed here. I'm referring to when the transfer has finished and it says Received xxx bytes, and before it lists all the files included in the mod.
I'm not sure what addon_manager is doing during that time, but my cpu goes sky high and it takes up to 5 minutes finishing doing whatever it is supposed to be doing.
Anyone knows what's going on here?
When a big mod (50+ MB) is downloaded with wesnoth_addon_manager, it takes much much longer to process the files than when the mod is downloaded from within wesnoth itself.
I'm not talking about the download speed here. I'm referring to when the transfer has finished and it says Received xxx bytes, and before it lists all the files included in the mod.
I'm not sure what addon_manager is doing during that time, but my cpu goes sky high and it takes up to 5 minutes finishing doing whatever it is supposed to be doing.
Anyone knows what's going on here?
- Lord-Knightmare
- Discord Moderator
- Posts: 2360
- Joined: May 24th, 2010, 5:26 pm
- Location: Somewhere in the depths of Irdya, gathering my army to eventually destroy the known world.
- Contact:
Re: very slow wesnoth_addon_manager
Hi, vghetto!
I actually noticed the same problem today. I just assumed that my internet was being a jerk to me as usual but since you're having the same issue, maybe this is something else?
The add-on client just disconnects after a while and refuses to download.
Also, this happened for the 1.14 add-on server (it has too many add-ons in it anyway).
I actually noticed the same problem today. I just assumed that my internet was being a jerk to me as usual but since you're having the same issue, maybe this is something else?
The add-on client just disconnects after a while and refuses to download.
Also, this happened for the 1.14 add-on server (it has too many add-ons in it anyway).
Creator of "War of Legends"
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
Re: very slow wesnoth_addon_manager
If you're just looking for a way to download add-ons from the command line, it's probably simpler to download over HTTP(S):
Of course, that will work only for downloads - if you want to upload an add-on from the command line, you will have to use
Code: Select all
wget https://files.wesnoth.org/addons/1.14/Wild_Frontiers.tar.bz2
tar xvf Wild_Frontiers.tar.bz2
wesnoth_addon_manager
(but then you might run into the same issue if the add-on is large).Re: very slow wesnoth_addon_manager
Thanks gombat,
Again this isn't an internet/download issue. It's the unpacking phase of the mod that takes forever to finish.
When I download the mod from within wesnoth->add-ons I don't experience any delays.
I have all the mods from 1.12, 1.14, and 1.15 and each of those versions' wesnoth_addon_manager suffer from the same problem.
As you might be aware recently there are a some really big mods that are being updated almost daily, and they are taking up to 5 mins each to unpack. The_Demon_of_our_Ourselves, A_Song_of_Fire as examples.
It has to do with either the mod size or number of files within that mod, for example Legend_of_the_Invincibles_beta is kinda big but doesn't take as long unpack relatively to the other two.
This is the command that I use to update all add-ons
Again this isn't an internet/download issue. It's the unpacking phase of the mod that takes forever to finish.
When I download the mod from within wesnoth->add-ons I don't experience any delays.
I have all the mods from 1.12, 1.14, and 1.15 and each of those versions' wesnoth_addon_manager suffer from the same problem.
As you might be aware recently there are a some really big mods that are being updated almost daily, and they are taking up to 5 mins each to unpack. The_Demon_of_our_Ourselves, A_Song_of_Fire as examples.
It has to do with either the mod size or number of files within that mod, for example Legend_of_the_Invincibles_beta is kinda big but doesn't take as long unpack relatively to the other two.
This is the command that I use to update all add-ons
Code: Select all
wesnoth_addon_manager -V -d '.*'
- Lord-Knightmare
- Discord Moderator
- Posts: 2360
- Joined: May 24th, 2010, 5:26 pm
- Location: Somewhere in the depths of Irdya, gathering my army to eventually destroy the known world.
- Contact:
Re: very slow wesnoth_addon_manager
5 minutes? It takes like 20 minutes to unpack in my case. I eventually just gave up on them. It takes like an hour to download each so I decided that I am better off. I did install TDOOO though. Will finish it.As you might be aware recently there are a some really big mods that are being updated almost daily, and they are taking up to 5 mins each to unpack. The_Demon_of_our_Ourselves, A_Song_of_Fire as examples.
Creator of "War of Legends"
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
- Elvish_Hunter
- Posts: 1575
- Joined: September 4th, 2009, 2:39 pm
- Location: Lintanir Forest...
Re: very slow wesnoth_addon_manager
wesnoth_addon_manager internally uses the gzip and zlib libraries supplied with Python's standard library. If in verbose mode (which I see you used) it gets stuck on
However, if the script gets stuck in another point, the situation might be different.
GZIP compression found...
or on ZLIB compression found...
there isn't much I can do.However, if the script gets stuck in another point, the situation might be different.
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
Re: very slow wesnoth_addon_manager
I suspect that the problem is that
wesnoth_addon_manager
is written in Python, which is abominably slow (and probably especially so for 50 MB add-ons).I would be inclined to use
wesnoth_addon_manager
to check for updates only, and use something else (like wget
) to perform the actual download:
Code: Select all
#!/bin/sh
for i in `wesnoth_addon_manager --status PATH/TO/ADD-ONS | grep '^ \* ' | awk '{ print $2 }'`
do
wget https://files.wesnoth.org/addons/1.14/$i.tar.bz2
...
done
- Lord-Knightmare
- Discord Moderator
- Posts: 2360
- Joined: May 24th, 2010, 5:26 pm
- Location: Somewhere in the depths of Irdya, gathering my army to eventually destroy the known world.
- Contact:
Re: very slow wesnoth_addon_manager
The 1.15 add-on server seems to be a lot faster when updating add-ons. So, I guess it's an improvement.suspect that the problem is that wesnoth_addon_manager is written in Python, which is abominably slow (and probably especially so for 50 MB add-ons).
Creator of "War of Legends"
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
Creator of the Isle of Mists survival scenario.
Maintainer of Forward They Cried
User:Knyghtmare | My Medium
Re: very slow wesnoth_addon_manager
I don't remember seeing either one of those messages. Here's an example of the output for illustration (it wasn't slow in this example, but I'm pointing out where the hangup usually occurs)Elvish_Hunter wrote: ↑April 11th, 2021, 3:27 pm wesnoth_addon_manager internally uses the gzip and zlib libraries supplied with Python's standard library. If in verbose mode (which I see you used) it gets stuck onGZIP compression found...
or onZLIB compression found...
there isn't much I can do.
However, if the script gets stuck in another point, the situation might be different.
Code: Select all
Opening socket to add-ons.wesnoth.org:15015 for 1.15.x
Connected as 42.
Receiving 298110 bytes.
Received 298110 bytes.
Not downloading pay_to_restart because it is already up-to-date.
......
Not downloading A_Vision_Blinded because it is already up-to-date.
Receiving 1246193 bytes.
The_Undead_Rising: 92366/1246193
The_Undead_Rising: 428046/1246193
The_Undead_Rising: 783618/1246193
The_Undead_Rising: 1106322/1246193
Received 1246193 bytes.
The_Undead_Rising: 1246193/1246193 <---- After this point
Unpacking The_Undead_Rising... <---- Or this point. sorry I'm not sure
The_Undead_Rising
COPYING.txt (18011)
_main.cfg (2335)
changelog.txt (22)
images
maps
turmap.png (779488)
portraits
malin_lich-ancient.png (303992)
malin_lich.png (129407)
units
mal-keshar-soul.png (3494)
owaec-lord.png (2062)
possessed-horse-lord.png (2062)
icons
icon-lich.png (1390)
maps
1_The_Beginning.map (5677)
2_Parthyn.map (7703)
scenarios
1_The_Beginning.cfg (16132)
2_Parthyn.cfg (11567)
units
Horse_Lord.cfg (1961)
Mal_Keshar.cfg (21006)
Mal_Keshar_Soul.cfg (4399)
Possessed_Horse_Lord.cfg (2012)
utils
abilities.cfg (28018)
bigmap.cfg (1762)
characters.cfg (1860)
Not downloading Water_Era because it is already up-to-date.
....
@Lord-Knightmare, yeah, I might have had a 20 minute delay a couple of times. I'll run
date
before and after the update next time to get a more accurate number.- Elvish_Hunter
- Posts: 1575
- Joined: September 4th, 2009, 2:39 pm
- Location: Lintanir Forest...
Re: very slow wesnoth_addon_manager
Not quite. I just tested by downloading one of the heaviest add-ons (The_Demon_of_our_Ourselves) and it took about five minutes to unpack it. What I noticed is that it froze between these two lines:
Code: Select all
The_Demon_of_our_Ourselves: 94774941/94774941
Unpacking The_Demon_of_our_Ourselves...
Code: Select all
while not mythread.event.isSet():
mythread.event.wait(1)
if pcounter != cs.counter:
print("%s: %d/%d" % (name, cs.counter, cs.length))
pcounter = cs.counter
if args.raw_download:
file(name, "w").write(mythread.data)
else:
decoded = cs.decode(mythread.data)
dirname = os.path.join(cdir, name)
oldcfg_path = os.path.join(cdir, name + ".cfg")
# Try to remove old campaign in case it exists.
shutil.rmtree(dirname, True)
try: os.remove(oldcfg_path)
except OSError: pass
print("Unpacking %s..." % name)
decoded = cs.decode(mythread.data)
, which is defined in wesnoth/campaignserver_client.py
and calls the decode
method, which in turn calls the decode_WML
method, which in turn creates an instance of wmlparser.Parser
. This one is actually wmlparser3
, which relies on the main Wesnoth executable (launched as a subprocess).Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
- Pentarctagon
- Project Manager
- Posts: 5564
- Joined: March 22nd, 2009, 10:50 pm
- Location: Earth (occasionally)
Re: very slow wesnoth_addon_manager
The in-game add-ons manager in 1.15 also gained the ability to do incremental uploads and downloads, which I don't believe the wam script is able to do. So for 1.15 at least, comparing how fast they can update an add-on wouldn't be an equivalent comparison.
Though to be honest I'm also surprised that people are using the wam script for regular uploads and downloads at all instead of using the in-game client.
Though to be honest I'm also surprised that people are using the wam script for regular uploads and downloads at all instead of using the in-game client.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
take one down, patch it around
-2,147,483,648 little bugs in the code
Re: very slow wesnoth_addon_manager
I think that the externalElvish_Hunter wrote: ↑April 11th, 2021, 4:47 pm This one is actuallywmlparser3
, which relies on the main Wesnoth executable (launched as a subprocess).
wesnoth
executable is used only to run WML through the preprocessor; I don't think it's actually used by the wesnoth_addon_manager
command.Re: very slow wesnoth_addon_manager
I use the in-game client to upload my own mods. In terms of downloading, my use case is slightly different from regular players.Pentarctagon wrote: ↑April 11th, 2021, 4:55 pm Though to be honest I'm also surprised that people are using the wam script for regular uploads and downloads at all instead of using the in-game client.
I download the mods for code grepping purposes rather than playing. I tend to watch replays if a mod is marginally interesting to me.
Of all the mods, what I have actually played might not amount to 1% of what is available.
The directory where I download them isn't the same as my wesnoth add-ons directory. I could do a symbolic link and then do an ingame update, but that has its own problems.
A lot of the mods especially on 1.14 are seriously broken, so loading up wesnoth with them installed would start popping up errors right and left and some of those mods break other things, especially with the campaign menu.
Using "wam" is more convenient to me because it keeps them all up to date without having to load any one of them even once. And, I don't even need to know the name of the mod, '.*' takes care of everything.When a new mod gets released, I'll get it automatically. presto
please don't deprecate wam
- Pentarctagon
- Project Manager
- Posts: 5564
- Joined: March 22nd, 2009, 10:50 pm
- Location: Earth (occasionally)
Re: very slow wesnoth_addon_manager
wam is used by some of the wesnoth.org services if I'm remembering correctly, so it's not going anywhere
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
take one down, patch it around
-2,147,483,648 little bugs in the code
Re: very slow wesnoth_addon_manager
I tried passing --raw_download so the decode step gets skipped and I got this error.
Code: Select all
$ /home/amnesia/local/games/wesnoth/1.14/share/wesnoth/data/tools/wesnoth_addon_manager -V -R -d '.*'
Opening socket to add-ons.wesnoth.org:15014 for 1.14.x
Connected as 42.
Receiving 687648 bytes.
Received 687648 bytes.
Receiving 17510 bytes.
Received 17510 bytes.
PYR_No_Preperation_Turn: 17510/17510
Traceback (most recent call last):
File "/home/amnesia/local/games/wesnoth/1.14/share/wesnoth/data/tools/wesnoth_addon_manager", line 279, in <module>
get(name, title, version, type, uploads, dependencies, args.campaigns_dir)
File "/home/amnesia/local/games/wesnoth/1.14/share/wesnoth/data/tools/wesnoth_addon_manager", line 133, in get
file(name, "w").write(mythread.data)
NameError: name 'file' is not defined
Closing socket.