wmlscope is sooooo slow

[Atreides]

OK, cheesy title aside has anyone noticed that wmlscope is much slower since 1.14?

My 1.12 wmlscope run from the GUI (includes core as well as whatever you choose) generally takes about 30 seconds with no extras picked.

1.14/16 it takes 10 times as long! 5 minutes.

Is it just me? I have to confess that it might be, just possibly. 1.12 is a debian installed package while 1.14+ is all flatpak (since 1.12.6 is the very newest as far as debian 9 (yes I know there's 10, I'll risk upgrading when I'm forced to) is concerned hehe) but surely the directoryies they are in is the only real difference for the core files... I really doubt that could be it unless wmlscope has some kind of hang up with the length of the pathname (the flatpaks tend to be in a longer path)...

Anyways, just airing my discontent and curious if anyone else has can relate...

[Elvish_Hunter]

Moved into the Coder's Corner.
There are a lot of possible reasons for this slowness. I must admit that I haven't been running the 1.12 version for a long time, so I don't have a reference about this.
First of all, one of the reasons is that we have more content in mainline: more macros, more images and more units mean more things that wmlscope needs to keep track of, which makes it slower. There isn't much we can do, except cutting content from mainline - but nobody wants to do that, right?
The second reason is directly depending on the porting to Python 3: one of the things that went lost was Python's cmp() function, and to have a certain part of the code correctly sorting items, I had to use the total_ordering module, which is slower than writing all the six comparison functions by hand (although it didn't seem really slower to me).
The third reason depends on the square braces notation that we use now to define animations: when things were simpler, each image path could point only to a specific file. This isn't the case any more: a path can now point to several sprites and I had to actually write in Python the square braces expansion code, because the checks for missing files had become useless.
And then I also had to add other features, like support for optional macro arguments (back then we didn't have them!), detection of deprecated macros (which I'm thinking to use to output a warning when a call to them is found)...
So, you get the idea. wmlscope back then was faster because it didn't had to handle several features that are part of current WML and also because it wasn't very much maintained. Can it be made faster? Probably yes, but it isn't easy to do so.

[gnombat]

debian 9 (yes I know there's 10, I'll risk upgrading when I'm forced to)

There's also Debian 11 now too.

I think just upgrading to Debian 10 or 11 would probably improve performance. Debian 9 has a version of Python 3 that is several years old (Python 3.5) and, as a general rule, early versions of Python 3 were incredibly slow. This has improved considerably in recent releases.

Here is a test I ran on a machine with several different versions of Python 3 installed (this is a Windows machine so the results might be considerably different on Linux, but I think it gives an idea how much Python 3 performance has improved):

Code: Select all

C:\Users\U\AppData\Local\Programs\Python\Python35\python.exe --version
Python 3.5.4

C:\Users\U\AppData\Local\Programs\Python\Python35\python.exe data/tools/wmlscope data/core
...
#Cross-reference time: 97 seconds

C:\Users\U\AppData\Local\Programs\Python\Python37\python.exe --version
Python 3.7.9

C:\Users\U\AppData\Local\Programs\Python\Python37\python.exe data/tools/wmlscope data/core
...
#Cross-reference time: 74 seconds

C:\Users\U\AppData\Local\Programs\Python\Python39\python.exe --version
Python 3.9.10

C:\Users\U\AppData\Local\Programs\Python\Python39\python.exe data/tools/wmlscope data/core
...
#Cross-reference time: 67 seconds

[Atreides]

Thanks for the replies. Interesting stuff! I suspect that the rewrite for 1.14 is the likely culprit. Amazing to hear how much had to be changed. I'd like to move to Debian 10 (and will be "forced" to when they stop updating 9 this summer (I think)) but it sounded very daunting when I read about it.

One thing I do that helps a lot is to run wmlscope from the command line just on the addon dir (not including the core, that I also avoid with wmllint as it too is super slow when the core gets checked).

[max_torch]

There's a thing in Python where you make use of multiprocessing/multithreading. wmlunits uses this that's why it's fast. But wmlunits will not run on windows,
not sure if it might be tricky to make multiprocessing/multithreading be cross-platform?
Also, it can be tricky to write this code to run on all devices, for example you would need to detect the number of cores/threads on a computer so that the script knows how many it can use.

[Atreides]

Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds. Not a very accurate comparison, nor sure I used same parameters or add-on but the difference is large enough to make that moot. Man... feels good to be back at the command line.

[max_torch]

Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds. Not a very accurate comparison, nor sure I used same parameters or add-on but the difference is large enough to make that moot. Man... feels good to be back at the command line.

Check if the passed arguments to wmlscope when you ran it in the command line are the same as the arguments that were passed to wmlscope when you ran it in the GUI version.

[Atreides]

Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds. Not a very accurate comparison, nor sure I used same parameters or add-on but the difference is large enough to make that moot. Man... feels good to be back at the command line.

Check if the passed arguments to wmlscope when you ran it in the command line are the same as the arguments that were passed to wmlscope when you ran it in the GUI version.

Mmm yes I'd thought of that, but didn't think about it in this context, obviously they could account for a difference. Thanks for pointing it out.

[Elvish_Hunter]

There's a thing in Python where you make use of multiprocessing/multithreading.

However, multiprocessing can't be really used in this case because wmlscope relies on the files being processed in a certain order (and so does wmllint). For multithreading the situation is even worse because of the GIL (global interpreter lock), which makes sure that all the Python code runs only one thread at any given time; since our Python scripts already seem to take one whole core for their computations, there wouldn't be any benefit.

Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds.

It's surprising and yet it isn't at the same time. I mean, if you think about it, when using Tkinter Python needs to call an interpreter of another language, which is Tcl/Tk. If you need to run it a lot of times, using the command line and excluding all the GUI stuff is a good idea.
BTW, since commits 21bd75c (master) and 0da1ea4 (1.16) wmlscope is capable of reporting usages of deprecated macros, for example like {SPECIAL_NOTES} and {FOREACH} {NEXT}.