wmllint and binary path

[Groggy_Dice]

It has always struck me as odd, when porting, that I have to manually change the binary path from data/campaigns to data/add-ons. This seems like just the sort of little detail that wmllint is meant to handle. Well, it turns out that there is code in wmllint that is meant to change the paths of the binary path and textdomain (right before the code that changes "{@campaigns/" to "{~add-ons/"), but it isn't working:

Code: Select all

  1601	            if in_binary_path or in_textdomain:
  1602	                lines[i] = re.sub(r"data/campaigns", r"data/add_ons", lines[i])

However, the actual problem may be in the previous lines; I don't know Python. Now, it's just as well that this line doesn't execute, since it should be add-ons, not add_ons. I'm sure that whoever figures out the bug will take care of that, too.

(Actually, it strikes me as odd that we have to specify a binary path at all, at least for the add-on itself. It seems to me that the game should automatically search an add-on's folder for binaries when the add-on is loaded.)

I'll also mention a couple of corner cases I've encountered (each affects only one campaign that I know of) that wmllint chokes on. Although I already reported them earlier, in a post called "wmllint barfs", I can now be a bit more specific about what causes them.

The first of these problems can be found in "Invasion of Arendia" (1.6 port). The last two scenarios are bare-bones placeholders for scenarios that haven't been written yet, but wmllint has a problem with this little line:

Code: Select all

    map_data=""

Those simple quotation marks force wmllint to exit. Delete both quotation marks, or type something random between them, and wmllint will leave the files untouched, seeing nothing to convert. Delete just one quotation mark, and wmllint goes back to barfing, but if you then add something after it, it will interpret the rest of the file as a map, and insert commas everywhere.

Code: Select all

wmllint: internal error on data/add-ons/Invasion_of_Arendia/scenarios/16-And_through_stone.cfg
Traceback (most recent call last):
  File "data/tools/wmllint", line 2373, in <module>
    changed = translator(fn, [maptransform], texttransform)
  File "data/tools/wmllint", line 1849, in translator
    outmap = [outmap[0]] + outmap + [outmap[-1]]
IndexError: list index out of range

wmllint: internal error on add-ons/Invasion_of_Arendia/scenarios/17-Trapped.cfg
## [identical to above]

## And as reported by 1.8:

wmllint: internal error on add-ons/Invasion_of_Arendia/scenarios/16-And_through_stone.cfg
Traceback (most recent call last):
  File "PATH/Battle for Wesnoth 1.8.4/data/tools/wmllint", line 2193, in <module>
    changed = translator(fn, [], texttransform)
  File "PATH/Battle for Wesnoth 1.8.4/data/tools/wmllint", line 1689, in translator
    outmap = [outmap[0]] + outmap + [outmap[-1]]
IndexError: list index out of range

The other error is with the first scenario of "August Kingdoms: Prologue" (1.6), and it is nasty. Earlier I'd tracked it down to this comment, but now I know that it is specifically the combination of two left tags (<<).

Code: Select all

	#>>>> !!!!! REMOVE THIS AFTER TEST !!!!! <<<<#
	#[endlevel]
	#	result=continue
	#[/endlevel]
	#>>>> !!!!! REMOVE THIS AFTER TEST !!!!! <<<<#

This bug doesn't just cause an exit, it lags the computer, sucking up CPU and memory until it finally gives up.

Apparently, it's in a part of wmliterator that deals with Lua strings.

Code: Select all

wmllint: internal error on add-ons/Autumn_Kingdoms_Prologue/scenarios/01_Winter.cfg
Traceback (most recent call last):
  File "data/tools/wmllint", line 2373, in <module>
    changed = translator(fn, [maptransform], texttransform)
  File "data/tools/wmllint", line 1944, in translator
    for nav in WmllintIterator(newdata, filename):
  File "PATH/data/tools/wesnoth/wmliterator.py", line 403, in next
    self.element, nextScopes = self.parseElements(self.text)
  File "PATH/data/tools/wesnoth/wmliterator.py", line 243, in parseElements
    text = text[:beginquote] + text[endquote+2:]
MemoryError

## 1.8 version:

wmllint: internal error on add-ons/Autumn_Kingdoms_Prologue/scenarios/01_Winter.cfg
Traceback (most recent call last):
File "PATH/Battle for Wesnoth 1.8.4/data/tools/wmllint", line 2193, in <module>
changed = translator(fn, [], texttransform)
File "PATH/Battle for Wesnoth 1.8.4/data/tools/wmllint", line 1781, in translator
for nav in WmllintIterator(newdata, filename):
File "PATH/Battle for Wesnoth 1.8.4/data/tools/wesnoth/wmliterator.py", line 391, in next
self.element, nextScopes = self.parseElements(self.text)
File "PATH/Battle for Wesnoth 1.8.4/data/tools/wesnoth/wmliterator.py", line 239, in parseElements
text = text[:beginquote] + text[endquote+2:]
MemoryError

I will understand if developers regard both of these bugs to be rare enough to ignore, though.

[Anonymissimus]

Note that wmllint's maintainer esr doesn't read the forums.

As for the folder name issue, I am guessing that it is disabled because it is harmful for mainline. At least I read somewhere (probably in wmllint or in one of esr's commit messages) a comment about that. Sounds logical, considering that mainline campaigns still live in a data/campaigns/ folder.

As for automated binary data loading; in my addon at least I have also an additional custom binary path which points to a mainline campaign. Also, the folder names are entirely arbitrary so the engine would probably not know where to search.

As for wmllint crashes on "unusual wml", that's what you have to accept if you want to use it. Forces you to use certain style, yeah. It's difficult enough to still get it to work that well.
For map_data="", I think there is a macro "NO_MAP_DATA" or so in mainline that may work.
For <<<<, well, just don't put such a line. You don't need it, do you ? (The engine itsself also has problems with nested << btw).

[Groggy_Dice]

Note that wmllint's maintainer esr doesn't read the forums.

Well, I remember Shadowmaster's blog post about developers and the forums. "hint: it’s not Noy, it’s not Dave, it’s not me" - leaving who?

As for the folder name issue, I am guessing that it is disabled because it is harmful for mainline. At least I read somewhere (probably in wmllint or in one of esr's commit messages) a comment about that. Sounds logical, considering that mainline campaigns still live in a data/campaigns/ folder.

Having recently become acquainted with the potential of sed's -f option, I've figured out my own post-wmllint path-cleaner that skips mainline directories.

Code: Select all

for FILES in $(find [UMCDIR] -name "*.cfg" -exec grep -l "data/campaigns" {} \;) ; do ls [WESNOTH/]data/campaigns | sed 's;.*;\\@&@s/^/EMMELL/;' | sed \$\ 'a\/EMMELL/! s;data/campaigns;data/add-ons;'|sed \$\ 'a\s/EMMELL//g' | sed -f - -i.bak $FILES ; done

(Of course, [UMCDIR] and [WESNOTH/] would be replaced by their paths, and EMMELL could be any arbitrary string odd enough that it's not going to be in use already.)

Now, wmllint doesn't know where the mainline campaigns are, from what I understand. But for usage types, wmllint doesn't scan for them, it has a list of core usage types defined. Following that example, I assume that a list of mainline campaigns could be defined:

mainline = ("An_Orcish_Incursion", blah blah)

Then wmllint could check the path for "data/campaigns", check again for "not in mainline", and then convert those that meet both conditions. This would make it safe for mainline campaigns, and any add-on that references them.

As for automated binary data loading; in my addon at least I have also an additional custom binary path which points to a mainline campaign. Also, the folder names are entirely arbitrary so the engine would probably not know where to search.

Yes, I realize that if a UMC has an external dependency, it will have to reference it somehow. But Wesnoth has to have an idea of where the add-on's folder is in order to read its _main.cfg to begin with. It can't be that the engine discards that information after reading _main.cfg into cache, because binary path only tells it where to look for binaries, not scenarios, units, etc. True, most add-ons use the {~add-ons} addresses, but a few (eg, A New Heir, Anya of Willshire) use relative paths (eg, {./scenarios}).

As for wmllint crashes on "unusual wml", that's what you have to accept if you want to use it. Forces you to use certain style, yeah. It's difficult enough to still get it to work that well.

Yes, I understand that some issues are probably going to be too "unusual" to be worth fixing. And I know some people have converted add-ons manually without wmllint, but I consider them masochists.

For map_data="", I think there is a macro "NO_MAP_DATA" or so in mainline that may work.

Well, I'm only concerned about people being caught unawares by the problem. If they already know of it beforehand, it would be just as easy to type something between the quotes as to call up a macro.

For <<<<, well, just don't put such a line. You don't need it, do you ? (The engine itsself also has problems with nested << btw).

No, it's not my comment, and I have no plans to port that campaign (or anything else before I upgrade my earlier ports from 1.8 to 1.10). Though I was impressed by the quality of the prose, and regret that the author seems to have dropped out of the UMC community. Again, it's another problem that's easy to avoid if you already know about it, and an easy 's/<</< </g' to fix.