Anonymissimus' lua thread

Discussion of Lua and LuaWML support, development, and ideas.

Moderator: Forum Moderators

Post Reply
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: Anonymissimus' lua thread

Post by silene »

Anonymissimus wrote:-I see that many wml action functionalities are currently being exported from the C++ engine to the external lua files - just curious: why ? (it's not that I have an opinion about it...)
  • It's a good way to check that the Lua engine is robust.
  • It ensures that the Lua engine is not missing core functionalities.
  • It's a good time for reviewing all the WML tags and extracting its basic functionalities. For instance, there are two tags [scroll_to] and [scroll_to_unit], but they are morally the same thing. (I'm not sure I'm making much sense there. Let's just say it's a way to straighten WML semantics as a whole.)
  • It avoids the cost of converting objects forth and back between C++ and Lua.
  • It simplifies the code considerably in some cases. I'm thinking about my getting rid of wesnoth.register_wml_action for instance.
  • And probably lots of other good and bad reasons.
By the way, it's a good time to point things that are not currently possible in WML but that the engine already handles. For instance, I have on my TODO list a way for scripts to dynamically modify theme elements (e.g. status display). But there are probably other features people would like.
Anonymissimus wrote:-You didn't yet disable the math.random() function (and add a replacement to helper), or is it wired to set_variable's implementation ?
I must admit I had forgotten about this one.

Ultimately, I want clients to communicate between them. (So it covers random, but also [message]). A Gsoc student is working on persistent variables, so I'm waiting to see how it evolves, since there is some overlap about these two functionalities.

Concerning the helper function, I will add it for 1.9 if I don't get to implement the mechanism above. I think there is already a version of it on the wiki.
Anonymissimus wrote:-I suggest adding the get_locations functionality to helper which you've posted somewhen earlier in this thread
This one is on my TODO list, but as a core function. (So [store_locations] would rely on it, instead of the contrary.) Actually, earlier today on IRC, someone proposed to code it. So it may arrive sooner than I expected.
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

Could you describe the security aspect of :unsafe_lua better ?
Let's say I have some add-ons installed, one of them is from an author with criminal intention who wants to scan my pc an email my bank account data to him (would soon disappear from the server...) Doesn't the lua code that does this need to be called ? So if there are no dependencies other than to own add-ons and core files it should be save ? Since from the warning that pops up it looks like such malicious code could be executed immediately, even if the campaign isn't loaded at all.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: Anonymissimus' lua thread

Post by silene »

Anonymissimus wrote:even if the campaign isn't loaded at all.
That's the main issue. You have no way to ensure that an addon didn't load something (except by opening its code and checking for yourself). Wesnoth doesn't enforce anything more than what authors make of #ifdef. (For instance, any multiplayer game is launched with the Legend of Wesmere Lua code running. That was quite a shock the first time I looked at such a MP savegame.) For instance, there may well be a moveto event lying around that constantly tests whether the engine has switched in unsafe mode.

Anyway, I'm not happy with the unsafe mode. I intend to replace it by an implementation of remdebug directly in the engine, so that Lua doesn't need more rights. But this is no small work and it will take a bit of time for me to finish it.
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

silene wrote:(For instance, any multiplayer game is launched with the Legend of Wesmere Lua code running. That was quite a shock the first time I looked at such a MP savegame.)
That's because it's also loaded from the MULTIPLAYER preprocessor scope in LoW's _main I'm guessing.
silene wrote:For instance, there may well be a moveto event lying around that constantly tests whether the engine has switched in unsafe mode.
Can it be activated from outside that add-ons's #ifdef ? So one is forced to have a separate BfW installation for installing add-ons where :unsafe_lua is never used.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: Anonymissimus' lua thread

Post by silene »

Anonymissimus wrote:That's because it's also loaded from the MULTIPLAYER preprocessor scope in LoW's _main I'm guessing.
You are guessing right. My point was that, if the author didn't put #ifdef around its code, then it will get load, even if you are not interested in that particular addon. The engine doesn't try to enforce that there is nothing outside the #ifdef/#endif section of an addon. (Agreed, the MULTIPLAYER thing was a bad example, but hopefully it's clearer now.)
Anonymissimus wrote:So one is forced to have a separate BfW installation for installing add-ons where :unsafe_lua is never used.
Yes, that's what I was trying to convey in the warning for :unsafe_lua. Before enabling it, you have to make sure you have removed all the addons you don't trust. For instance, you could use the --config-dir option to point Wesnoth to an alternate userdata directory that would contain your own addon only.
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

I've been working on a universal debug message function, intended to ease lua programming in a wesnoth environment. Functionality (partly aimed for):
-For the primitive types nil, boolean, number and string it outputs their types and values (maybe for string also the length); that's the easy part.
-For function it would be nice to get it's code displayed (possible ?) although the lua author may already have written that function onself so the code is there.
-For any table I want an output that can be just copied into the code to assign it to the variable and "produces itself again", the code from the tutorial
Spoiler:
currently procuces this output:
Spoiler:
-For any userdata value (also if it appears not at the toplevel), it is attempted to dump it with __cfg and __literal, if one of them succeeds it's treated like a table above. If not, it calls tostring() for the userdata object (this still works for translatable strings then such as unit names).
-I want an option to limit the output if e.g. it is an array from get_units. Currently it's a number that limits both the absolute number of tables shown as well as the number of named fields per table.
code:
Spoiler:
EDIT
I know that melinath has something similar in the lua pack; that code looks easier, I think I'll continue by improving that...
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

Are the following conclusions about tables correct ? Let's stay in a wesnoth environment and there are only tables created with the intention to make them understandable by the engine.

- (When iterating over any table with pairs) Every field encountered (all fields) is either named (a string) or unnamed (can be written as [1] or [2] etc), so that tables have the structure (sort of regexp, * = variable, | = or)

Code: Select all

{(([1] = any lua type)|([string] = any lua type))*}
-These are equivalent:

Code: Select all

{[1] = { [1] = "bar", [2] = { v = 1, w = 2 } }}

Code: Select all

{{ "bar", { v = 1, w = 2 } }}
A table is not a wml table if
1. Any named field's value is not one of the types nil, boolean, string, number (scalars), or it is of type userdata which becomes a table when dumped with .__cfg or .__literal
2. Any unnamed field's value (at the same depth as 1.) is neither of type table nor userdata (try dumping, if still userdata => wml_table = false)
3. Any unnamed field's value does not consist of a pair/table of unnamed fields holding a string and a table as their values. (introduction of a subtag)
4. The subtable from 3. doesn't comply with rule 1 to 4.

I can produce syntactically correct output and I intend on adding the debug_msg function to helper.lua. Ideally, I want to check whether a table passed is a wml table, and if it isn't the output shall indicate why. This makes lua coding substantially easier.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: Anonymissimus' lua thread

Post by silene »

Anonymissimus wrote:- (When iterating over any table with pairs) Every field encountered (all fields) is either named (a string) or unnamed (can be written as [1] or [2] etc), so that tables have the structure (sort of regexp, * = variable, | = or)

Code: Select all

{(([1] = any lua type)|([string] = any lua type))*}
Yes.
Anonymissimus wrote:-These are equivalent:

Code: Select all

{[1] = { [1] = "bar", [2] = { v = 1, w = 2 } }}

Code: Select all

{{ "bar", { v = 1, w = 2 } }}
Yes. (And not just WML tables; Lua tables satisfy this property too.)
Anonymissimus wrote:A table is not a wml table if
1. Any named field's value is not one of the types nil, boolean, string, number (scalars), or it is of type userdata which becomes a table when dumped with .__cfg or .__literal
Concerning userdata, the only possible ones have the metatable of translatable strings. The previous sentence may confuse you, so hopefully the following line will make it clearer:

Code: Select all

if type(value) == "userdata" and getmetatable(value) ~= "translatable string" then error "invalid userdata (not a translatable string)" end
Anonymissimus wrote:2. Any unnamed field's value (at the same depth as 1.) is neither of type table nor userdata (try dumping, if still userdata => wml_table = false)
Again, a better test for these userdata is to check that they have a "wml object" metatable.
Anonymissimus wrote:3. Any unnamed field's value does not consist of a pair/table of unnamed fields holding a string and a table as their values. (introduction of a subtag)
4. The subtable from 3. doesn't comply with rule 1 to 4.
Yes.
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

It catches many common mistakes, but the third condition is difficult (how can one compare the structure of a table to a regexp ?) The output works well in any case.
Are wml tables including translatable strings (other userdata ?) still considered wml tables ?
Is there a way to retrieve the named fields of a table (their values and/or only the names) without knowing these names and without pairs ?
If it's neccessary to do .__cfg or .__literal for any conained value it's no longer considered a wml table however.
Spoiler:
Spoiler:
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Re: Anonymissimus' lua thread

Post by silene »

Anonymissimus wrote:It catches many common mistakes, but the third condition is difficult (how can one compare the structure of a table to a regexp ?)
You may try to check that "pairs" (or "next") doesn't return more than two elements. If there are only two and you know (because of your other checks) that [1] is a string and [2] is a WML table, you are done. The engine doesn't even try that far. It complains only when it doesn't understand what it sees. If the table wasn't a pair but still had correct [1] and [2] fields, the mistake would go unnoticed by the engine.

I haven't looked in-depth to your code, but you seem to have an issue there. "pairs" doesn't return elements in numerical order for hash tables. So, depending on the way the pair was created, you may get [2] before [1]. Your code works because the pair will happen to be a proper array most of the times. But if it isn't, it will break.

The behavior above may seem strange if you don't know how tables are stored internally, and I don't think it is explained in the official documentation. (I think there are only some cryptic comments about "array and nonarray elements".) So let me try to explain a bit more. A proper array is created when only numerical keys are added and they are added in increasing order starting from 1. It's a contiguous storage with a small memory footprint and instant access to elements. (It's the kind of arrays you find in most programming languages, but 1-based.) But if you start adding nonnumerical keys, or if you had them in a different order, Lua switches to a hash implementation. Even if there are only numerical keys in the end, the storage may still be a hash table. The keys are then iterated with respect to their hash buckets, and there is no reason for increasing keys to fall into increasing buckets.

That's not really important anyway. If you don't feel like fixing it, you can always assume that only perverted programmers will hit this corner case.
Anonymissimus wrote:Are wml tables including translatable strings (other userdata ?) still considered wml tables ?
The answer seems obvious, so perhaps I didn't understand your question. Named fields containing translatable strings do not prevent a table to be a WML table, fortunately. (No other userdata are possible as named fields.)
Anonymissimus wrote:Is there a way to retrieve the named fields of a table (their values and/or only the names) without knowing these names and without pairs ?
I only know of "pairs" and "next" for this purpose.
Anonymissimus wrote:If it's neccessary to do .__cfg or .__literal for any conained value it's no longer considered a wml table however.
For __cfg, that's right. For __literal, not so. As their name implies, "wml object" userdata do represent WML objects, yet they provide a "__literal" field. (And they are the only userdata currently providing this field.) So the corresponding line of "dump_user_data" is not correct.
User avatar
Dixie
Posts: 1757
Joined: February 10th, 2010, 1:06 am
Location: $x1,$y1

Re: Anonymissimus' lua thread

Post by Dixie »

silene wrote:
Anonymissimus wrote:Is there a way to retrieve the named fields of a table (their values and/or only the names) without knowing these names and without pairs ?
I only know of "pairs" and "next" for this purpose.
Well, I guess you could rewrite the next function or something. I stumbled upon that yesterday while searching infos and examples for the for iterator and pairs function. It was a bit advanced for me so I didn't get all of it, but maybe it can be of some use for you :)

That Lua-users wiki is a great mine of infos btw! Maybe it should be linked in our LuaWML wiki along with the Lua user manual?
Jazz is not dead, it just smells funny - Frank Zappa
Current projects: Internet meme Era, The Settlers of Wesnoth
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

Spoiler:
I think I'll rather leave it this way now. Error messages ar no longer that verbose but there seem to be no false positives, only some false negatives which don't hurt. (for tables introducing subtables, the {"tag", {}} structure is enforced)
Could the function that's called by :clear be made available to lua ? I don't suppose it would be problematic...
Btw the sript should work in 1.8 too, the last line with wesnoth.wml_actions must be wesnoth.fire(... then.

EDIT
Code updated, there are corner cases left that I fix one after another...
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

why is this parameter a table instead of userdata ?

Post by Anonymissimus »

I hope I've included everything that's relevant.
Spoiler:
This outputs "table" for the parameter type, although the function is called by wml directly.

EDIT
For anyone interested in the answer and related discussion, read the irc log #wesnoth-dev from about date and time the posting was made.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

I'd like to point out that lua can be used to perform wmllint-wmlscope-ish operations on wml or lua files. The script below can read macro definitions from a file and replaces all calls to these macros in another file while inserting arguments correctly. It's meant to be run independently from the wesnoth engine. It's far from being perfect and I have a hard time to get all possible cases working... :P The biggest problem is that "" and () can make macro arguments span over spaces. # characters anywhere in the macro definition or call break this currently and indentation gets completely messed up, but emacs can reindent whole files so it's not so important.
Spoiler:
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: Anonymissimus' lua thread

Post by Anonymissimus »

a question for silene:
For C++ I've been told that in these code snippets it's better to use the first variant due to memory allocation (for the variable j, it needs some time to reserve the memory every time it is declared):
lua:

Code: Select all

		local j
		for i = 2, 5 do
			j = i
		end

		for i = 2, 5 do
			local j = i
		end
C++:

Code: Select all

	int j;
	for(int i = 2; i <= 5; ++i) {
		j = i;
	}

	for(int i = 2; i <= 5; ++i) {
		int j = i;
	}
(And let's say that we need "j" only within each step of that loop, so we declare it outside only because of efficiency.)
That is, declare variables outside of loops, only assign them within, and it's what I tend to do in all languages, but the core lua codings don't hesitate to declare variables locally within loops.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
Post Reply