Why not LUA?

[Viliam]

I think we could get some ideas from XSLT language. It is a markup language which allows writing algorithms.

To say it shortly, XSLT is like two languages in one. It has a system for writing tags and attributes. And it has a system for writing expressions, which can appear in some attributes. (Actually the expression language is a separate language called XPath.) The tag language guarantees the full conformity to XML format; the expression language allows writing easy expression calls, for example mathematical operations.

So this is how I imagine adapting the system into Wesnoth.


Expression language

Expression is something written in WML attribute. It can be evaluated, and it returns an "R-value" (a value that cannot be modified). It understands some variables and functions (defined externally). Variables can contain any types: scalar or collection. It supports some operators.

The role of expression language is to parse the expression and understand: this is variable, this is constant, this is operator, this is function call. The following expressions would be legal:

Code: Select all

example1 = $x + $y + 3
example2 = "Hello World"
example3 = _"Hello World"
example4 = f($x, $y)

The possible variables and functions are defined by context. The constants and operators are hard-coded in language.

To make the language more regular, it may be necessary to impose stricter rules on writing WML atributes. For example it may become necessary to always write string constants in "" marks -- this will become obvious during implementation.

I think it has to be in 'certain places'. in your example, $unit is only going to be valid in certain contexts.

The variable $unit is defined only in some contexts. But the concept "if it has $ at the beginning, it is a variable" is valid in any context. Some contexts only provide additional variables with specific meaning.


Tag language

This is the WML as we know it, but for any attribute it is known whether it requires a constant value, or an expression. (Or maybe any atribute can be an expression? I am not sure about this.) Some tags may create specific local variables, such as [event] tag. There can be some new tags to define functions for expression language -- to remove some situations where macros are used.

Some example tags:

Code: Select all

[set_variable]
  name = "x"
  value = $x + 5
[/set_variable]

[if]
  test = $x + 1 > 1000
  ...some action...
[/if]

[switch]
  [when]
    test = $x = 1
    ...some action...
  [/when]
  [when]
    test = $x = 2
    ...some action...
  [/when]
  [otherwise]
    ...some action...
  [/otherwise]
[/switch]

[for_each]
  iterate = get_units_for_player(1)
  ...some action...
[/for_each]

I know some of these things are already implemented. I just feel there should be some system it in; something clearly explaining what can be used where (preferable everything everywhere, if it makes sense). The more regular system could be easier to implement (it may require refactoring a lot of existing code, but later will be easier to maintain), and easier to use.

Oh, and maybe a bit of syntactic sugar, to make long expressions shorter. It would be nice to allow writing attribute on the same line as is the opening tag, such as:

Code: Select all

[set_variable] name = "x"
  value = $x + 5
[/set_variable]

[if] test = $x + 1 > 1000
  ...some action...
[/if]

Or maybe...

Code: Select all

[set_variable/] name,value = "x", $x + 5

[WK1]

Here are the advantages and disadvantages of adding a lua implementation for maps/scenarios:

Advantages of Lua:
1) Lua is a well-known language. It is documented elsewhere. Map authors won't have to learn some obscure language that only works in one place. Help to stop language/syntax proliferation! Map authors would only have to learn the Wesnoth API (and basic Lua if they don't already know it).

2) Lua is powerful. It is turing complete. You don't have to think up in advance everything a map author might want to do. This doesn't mean you will never expose more APIs to the lua maps, or refine them, but map authors are not at the developers' mercy when they have a unique idea for a map.

3) Lua is simple. It is a mature language, designed from the ground up to be simple. WML is not-so-simple. (opinions vary on which is simpler, of course)

Advantages of WML, and disadvantages of Lua:
1) I'm surprised no-one has yet mentioned the biggest drawback to using Lua. If you allow authors to execute arbitrary code in a language like Lua or Python, then any player who runs a map is exposing his computer to the map author. Lua is not "secure"; I think WML is (or at least can be).

2) Plenty of maps are already in WML. The WML implementation is already present. AKA inertia. Also, there is something to be said of only having one map language.

Also, did somebody say that we can write maps in Python? Python is just as popular as Lua, and has all of its advantages. There isn't much reason to write a Lua implementation if there is already a Python one. Regardless if there is a Python map API, if we make a scripting map implementation, we should use Python instead of Lua, if only because the AI has a python interface. It is silly to use one scripting language in one place, and then another elsewhere in the same project.

[Viliam]

I'm surprised no-one has yet mentioned the biggest drawback to using Lua. If you allow authors to execute arbitrary code in a language like Lua or Python, then any player who runs a map is exposing his computer to the map author.

I think Lua can only use those functions which are provided to it by C++ code. If all functions can only manipulate game data, there is no chance for Lua script to -- for example -- format your hard disk. Because it does not have "format" function, unless you provide one. So if you provide only harmless functions for manipulating data, nothing wrong should happen. Except the risk of running in infinite loop, or allocating too much memory. And if there are bugs in the implementations of those functions, these bugs could be abused.

[Sapient]

WK1, your exhaustive list seems biased and incomplete. You haven't even included the items previously mentioned in the thread here. So while it may accurately reflect your opinion, I'm not going to be swayed such an analysis.

[WK1]

WK1, your exhaustive list seems biased and incomplete.

Wow. That's pretty harsh, Sapient. Have anything to add to the list? Please share.

You haven't even included the items previously mentioned in the thread here.

All I could find in this thread was the following:
* Lua is better
* Lua is simpler
* WML is smipler
* Lua is more powerful
* We don't need or necessarily want more powerful
* Let's do it some other way instead.

I feel I summed these up in my list quite well. And added a few more.

As for being biased, that is true. I think someone should always think critically before inventing a new language or file format. 99% of the time, one of the popular, already existing ones will work just fine. It seems to me that WML never filled any unfilled niche.

Perhaps you, or someone else who is biased the other way, can add to the list of advantages of WML?

Viliam, I'll look into that. You might be right. I was reading in another thread about how the Wesnoth Python API for AI is restricted, and harmless. Anything is possible.

[Viliam]

Viliam, I'll look into that. You might be right. I was reading in another thread about how the Wesnoth Python API for AI is restricted, and harmless. Anything is possible.

I have no experience with Lua or Python coding, but it seems to me that Python comes with some "standard library" which contains potentially abusable code (such as creating files, running OS commands, etc). And the restricted mode is an attempt to disable all potentially harmful things, and hope you did not forget any.

In Lua there are also some provided libraries, but they are completely optional; and some of them are doing only harmless things such as mathematical operations. The core of Lua is just a parser and interpreter in style: "this is variable, this is operator, this is function call... do I have any function registered with that name?" And from the other C++ code you only say "there will be function F(x), and when anyone calls it, you call my C++ function F_IMPLEMENTATION".

[allefant]

Also, did somebody say that we can write maps in Python? Python is just as popular as Lua, and has all of its advantages. There isn't much reason to write a Lua implementation if there is already a Python one. Regardless if there is a Python map API, if we make a scripting map implementation, we should use Python instead of Lua, if only because the AI has a python interface. It is silly to use one scripting language in one place, and then another elsewhere in the same project.

The Python API is only for AI control currently. And the next step to actually provide more, like the [script] idea Sapient mentioned, would not be so much writing/extending a Python or Lua wrapper. Instead it would be the design and implementation of the API to expose. What classes do you want exposed (like Map, Terrain, Unit, UnitType, Side, ....), and what methods should they have. Then look at the C++ code (which often is intrinsically tied to WML by its use of config objects) and try to implement this API, in C++, where it does not exists yet. And in most cases it will not. Only then wrap the API 1:1 to the scripting language.

That last step would be trivial, e.g. in Python it would be fully or mostly automated by using one of Boost-Python, SWIG, ctypes or pyrex. And likely equally easy for Lua or any other general scripting language.

The Python-AI instead was done the hard way, implementing lots of glue code as no C++ API for AI control exists, and then on top of that (or because of that) using the Python-C API directly, which among other things has you manually deal with reference counting. It still turned out nice, as the bruteforce.py AI shows, but I think it was done the wrong way. Not too much thought required initially, but then a lot of work to fix details later (when I had taken over maintenance ), and now difficult to extend.

[IndieTrannie]

For what it's worth I would certainly appreciate an opening for scripting. I'm learning Ruby right now and could probably pick up Python/LUA easy enough, I've done a lisp-like and LOLCODE before also. Sadly I'm not enough of a C++ guru to really move this forward myself.

[CIB]

Instead it would be the design and implementation of the API to expose.

The process of exposing an API has already been in progress for a long time, specifically since ActionWML was invented. You can now already modify very much via ActionWML, so we only need to make ActionWML more powerful(functions, tag inserts, heck, even eval would be possible)

[mihoshi]

Could someone please explain all the merits of Lua over WML?

I'd like to see a line item for each improvement that you feel Lua offers over WML, rather than just a blanket "it's better" statement.

1. No need to extend and maintain WML - less work for programmers.

2. No need to learn WML. WML is much harder to learn than Lua. And Lua is also useful outside Wesnoth.

3. More natural, readable and compact syntax. About two times at least, much more for conditional-heavy scenarios. LUA was build as a language for describing data structures, it's hard to compete with it on it's field.

4. Lots of free features you won't have to implement yourself. Like math, fancy data structures, etc.

Also, please explain how non-coders will be expected to use it when creating their user made campaigns.

Pretend that Lua is not a programming language. Just like WML, Lua can be used as just a markup language, but with better syntax.

For example, there is a claim that the syntax would be cleaner.

In LUA, I presume all strings would need to be surrounded by quotes and all attributes would be set by function calls; so, instead of writing key=value, would you write set_key("value")
And for translateable strings, it would be set_key(_("value"))
That doesn't seem very advantageous to me.

In Lua that will be key=value too. For translateable that will be likely key=t("value"), but in most cases there wpn't be need to specify for each distinct line that it must be translated.

One nice thing would be the ability to do mathematical manipulation inside expressions. There are ways this could be added to WML, too; probably the only reason it hasn't been added yet is because how often do you need to do advanced arithmetic for ScenarioWML ? WML features are added as they are needed. To program a Wesnoth scenario, you probably don't need access to sinusoidal, exponential, or logarithmic calculations. If you did, it could be added.

More to code, more to maintain, more to document, more to learn.

Another problem is that some of the limitations of WML are intentional. The C++ code could become very unstable if WML were allowed to change any variable at any time.

Can be done in LUA too.

All in all, I am not convinced. Why would a general purpose scripting language be better for writing Wesnoth scenarios than WML, a language designed specifically for writing Wesnoth scenarios? You have to back up your claim.

Because that's less work for programmers and campaign designers and more interesting campaigns for rest of us. And it's not hard to do, I think...

WML repeats Lua history, in fact. Read about it - it's very interesting, actually http://www.lua.org/history.html

[Dave]

2. No need to learn WML. WML is much harder to learn than Lua. And also useful outside Wesnoth.

I doubt most of our scenario designers would agree with this.

[Sapient]

mihoshi, first of all, thanks for a serious and thoughtful answer. I'm going to analyze your points one by one now.

1. No need to extend and maintain WML - less work for programmers.

Will you really get rid of WML entirely by implementing Lua? Even if you do, there is still a need to "maintain" and "extend"... only now you will "maintaining" with finding bugs in Lua scripts (instead of finding bugs in old WML), and you will be "extending" with new hook-ins for Lua (instead of new tags for WML). The only thing you gain here is a class of extensions that are not Wesnoth specific; these would be generic data manipulation and data management routines. That's it.... no giant work saver, sorry. Next point.

2. No need to learn WML. WML is much harder to learn than Lua. And Lua is also useful outside Wesnoth.

A) "No need to learn WML." Ok, again we are assuming that WML can be done away with entirely. Now balance that with "No need to learn LUA" which is the current state of affairs. We can say this is not a net gain, until you prove that learning LUA is faster, easier, and more useful.

B) "WML is much harder to learn than Lua." I disagree. If you already know a programming language that is similar to LUA, then of course it would be easier to learn LUA. However, if you come from a background of zero coding experience whatsoever, I'd argue that LUA presents a steeper learning curve and is more likely to result in new user syntax errors.

C) "Lua is useful outside Wesnoth." This is true; once you strip it away from the rest of the sentence you do have a valid point.

3. More natural, readable and compact syntax. About two times at least, much more for conditional-heavy scenarios. LUA was build as a language for describing data structures, it's hard to compete with it on it's field.

A) "more natural"-- ok, neither LUA nor WML are remotely "natural" when compared with the spoken english language, but if I had to pick one of the two as the winner in that category, I'd probably say WML by slim margin.

B) "readable and compact syntax"-- this is true when we get to complex code; it's probably the biggest thing in LUA's favor. But, as we've seen... for very simple things, such as translateable strings WML can actually win this category.

4. Lots of free features you won't have to implement yourself. Like math, fancy data structures, etc.

We already covered that in point #1 "extensions"
(and performing advanced mathematics for UMC is really an edge case scenario).

Pretend that Lua is not a programming language. Just like WML, Lua can be used as just a markup language, but with better syntax.

This I did not know... can we see some examples of LUA being used as a highly nested markup language?

In Lua that will be key=value too. For translateable that will be likely key=t("value"), but in most cases there wpn't be need to specify for each distinct line that it must be translated.

Maybe you don't know the way translateable strings work using gettext, but I'll just cut it short and say that this statement is entirely false.

More to code, more to maintain, more to document, more to learn.

And maybe we like coding some features?
It's not like we'd be re-inventing Pythagora's Theorum ... game_events.cpp can call upon any C++ libraries that are deemed necessary.

[mihoshi]

Will you really get rid of WML entirely by implementing Lua?

Syntax-wise - yep. I think nearly everything in "serialization" folder will be not needed anymore.
And config.cpp will likely be a front to a Lua data structure.

B) "WML is much harder to learn than Lua." I disagree. If you already know a programming language that is similar to LUA, then of course it would be easier to learn LUA. However, if you come from a background of zero coding experience whatsoever, I'd argue that LUA presents a steeper learning curve and is more likely to result in new user syntax errors.

Lua has a good track as a programming language for non-programmers. It's very simple, simpler than any other full-featured language I have seen. And simpler than WML too, because it uses less ad hoc hacks.

A) "more natural"-- ok, neither LUA nor WML are remotely "natural" when compared with the spoken english language, but if I had to pick one of the two as the winner in that category, I'd probably say WML by slim margin.

"While" and "if" in WML are much harder to work with. Even for a programmer. In most other cases they are about on par.

This I did not know... can we see some examples of LUA being used as a highly nested markup language?

http://code.google.com/p/luadialogexamp ... BottlesDlg
There are also scripts that work with that structure here http://code.google.com/p/luadialogexamp ... /FrontPage, but they are with comments in Russian.

In Lua that will be key=value too. For translateable that will be likely key=t("value"), but in most cases there wpn't be need to specify for each distinct line that it must be translated.

Maybe you don't know the way translateable strings work using gettext, but I'll just cut it short and say that this statement is entirely false.

There are more than one way to deal with translateable strings. We can assume that specific tags, like unit.user_description, story.part.story, message.message etc. must be translated by default - that should cover 99% cases.

More to code, more to maintain, more to document, more to learn.

And maybe we like coding some features?
It's not like we'd be re-inventing Pythagora's Theorum ... game_events.cpp can call upon any C++ libraries that are deemed necessary.

Well, that's only reason to use WML, I think. I know that programming a programming language is more fun than binding one - I was doing both for living last three years.

[appleide]

All we really need is a quick way to set and test variables right? We don't want to implement LUA scripts in every child tag of [event]. Here is a quick and dirty solution I thought up, there might be flaws... but what do you think?

Code: Select all

#! A new type of variable
[set_function_variable]
name=myVar
#! Here is where the scripting happens.
value="var('x')*12;"
[/set_function_variable]

#! Now, anywhere where variables are , you can use "$myVar" instead. Each time $myVar is referenced, it would be re-calculated.

#! A new type of condition; This isn't as useful, though.
[set_function_condition]
name=myCond
value="var('x').greater_than(12)"
[/set_function_condition]

#!Example; Each time $myCond is referenced, the condition will be recalculated.

[if]
condition=$myCond
[then]
#! do something
[/then]
[/if]

By doing this, instead of implementing the entire concept of "functions", we would still be able to solve a problem of WML; Its verbosity when doing simple arithmetic or testing conditions. The 'down-side' is that we are doing this in a traditional WML way; extending it tag by tag.


EDIT: We could also implement this in a way that doesn't require the use of scripts. However, its advantages in reducing verbosity would be lessened; It'd then only save time (and words) if you do similar arithmetic or test similar conditions repeatedly.

[mihoshi]

Code: Select all

#! A new type of variable
[set_function_variable]
name=myVar
#! Here is where the scripting happens.
value="var('x')*12;"
[/set_function_variable]

#! Now, anywhere where variables are , you can use "$myVar" instead. Each time $myVar is referenced, it would be re-calculated.

#! A new type of condition; This isn't as useful, though.
[set_function_condition]
name=myCond
value="var('x').greater_than(12)"
[/set_function_condition]

#!Example; Each time $myCond is referenced, the condition will be recalculated.

[if]
condition=$myCond
[then]
#! do something
[/then]
[/if]

How is that better than

Code: Select all

function myVar(x) 
  return x*12 
end

function myCond(x) 
  return x>12 
end

if myCond(x) then 
  ... 
end

?