WesScriptLayer

[fendrin]

Please note, this first post is still work in progress.
I will inform in the thread about updates.


I have been developing recently an alternative scripting language which builds on WML, which hopes to be easier to use, with some powerful features that WML lacks, and which is easier to implement in a game engine. I am already quite pleased with how the current design is, but now I would like to ask for some feedback from UMC developers.

Wes(mere)ScriptLayer (or short WSL) is an API
for defining content and scripting game mechanics for Wesnoth like games in Lua.

WesScript is a language used to define content and make scripts which use the WesScriptLayer. WesScript is a 'flavor' of MoonScript. MoonScript is a recent scripting language which builds on Lua. MoonScript is entirely compatible with lua (and indeed, is compiled to lua) -- it introduces some syntactic sugar for making tables and functions, and makes use of significant whitespace in order to reduce notational burden to a minimum. MoonScript also adds improved support for object-oriented programming. MoonScript ends up feeling like a hybrid between Python and Lua, perhaps. WesScript is 'flavored' moonscript in that it introduces a further bit of syntactic sugar which allows defining lists in a manner very similar to WML.

Motivation

While WML provides a proper solution for storing static values like unit_type or terrain definitions,
it falls short in being a user friendly scripting language.

Lua on the other hand, is an excellent scripting language but introduces a lot overhead when used for pure value storage.

Most games (or other applications) rely on using a different syntax for config storing and scripting.
Being often edited by special tools, the config syntax does not need to be human write or readable in those cases.

Wesnoth like games require scripting languages with static content specialized syntax focused on human readability.
WesScript fits into the requirements.

Tags & Tables

In Lua and Moonscript, any kind of collection / aggregate data is held in a table.

Tables can be used to store a WML-tag's content, and using tables, WSL can mirror the classic WML API quite faithfully.

Let's compare the syntax for table definitions between the discussed languages:
WML

Code: Select all

[side]
    side= 1
    [unit]
        id= "kalenz"
        type= "Elvish Fighter"
    [/unit]
    [unit]
        id= "landar"
        type= "Elvish Archer"
    [/unit]
[/side]

Lua

Code: Select all

side = {
    side = 1,
    unit = {
        {
            id = "kalenz",
            type = "Elvish Fighter"
        },
        {
            id = "landar",
            type = "Elvish Archer"
        }
    }
}

Making use of white-space sensitivity improves the readability a lot:
MoonScript

Code: Select all

side:
    side: 1
    unit:
        {
            id: "kalenz"
            type: "Elvish Fighter"
        }
        {
            id: "landar"
            type: "Elvish Archer"
        }

The additional list syntactic sugar is even more clean and very close to the WML counterpart:
WesScript

Code: Select all

side:
    side: 1
    unit:
        id: "kalenz"
        type: "Elvish Fighter"
    unit:
        id: "landar"
        type: "Elvish Archer"

Tables & Functions

After we defined how to translate tags into tables we need to ask how to put them in use.

The simplest and most obvious solution is to provide functions which take the table type in question as argument.

This data file handling concept is straight how presented in "Programming in Lua".

The GameServer's interpreter makes sure to execute the code in a sandboxed environment not allowing to do any harm.

Functions & Macros

WML is unaware of the function concept while Lua does not come with a buildin macro preprocessor and WSL is not going to add one.

The function concept should in general be enough for the game's purposes.
But it's not easy to translate every macro into a function call.
Writing a script dealing with translating all sorts of macros properly isn't a trivial task.

Unlike the WSL functions the functions translated from macros can take more than one argument and we are not restricted to tables matching a certain scheme.

I will add a description of different macro types and how they can be translated later here.

Have a look at the "former macros" functions being in use by the example scenario.

Macros & Scopes

The most functionality provided by Wesnoth's macros only work in a certain context.

The approach to this is to introduce some scopes.
WSL defines a scope for each former WML toplevel tag.

Thus we have a scope for UnitsWSL, TerrainWSL, EraWSL and so on.

Scopes & Folders

Folders & WesMods

WesScripts group together in containers WesMods.
That is short for "WesmereModules".

A network enabled application allowing to connect a player's "Wes(mere)GameClient" while enabling WesMod provided gameplay is called a "Wes(mere)GameServer".

I am going to write more about Mods, GameServer and GameClient later in a different thread.

Layers & Kernel

1. Game-State
   From the WSL codders point of view the actual Game-State is a single lua table.
   Changing values in the table changes the game state.
   
   This gives the UMC coder full controll over everything but it is also very error prune and not comfortable.
   Nothing prevents the table to get corrupted or to cantain a valid but senseless state.
   [br]
2. Game-Kernel
   In object oriented programming you
   encapsulate a bunch of related data (like the Game-State) into objects which watch that every operation results in a meaningful state.
   See Invariant at wikipedia.
   
   The Game-Kernel (just Kernel from now on) encapsulates the Game-State.
   Although WSL allows direct Game-State access the prefered method is always to ask the Kernel to get things done.
   [br]
3. WSL API
   While the Kernel guards the Game-State and is safe to use, coding is still not comfortable since one has to take care about everything themselve.
   
   Firing events, requesting animations from the display clients, handling side effects...
   
   Thus the WSL provides the WML like API Wesnoth's UMC Designers know so well for years.
   [br]
4. Utility functions
   Wesnoth's macro library, transformed into Lua functions.
   Functions using the WSL to provide commonly used patterns.
   
   TODO: write some more about them.

Coding Conventions

Before we have a look at some WSL coding examples, let's talk about some coding conventions.

Wesnoth's WML attributes are meant to be in snake_case.
WSL is going to adopt this coding style for simple (being not tables themselves) table entries.

There are some irregular attributes like "canrecruit",
since WML needs to be translated anyway, all of them are going to be resolved.

So far I have kept all former MACROS in UPPER_CASE.
Lua and MoonScript's coding conventions suggest to use UPPER_CASE for constants.

Since Macros translate into different kind of objects,
it might make sense to introduce some naming convention to differentiate them.

Content Definition

Let's have a look at a real example now.

The UMC designer calls for Lua functions registering the item in question.
Every of those WSL config functions takes a single table as argument,
allowing a simple and WML like syntax:
Movetype file
Have a look at the WML counterpart when unsure.

Scenario Scripting

Each Scenario is defined by a single table given as arguement to the scenario function.

Capitalized entries in the scenario's table are going to be registered as event handlers.

Code: Select all

scenario

    Moveto:
        filter:
            id: "kalenz"
        command: ->
            message
                id: "kalenz"
                message: "Onwards!"

This can be used to define handlers for custom events as well.

Event handlers without filter usage can assign the handler function directly.

Code: Select all

scenario

    Start: ->
        message
            id: "kalenz"
            message: "Hello World!"

Nested events:

Code: Select all

scenario

    Turn_10: ->
        event
           name: Moveto
           filter:
               id: "kalenz"
               x: 5
               y: 8
           command: ->
               message
                   id: "cleodil"
                   message: "You are comming home late, Kalenz!"

That said, enjoy AOI's first scenario

[Pentarctagon]

So is this meant as a replacement for WML? For lua? Both?

[fendrin]

So is this meant as a replacement for WML? For lua? Both?

The WSL can be used with Lua or WesScript.

So it replaces only WML.
Although Lua code making use of Wesnoth's Lua api is not supposed to be supported.

So maybe "both" is the correct answer.

[Pentarctagon]

Reading through the scenario, I think the thing I dislike the most (beyond 'OMG it's different!') is the lack of any closing block markers. It's not really a problem for the scenario since they never get nested too deeply, but it seems like it would be a real pain to try and figure out exactly where certain things end when there are 10, 20, or more tags layered and the full block of code is larger than the height of the screen. One of the things that I always disliked about lua was the mass of useless brackets which tell me nothing about what's actually happening and while this solves the "mass of brackets" annoyance it still gives me less information about what's going on.

A couple other questions:
1) Why do some values use quotes and others use double brackets? For example:

Code: Select all

DUSK =
    id: "dusk"
    name: [[Dusk]]
    image: "misc/time-schedules/default/schedule-dusk"
    green: -20
    blue: -20
    sound: "ambient/night"

2) When is the "->" supposed to be used? The scenario has it right after an event:

Code: Select all

    TimeOver: ->
        message

but also not right after the event:

Code: Select all

    LastBreath:
        filter:
            id: "Erlornas"
        command: ->

[fendrin]

Reading through the scenario, I think the thing I dislike the most (beyond 'OMG it's different!') is the lack of any closing block markers. It's not really a problem for the scenario since they never get nested too deeply, but it seems like it would be a real pain to try and figure out exactly where certain things end when there are 10, 20, or more tags layered and the full block of code is larger than the height of the screen. One of the things that I always disliked about lua was the mass of useless brackets which tell me nothing about what's actually happening and while this solves the "mass of brackets" annoyance it still gives me less information about what's going on.

Both, the indentation level (== tags layered) and the size of a block are manageable.
Every language offers features to reduce them.

In WML you have 3 different possibilities, that is why I have never seen 20 or more tags layered:

• Macros
• Calling custom event handlers (emulates the function concept)
• The [insert_wml] tag

Lua/MoonScript offers the function concept for this purpose (and others of course).

Also note that WML tends to get quite nested when it comes to flow control, Lua and WesScript do much better there.
This is one of the points why WML is not a good scripting language.
I will show you some nice examples later.

That said, you can of course when you like to write code that is deeply nested use brackets.
MoonScript knows about them just like Lua does.
They will not only give you a better readability but also throw errors when your indentation fails.

A couple other questions:
1) Why do some values use quotes and others use double brackets?

Strings in [[ ]] are meant to be translatable.

2) When is the "->" supposed to be used? The scenario has it right after an event:

Code: Select all

    TimeOver: ->
        message

but also not right after the event:

Code: Select all

    LastBreath:
        filter:
            id: "Erlornas"
        command: ->

"->" is the MoonScript syntax for telling the interpreter that the variable or table entry is to be bound to a function.
After -> the body (a block) of the function starts.
In the case of the "TimeOver" event handler there is no filter usage, thus the table would only contain the command entry.

Code: Select all

    TimeOver: ->
        message

is syntactic sugar for:

Code: Select all

    TimeOver: 
        command: ->
            message

[Pentarctagon]

Reading through the scenario, I think the thing I dislike the most (beyond 'OMG it's different!') is the lack of any closing block markers. It's not really a problem for the scenario since they never get nested too deeply, but it seems like it would be a real pain to try and figure out exactly where certain things end when there are 10, 20, or more tags layered and the full block of code is larger than the height of the screen. One of the things that I always disliked about lua was the mass of useless brackets which tell me nothing about what's actually happening and while this solves the "mass of brackets" annoyance it still gives me less information about what's going on.

Both, the indentation level (== tags layered) and the size of a block are manageable.
Every language offers features to reduce them.

In WML you have 3 different possibilities, that is why I have never seen 20 or more tags layered:

• Macros
• Calling custom event handlers (emulates the function concept)
• The [insert_wml] tag

Lua/MoonScript offers the function concept for this purpose (and others of course).

Also note that WML tends to get quite nested when it comes to flow control, Lua and WesScript do much better there.
This is one of the points why WML is not a good scripting language.
I will show you some nice examples later.

That said, you can of course when you like to write code that is deeply nested use brackets.
MoonScript knows about them just like Lua does.
They will not only give you a better readability but also throw errors when your indentation fails.

My own case is admittedly a bit of an outlier, since it's a modification that contains a single right-click menu item. I split things apart with file includes to help deal with it and better organize things, but if the entire menu was in a single file the greatest tag depth would be in the mid 30s.

[fendrin]

My own case is admittedly a bit of an outlier, since it's a modification that contains a single right-click menu item. I split things apart with file includes to help deal with it and better organize things, but if the entire menu was in a single file the greatest tag depth would be in the mid 30s.

I have promised some examples earlier, now I think translating your menu item would be a good demonstration of what to expect with WesScript.
Can I have the code?

[Pentarctagon]

It's on the add-on server here. Translating the entire thing is probably a bit more effort than it's worth though.

[fendrin]

Here is some update, I plan to integrate it into the first post later.

In Wesnoth a UMC Designer can register additional actionWML tags by using the lua api.
Wesmere also allows to register additions to the WSL api which are in other scopes of the game.

Functions used to register game content are called WSL table handlers.
The [unit_type] WML tag's equivalent is one of those for example.

Before the engine reads the data directory it knows only about a single table handler called wsl_table.
It is used to register every other WSL table handler.

The terrain_type makes a simple and good example.

id: "terrain_type" The identity later used to call the function.
scope: "Terrain" terrain_type will be available when scanning the "Terrain" folder of a WesMod. Scopes are derived from WML toplevel tags. If scope is missing the table handler is used in WesMod toplevels.
description: "Some Text" description is used to store the WSL api reference entries. This will be used to generate the WSL reference webpage and in the in-game command line.

scheme: The scheme table is used to check against the given table for type correctness and completeness. The engine will also error on unrecognized keys. Note the description: for each key.

Those are optional:
on_scan: a hook function to do additional checks
on_load: a hook function to do additional data transformation

[fendrin]

After some more testing the idea how to translate WML variables is getting more stable.

MoonScript comes with the really nice feature to invert Lua's syntax and behavior regarding local and global variables.

Lua

Code: Select all

side_number = 2 -- a global variable
local side_number = 2 -- variable is only visible in the current lexical scope

MoonScript

Code: Select all

side_number = 2 -- variable is only visible in the current lexical scope
export side_number = 2 -- a global variable

So in the actionWSL context (which is used to execute all event handlers and thus the scope where scenario scripting takes place) local variables can live during the whole execution of the event handler's function.
There is no more need to clean these variables, they are removed from the memory when the execution leaves the current scope (a block in most cases).
UMC scripters not feeling firm with the scope system yet can just define all needed variables at the start of the event-handler function,
making sure they are visible during its whole execution.

Whenever a variable is needed that can be accessed in all event handlers (or in a successive scenario) the global variable concept of Lua comes in handy.
So global Lua variables pretty much work like WML variables do.
Cleaning them up like side_number = nil is a good idea to avoid cluttering the global namespace and to reduce memory and safe file footprints.

Since in Lua (and thus in MoonScript as well) functions are just variables themselves both share the same namespace.
This means store_unit = some_value will overwrite (or at least shadow) the WSL store_unit function.
I still wonder if WSL functions should be made read only to avoid this.

Note that I have avoided to use the term "persistence" deliberately until now.
Wesnoth's "persistent variable" concept is a different issue.

[fendrin]

I have promised some examples earlier...

While translating some of the macros from data/core/macros I found a nice example:

https://gist.github.com/fendrin/5e74b3b17c62ea133faf

the WML counterpart:

https://github.com/wesnoth/wesnoth/blob ... s.cfg#L434

[Pentarctagon]

Cleaning them up like side_number = nil is a good idea to avoid cluttering the global namespace and to reduce memory and safe file footprints.

Maybe I'm mis-remembering, but I thought lua variables weren't stored in savefiles currently.



Also, since there isn't going to be any preprocessing, is there a replacement planned for things like #ifdef, #ifhave, #ifver, etc?

[fendrin]

Maybe I'm mis-remembering, but I thought lua variables weren't stored in savefiles currently.

Yes, I think that is right.

Also, since there isn't going to be any preprocessing, is there a replacement planned for things like #ifdef, #ifhave, #ifver, etc?

Most of their use cases can be solved by using global variables or constants.

The answer how to translate them pretty much depends on the exact use case.

[Pentarctagon]

Also, since there isn't going to be any preprocessing, is there a replacement planned for things like #ifdef, #ifhave, #ifver, etc?

Most of their use cases can be solved by using global variables or constants.

The answer how to translate them pretty much depends on the exact use case.

I mean more generally though, if there isn't any preprocessing then they'd all basically have to be replaced with if statements as far as I can tell. #ifver's replacement in particular would need a global constant variable to be set and updated per release.

[fendrin]

I mean more generally though, if there isn't any preprocessing then they'd all basically have to be replaced with if statements as far as I can tell. #ifver's replacement in particular would need a global constant variable to be set and updated per release.

Indeed.
Global variables or constants processed by conditional statements will do the trick.