Strict mode in lua

Discussion among members of the development team.

Moderator: Forum Moderators

Post Reply
User avatar
iceiceice
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Strict mode in lua

Post by iceiceice »

One change which I pushed recently to master was to add "strict globals" mode to wesnoth's lua.

Strict globals is a standard modification to the lua environment, documented on the lua users wiki and distributed with lua (http://www.lua.org/extras/5.2/strict.lua). It makes lua treat it as an error if you try to access a global variable that does not exist, rather than silently return nil. There are two main reasons that I did this:

1. Helps users (and us) to write bug-free code.
2. Makes the lua interpreter much more useful. It was suggested from reading documentation of "ilua", an extra which is meant to make lua run much more comfortably in an interactive interpreter. (http://lua-users.org/wiki/InteractiveLua) The version I merged is based on this version.

Helps to catch bugs
The issue is that by default in lua, variable names are assumed to refer to global variables, unless explicitly declared to be local. Moreover, global variables are not declared, they are instead given the value "nil" if they are dereferenced and do not exist. This has the unfortunate consequence that if you make a typo in a variable name, it is not necessarily a syntax or a runtime error. You can only find such mistakes by reading extremely carefully, or by laboriously debugging your code to find out why variables are `nil` when they shouldn't be. This is merely tedious and annoying for an experienced developer, but it is vastly more difficult for a beginner, who may have poor typing skills and little experience debugging software.

For small scripts it's maybe not a big deal, but for large programs, or programs which act as libraries, it's bad because typos like this are inevitable. There is much discussion on the lua mailing list from users who complained about this behavior and sought solutions, with reactions and suggestions from the creators of lua:

http://lua-users.org/wiki/LocalByDefault
http://lua-users.org/wiki/DetectingUndefinedVariables

There are multiple proposals to change lua to better address this, including "globals must be declared" and the "keyword outer" proposals linked above.

In the meantime however, their main suggestions are
1. Use strict mode, which makes it a runtime error if a global variable is accessed which was never assigned before.
2. Use a static checking module like "globals.lua" which creates a list of all the global identifiers your program accesses. If the list is sorted then hopefully typos become obvious.

The static checker module seems cumbersome and impractical for wesnoth users, so (1) seems better. Many websites which suggest using lua for game development also suggest to use strict.lua.

http://stackoverflow.com/questions/1014 ... on-for-lua
http://notebook.kulchenko.com/programmi ... ugly-parts
http://forums.civfanatics.com/showthread.php?t=532099

Using (1) means that if someone was aggressively using global variables instead of preferring locals (which is not good practice anyways in lua, and much slower), and assumes the "nil by default" behavior, their code may be flagged with a runtime error "variable 'name' must be assigned before being used", when it wouldn't have given an error before.

However, in practice, I didn't find that any mainline campaigns were broken by this change, and none of the micro AI's were broken either. The only thing that broke was that strict mode flagged a bug in our wml tags implementation, which has existed since it was first committed. (This bug has now been fixed.) So I expect that few if any UMC campaigns should actually break from this, and it is more likely to flag bugs in their code than to create problems.
Helps the lua interpreter
Besides helping to catch bugs, strict mode helps the lua interpreter to be useful.

One of the purposes of the lua interpreter is to provide a "gamestate inspector" for the lua state. Suppose I have a variable named "variable", which might contain a number, or nil, during the normal operation of my program. In the current setup, I can simply input "variable" to the interpreter. If "variable" contains "5", then "5" will be printed to the interpreter. If "variable" contains nil, nothing will be printed -- that is how lua handles "print(nil)". If I mistype "varaible" instead, then because of strict mode, this will result in an error "variable 'varaible' must be assigned before being used", and because it was an error, the line will not be cleared, so that I can easily correct the mistake.

If strict mode is disabled, then the case that I made a typo "varaible", and the case that I made no typo and the variable is actually nil, become indistinguishable, and even more cryptically, no output is even printed when I make a mistake. This makes it much more awkward to use the interpreter -- if I make a typo and I don't immediately realize it, I can easily become confused while debugging something.

Without strict mode, it's not clear how to figure out what variables I am actually using, from the interpreter. If this cannot be done comfortably then there's not a whole lot of point in providing an interpreter at all -- querying variables is really much more useful than just being able to call functions that you defined. I think this is the main reason why "ilua" uses this.

You might suggest "why don't we use strict mode only in the interpreter", but I think that won't work, code should run in the same environment within the interpreter that it does normally, otherwise it can become extremely confusing. Besides, if strict mode is enabled for interpreter calls but disabled for normal execution, it will mean that code will give errors when called in the interpreter even if it appears to run fine normally.
Fixing undesired strict mode errors
Most strict mode errors should indicate actual bugs, like you made a typo in a variable name.

If you get an error that isn't a bug like this, here are some ways you can fix it:

0. Assign "nil" at the start of the program to any global variable that you intend to use. Since they are assigned they won't make strict mode errors. Having a list like this is nice anyways because it helps you keep track of what variables you are using.

1. Trap the error with pcall when getting the value of the global "variable"

error, value = pcall(function() return variable end)
if error then
value = nil
end

This is the usual way to handle errors gracefully in lua. (But it can be a little awkward in some cases.)

2. Use rawget to circumvent the check (this has to do with the implementation of strict mode)

rawget(_G, "variable")

This expression can be substituted anywhere for `variable`, so this fix doesn't require adding any lines or restructuring the program at all.

The reason it works is that strict mode works by adding a metatable to the global table "_G". Using the rawget bypasses the checks completely,
and therefore runs as if strict mode was not enabled. It is faster than method (1) also.

3. Disable strict mode for a little bit and put it back later. This is simple but it can create problems:

Code: Select all

ilua.strict = false

--... some code ...

ilua.strict = true
As long as the variable ilua.strict is false, the errors will not be thrown. However, if you don't restore strict mode,
then anything that assumes that strict is on, like the lua interpreter, won't work as intended. If any of the code between
those two lines causes an error, lua won't reach the `strict = true` line and for the rest of your session the lua interpreter
will work differently, which might confuse you. If you put all the code in between those two lines in a pcall then this
won't be an issue.

Here's another reason not to do this. Suppose that I get an error in my lua script. If I know that a particular line is causing the
error, one thing I might do to try to fix it is call "wesnoth.show_lua_console()" just before that line. This will cause wesnoth
to bring up the interpreter at the exact moment before the error happens, so you can inspect the state, call functions, or
try to change things. If ilua.strict is false at that time, it won't work properly... so you have to set it on and off again... but then
if you try to actually run your code from the interpreter, it will give errors because strict mode is on.

4. Put your globals inside a table.

The strict mode errors I added only apply to global variables, in the global table (_G). If foo is a table and foo.bar has not been assigned, it is not an error to refer to it, it's simply nil as usual. (Some authors think this also should be an error, for instance see here for an extension of strict mode which applies to all tables.)

Therefore you may sidestep the errors by putting your globals in a table. If you are writing a module which must require a table anyways, for example if intended to be loaded using wesnoth.require, then this is pretty natural anyways.

We could consider making all tables strict, but imo the main thing is just to get checking for names of loop variables and function arguments, and that should get most of the mileage out of it.

When I first learned of strict mode I was not keen on the idea, but I have gradually warmed to it. I now think it is the simplest and most elegant solution to these problems.

If you don't like strict mode and you want to revert it, that's fine -- I'm writing this post because at least one dev doesn't like it, and it seems hard to have a complete discussion on the irc channel. Compatibility breaking changes are bad because they create work for maintainers, but it seems that this change actually will reduce the work for everyone in the long run, since it adds a safety mechanism to catch bugs (and has already done so in core lua), so to me strict mode seems worthwhile. If you want to revert, please explain why it isn't, and please suggest an alternate way that you think the lua interpreter should work.
Post Reply