inserting into tables out of bounds

Discussion of Lua and LuaWML support, development, and ideas.

Moderator: Forum Moderators

Post Reply
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

inserting into tables out of bounds

Post by Anonymissimus »

It seems that this code

Code: Select all

	local a = {}
	table.insert(a, 2)
	table.insert(a, 2, 3);
	dbms(a)
has worked in the past in wesnoth lua even if the second line is left out, so that a table with something at [2]= but nothing at [1]= is created, according to the code I had written back then.
Wasn't there some "strict lua" kind of change ? I find a post from 06 Jan 2015 but that's not far enough in the past; I am speaking of 1.12.1@13b42234ed5267077b7a515f2b38cab102dcc0a4. It seems that the problem wasn't present yet when I updated at the start of the 1.11 beta series.
This forces a bug into my once valid unchanged code that is at least not easy to fix, since that index is carrying needed information and cannot be just left unspecified. I have to think through the whole thing probably.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
User avatar
iceiceice
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Re: inserting into tables out of bounds

Post by iceiceice »

Strict lua is not in 1.12, its in 1.13. And it is not a semantic change, the only difference you would see is an error being thrown or not. (I guess that's a semantic change if you are doing something with pcall but for an example like above it couldn't possibly be relevant.)

The most significant change in 1.11.x was that our lua was upgraded from 5.2.0 to 5.2.3.
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: inserting into tables out of bounds

Post by Anonymissimus »

It is unlikely that I was unaware of the possibility that such table can be created or that I didn't test that it's working as expected. I just checked that my system's lua (Lua 5.1.4) can execute the code above correctly, no error because of the missing entry at [1]. So it looks as if that' s a silent change between the lua versions which I consider kind of significant, should be noted somewhere.

EDIT
Still okay with my other system's Lua 5.2.0, could pls someone with newer lua check that it fails ? (Aside from the dbms(a) call which is only my debug message function).
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
User avatar
iceiceice
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Re: inserting into tables out of bounds

Post by iceiceice »

I think that you should simply upgrade your systems lua, lua 5.1 has major differences with lua 5.2.

The most recent change was 5.2.0 -> 5.2.3 which was conducted by shadowm here: https://github.com/wesnoth/wesnoth/comm ... 06d857f9b3

The upgrade to lua 5.2.0 was conducted by crab, more than 3 years ago:
https://github.com/wesnoth/wesnoth/comm ... 86931cf135

So wesnoth hasn't used lua < 5.2 in more than 3 years.

Edit: I reread your post. The following things aren't clear to me:
  • What does dbms do?
  • What do you expect the value of a to be.
  • What do you observe? Do you observe a lua error?
  • When you say "still okay" at 5.2.0 what exactly does that mean
User avatar
Elvish_Hunter
Posts: 1575
Joined: September 4th, 2009, 2:39 pm
Location: Lintanir Forest...

Re: inserting into tables out of bounds

Post by Elvish_Hunter »

Anonymissimus wrote:I just checked that my system's lua (Lua 5.1.4) can execute the code above correctly, no error because of the missing entry at [1].
On 5.2.3, this is the result:

Code: Select all

Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio
> t={}
> table.insert(t,3,"value")
stdin:1: bad argument #2 to 'insert' (position out of bounds)
stack traceback:
	[C]: in function 'insert'
	stdin:1: in main chunk
	[C]: in ?
This behavior, according to this page, was changed in Lua 5.2.2, and affects both table.insert and table.remove.
iceiceice wrote:I think that you should simply upgrade your systems lua, lua 5.1 has major differences with lua 5.2.
Yes, but Anonymissimus is on Windows, so I guess he's using Lua for Windows, which still uses Lua 5.1.4.
By the way, if someone is interested in it, please grab it while you can, because Google Code will close in 2016.
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
User avatar
iceiceice
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Re: inserting into tables out of bounds

Post by iceiceice »

According to lua docs here: http://www.lua.org/pil/19.2.html

Code: Select all

a = {}
table.insert(a, 2)
should result in an array of size 1, with a[1] = 2.

@Elvish_Hunter: He could be using cygwin with the standard lua interpreter I think.
User avatar
Elvish_Hunter
Posts: 1575
Joined: September 4th, 2009, 2:39 pm
Location: Lintanir Forest...

Re: inserting into tables out of bounds

Post by Elvish_Hunter »

iceiceice wrote:According to lua docs here: http://www.lua.org/pil/19.2.html
Unfortunately, that documentation refers to a version of the Programming in Lua book that targets Lua 5.0. That behavior has changed: for example, Lua 5.2 has a bit32 library (bitwise operations), but in that documentation it isn't mentioned; it is, however, in this one.
iceiceice wrote:@Elvish_Hunter: He could be using cygwin with the standard lua interpreter I think.
There are also some precompiled binaries for 5.2.3 here.
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
User avatar
iceiceice
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Re: inserting into tables out of bounds

Post by iceiceice »

it is, however, in this one.
Okay, but then the answer to the question at hand is the same.
Lua 5.2 manual wrote: table.insert (list, [pos,] value)

Inserts element value at position pos in list, shifting up the elements list[pos], list[pos+1], ···, list[#list]. The default value for pos is #list+1, so that a call table.insert(t,x) inserts x at the end of list t.
If the thing I linked earlier was lua 5.0, then it appears that for every version of lua, table.insert(t, 2) would have inserted an element 2 at the end of the array.
anonymissimus wrote: ... so that a table with something at [2]= but nothing at [1]= is created, according to the code I had written back then.
What exactly does that mean? You are expecting that [1]=nil, and [2]=2 ?
So that insert(t, 2) creates a 2 at position 2?
It seems very implausible to me that that it ever worked that way.
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: inserting into tables out of bounds

Post by Anonymissimus »

iceiceice wrote:
  • What does dbms do?
  • What do you expect the value of a to be.
  • What do you observe? Do you observe a lua error?
  • When you say "still okay" at 5.2.0 what exactly does that mean
  • Displays very comprehensive information about any sort of lua variable. I wrote it once, it's in the WLP and pretty well-known among wesnoth lua coders. Not relevant for the problem at hand.
  • Without the second line, basically { [2] = 3 }
  • Yes ("out of bounds").
  • No lua error.
iceiceice wrote:The most recent change was 5.2.0 -> 5.2.3 which was conducted by shadowm here: https://github.com/wesnoth/wesnoth/comm ... 06d857f9b3
Which is indeed the responsible change, as I just figured out by playing with my lua version.
Thanks to E_H for investigation.
iceiceice wrote:He could be using cygwin with the standard lua interpreter I think.
No.

EDIT
What exactly does that mean? You are expecting that [1]=nil, and [2]=2 ?
So that insert(t, 2) creates a 2 at position 2?
It seems very implausible to me that that it ever worked that way.
[1]=nil and [2]=something only in case nothing was inserted to [1] but something to [2], with index specified. I agree that it's looking strange, but it worked this way until recently, as I just tested.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
User avatar
iceiceice
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Re: inserting into tables out of bounds

Post by iceiceice »

Edit: I didn't understand your first post, I thought that you were interesting in table.insert(a, 2) but I guess you are talking about table.insert(a,2,3). Still wasn't clear to me what the "problem" was.

This is what I currently get in wesnoth 1.13.0+dev. (took me a while to compile it)

http://i.imgur.com/JSdY4jL.png

I think that's just how lua 5.2.3 works. Probably they throw an error because, if you mess up your code and put insert(t, x, y) where x is very large, it will allocate a lot of memory and take a very long time, which probably you would rather not find by profiling.

Anyways it was announced that we changed to lua 5.2.3 here: http://forums.wesnoth.org/viewtopic.php ... 90#p577543
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: inserting into tables out of bounds

Post by Anonymissimus »

iceiceice wrote:I guess you are talking about table.insert(a,2,3).
Indeed.
Anyways it was announced that we changed to lua 5.2.3 here: http://forums.wesnoth.org/viewtopic.php ... 90#p577543
I don't see this change (table.insert) listed anywhere in the corresponding links in that post, and it unfortunately introduces a severe bug in my addon in wesnoth's late stabilization phase. :shock: Ofc nobody could expect that, and it fixed other bugs.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
User avatar
iceiceice
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Re: inserting into tables out of bounds

Post by iceiceice »

Yeah... it is a bit funny that "compatibility breaking lua upgrade" was announced in wesnoth RC3.

It sounds like the older behavior of table.insert was worse though. http://lua-users.org/lists/lua-l/2013-01/msg00442.html
:hmm:

TBH though I think the folks at lua might be somewhat to blame. All these releases 5.2.1, 5.2.2, 5.2.3 are listed as "bugfix releases": http://www.lua.org/news.html

I wouldn't expect code that used to work to start throwing safety errors because of a bugfix release.
Anonymissimus
Inactive Developer
Posts: 2461
Joined: August 15th, 2008, 8:46 pm
Location: Germany

Re: inserting into tables out of bounds

Post by Anonymissimus »

It was much easier than I thought.
A table of the form { [2] = ... } is still valid, just not via using table.insert().
Instead of table.insert(a, 2, 3); one can do (much more common, actually) a[2] = 3; without the need of something being at [1]=. That way one can choose to have an out-of-bounds check via table.insert if that's wanted.
projects (BfW 1.12):
A Simple Campaign: campaign draft for wml startersPlan Your Advancements: mp mod
The Earth's Gut: sp campaignSettlers of Wesnoth: mp scenarioWesnoth Lua Pack: lua tags and utils
updated to 1.8 and handed over: A Gryphon's Tale: sp campaign
Post Reply