RBY Unofficial Patch

[iceiceice]

Previous Title: "Issues with Random Faction/Hero in RBY No Mirror Era"

After playing many games 1v1 using RBY No Mirror era, I noticed that I seemed to get Orcs more than half the time, which seemed very odd. Later more experienced players told me that there seems to be some issues with RBY No Mirror randomness; in fact just based on what hero you recieve you can usually be sure of what hero your opponent has and that you shouldn't use this era on ladder for this reason.

Curious, I decided to test this while I was on the train, starting 50 local 1 v 1 games with RBY no mirror. Only 9 different hero matchups occurred in this test, and half of the games were Orcish Grunt vs Loyalist Bowman. I have not attempted to replicate this on other machines. (Detailed results below)

Later in a ladder match, Garetjax convinced Cackfiend that there was a problem by correctly guessing his faction and hero on turn 2 on Fallenstar Lake.



I decided to look at the code and see what was going on. The macros in RBY_Era.cfg use the wesnoth "rand" command to randomly sample factions, and draws new ones whenever a repeat occurs.

It seems that after 1.10 "rand" was known to have some issues, although I'm not sure if this was directly related. One of the suggested fixes seems be to replace rand with a lua alternative.

http://webcache.googleusercontent.com/s ... clnk&gl=us

In fact, Quetzalcoatl who (unless I am mistaken) maintains RBY addon noticed problems with the map selector using wesnoth rand, and in this post he says that he fixed it by translating it all to lua:

http://forums.wesnoth.org/viewtopic.php ... 7e#p517711

1.10.9 - "RNG haters edition" is out

No more Wesnoth's random variables. After reciving about 10 times in a row same map from pool of 20 i decided to switch to Lua math.random .
Non technical explanation: RBY does not user Wesnoth's RNG anymore.

Cheers
Q

So what I would like to request is, can we get a version of RBY addon in which the faction / hero selection in RBY Era is also done using Lua math.random?

Alternatively, is there anyone who cannot replicate the results I got by starting 50 RBY Era games?

Thanks,
iceiceice

Results of 50 games:

P1 P2 Times
Bowman Grunt 14
Grunt Bowman 11
Deathblade Troll 2
Troll Deathblade 2
Rocklobber Boneshooter 2
Boneshooter Rocklobber 1
WM Fire Drake 1
Bowman Slayer 5
Slayer Bowman 1
Thrasher Leiutenant 1
Leiutenant Thrasher 3
Captain RM (male) 3
RM (male) Captain 1
Flare Thunderguard 1
Thunderguard Flare 1
Stalwart RM (female) 1

Edit: If this should be a reply to Quetzalcoatl's Rushed By Yetis thread rather than its own topic, sorry about that.

[iceiceice]

I realized that since part of the add-on was already switched over to Lua it was actually very easy to hack up the code and switch the rest to Lua even without wesnoth dev experience.

Announcing...

RBY Unofficial Patch

This add-on creates a new era "RBY No Mirror Unofficial" which works almost the same as the old era except with (putatively) better randomness. It also fixes a bug which caused all the heroes to be randomized even if they weren't chosen as random.

Limited testing suggests that the matchups this generates are not nearly as predictable as the old version. I do not know whether or not there is some nastier correlation problem; the next time I am on the train I will collect some more significant data.

I am not a wesnoth dev nor have I made any add-ons before -- if for some reason this add-on does not work, crashes, or gives you bugs I definitely want to hear about it. Please use this patch and let me know if you have any problems or suggestions!

Thanks,
iceiceice

[The Black Sword]

Hi, thanks for reporting this, I remember the issue with the map getting repeated(though I never knew what Q did to fix it!), I never noticed this leader problem before.

I haven't had time to check this yet, but assuming it works I can put it into the official RBY.

We should give Q some time to respond though, but I'm not sure if he checks the forum much anymore.

[iceiceice]

We should give Q some time to respond though, but I'm not sure if he checks the forum much anymore.

Yes, I agree this would be best -- I simply wanted to make something temporary for the meantime. I find RBY addon is the single addon that I use the most, and I think the RBY Era was a really good idea; once I realized that I could do something about the problem I figured I should just do it rather than wait for someone else to address my complaint. Most likely the little code that I had to write is inefficient somehow since I do not know my way around WML, so perhaps Q may discard my version, but if someone finds it useful in the meantime then that's great. It seems like RBY is used by a lot of people.


Also:

When testing the patch with various RBY map pools, I noticed that sometimes I will intermittently get a prestart error and crash:

Error
Error while playing the game : game_error: creating unit with an empty type field

1) I have been able to generate this consistently by making an RBY_Custom map pool which contains only the map Clearing Gushes, in any era , including default era.

2) Same for the map Scarred Foothills.

3) I do not get the error when I start either map directly, without the RBY map pool. So I think it is likely a problem with the cfg files used for each of these maps by RBY add-on ; I am not familiar with these maps and I don't think I am prepared to patch whatever is going wrong there.

4) I have not seen any such errors using Unofficial RBY Era (or any era for that matter) on any RBY map pool excluding these two maps. If you got such an error please let me know.

[Cackfiend]

great work, keep working on it

once it's all ironed out I'll announce it on the ladder main page too

[iceiceice]

tl;dr

I wrote down the results of 50 games from the RBY patch era currently on the add-on server. I didn't get any errors and didn't see any patterns that looked out of the ordinary to me. I think this means the patch is pretty good.

Long version

On the train yesterday I started 50 local 1 v 1 games using RBY No Mirror Unofficial v 3.1 (on the add-on server) and using RBY_Custom map pool which contained all maps except Clearing Gushes and Scarred Foothills.

First of all, I did not get any prestart errors, so I am quite confident now that those are the only maps which have problems in RBY, and I'm reasonably confident that RBY No Mirror Unofficial is mostly if not totally free of such errors.

Now, about the statistics observed: Here's how many times each faction and faction matchup occurred.

. D K L N R U
D . 2 2 6 6 3
K . . 0 6 5 5
L . . . 1 4 4
N . . . . 2 3
R . . . . . 1
U . . . . . .


D: 19
K: 18
L: 11
N: 18
R: 20
U: 16

Q: Should we be concerned that loyalist appeared fewer times than the other factions?

A: IMO this is not unreasonable. Each faction would be expected to occur in about 16 of the 50 games. The variance is (1/3- 1/9) * 50 ~= 10, so the standard deviation is a bit more than 3, so everything was well within 2 standard deviations of the mean. (As a general rule of thumb, 2 standard deviations corresponds to a 95% confidence interval.) So I don't think the fact that e.g. Rebels appeared twice as often as loyals above should be considered statistically significant.

(Besides everyone knows loyals are OP, the less loyalist games the better )

Q: Should we be concerned that some faction matchups occurred as many as 6 times and others like Loy vs. Knalgan didn't appear at all?

A: There are 15 matchups, so each should occur 3 or 4 times in expectation. Again, because 50 is not that many points, the standard deviation is going to be like 2 or 3. It is a bit odd that a matchup was skipped, the odds that that would happen are probably about 1/10 or 1/20. (I estimate -- (1-1/15)^50 ~= e^{-50/15}, somewhat less than 2^{-3}). So it could be significant, but I don't think this test is conclusive and I'm not that concerned. Given that Loy occurred so rarely in this sample its not really that surprising. The chance that in 11 loy games a random opponent would never be knalgan is about 20% or so. (I estimate -- (1-1/5)^11 ~= e^{-11/5}, somewhat less than 2^{-2}). For what its worth, I have observed that other times that I started test games I did observe Loy vs. Knalgan, just not in these particular 50 trials, so its not like a bug is preventing this matchup somehow.

Q: As far as what leader matchups occurred: No leader matchup was exactly repeated, but some leader matchups sort of repeated -- for instance we had (p1) Stalwart vs (p2) Deathblade and (p1) Deathblade vs (p2) Stalwart, and the same for Stalwart vs Crossbowman. Should we be concerned?

A: There are about 50 leaders, hence about 50^2 = 2500 matchups. As a general rule of thumb if we sample m numbers independently and uniformly at random from 1 to n, we should expect collisions to begin to appear when m gets as large as ~ n^{1/2}. (See Birthday Paradox for more info.) Since we took 50 samples, I think seeing a handful of collisions, i.e. a matchup occurs 2 or maybe 3 times, is normal and not aberrant.

What was aberrant in the sample from the first post was seeing Grunt vs Bowman half the time, and only 9 different matchups overall; in a truly random sample of 50 games the chance of that many collisions would be astronomically small.

Q: The problem we were supposed to be fixing was that in the official RBY era, knowledge of your own leader often determines the enemy leader and faction, and thus could significantly affect your strategy. Do we know that this is not the case now?

A: Unfortunately we can't really know this from such a small sample size. There are 50 leaders and we took 50 samples, so each leader occurred at most a handful of times. This means that we can't really learn anything about the conditional distribution of e.g. the p2 leader given the p1 leader, and we'd need to know that that is always relatively uniform to know that the era is not affecting strategy. To do that, ideally, we would need need several hundred sample points with each p1 leader possibility, for a total of maybe ten or twenty thousand sample points.

Of course what is most important is the enemy faction, not so much the leader. Still, for similar reasons to say anything about the distributions of e.g. p2 faction given that p1 leader = x should require thousands of sample points. For example, suppose that it turns out that because of nasty correlations in Lua RNG, whenever you have the Elvish Hero leader you are very *unlikely* to be against an Undead opponent. This could be actionable because elvish archers are pretty effective against every faction except Undead; if you knew about or suspected this correlation, you might feel comfortable recruiting extra archers whenever you get the Hero, since there's definitely upside if your opponent turns out to be Drake and little downside if its never UD. On the other hand 50-100 truly random samples have good odds not to contain Elvish Hero vs Undead anyways, so there is no way to detect that this is happening without taking many more data points.

Even though we can't rule out any scenario like that right now, I would still argue that we shouldn't really worry about it. The reason is that even the most experienced players may only play at most 1000 - 2000 games on ladder in total it seems, and even then over several years. If there is a pattern but it can't be noticed after only 50-100 games or so, most likely no one will notice it at all. Since we didnt see any serious problems in the 50 game sample, we should just run with it and see if later anyone announces a pattern.

If someone were really motivated (bored), they could try to rig up an automated system to take thousands of samples, and then, since a histogram of leader matchups won't really be human readable, they could write a python script that tallies it all, computes the expectation and variance for the numbers for each matchup, and flags any large deviations. But without any reason to be suspicious I think that is serious overkill.

[u]Bottom Line[/u]

50 test points is not enough to be sure that the new era is definitely fair. However 50 points was enough to see that the old era was obviously problematic. So at least we should be confident that the new version improves the situation, whether or not it solves the problem completely.

~ iceiceice out ~

Raw Results

P1 P2 Times
Drake Fighter - Revenant - 1
Druid - Drake Fighter - 1
Thrasher - Javelineer - 1
Drake Fighter - Sorceress - 1
Rogue - Sorceress - 1
Stalwart - Rocklobber - 1
Bowman - Deathblade - 1
Steelclad - Deathblade - 1
Thunderguard - WM (elves) - 1
Trapper - Deathblade - 1
Boneshooter - Trapper - 1
Troll - Fire Drake - 1
Grunt - Druid - 1
Deathblade - Stalwart - 1
Dark Sorcerer - Bowman - 1
Flare - Marksman - 1
Revenant - Grunt - 1
Trapper - Thrasher - 1
RM (elf) - Steelclad - 1
Bone Shooter - Marksman - 1
Flare - WM(loy) - 1
Stalwart - Crossbowman - 1
Slayer - Deathblade - 1
Fire Drake - Rocklobber - 1
Dark Sorcerer - Troll - 1
Thunderguard - Fire Drake - 1
Pikeman - Bone Shooter - 1
RM (loy) - Marksman - 1
RM (elf) - Arbiter - 1
Crossbowman - Stalwart - 1
Drake Warrior - Deathblade - 1
Elvish Hero - RM (loy) - 1
Stalwart - Deathblade - 1
Arbiter - Dark Sorcerer - 1
Thrasher - Elvish Hero - 1
Troll - Stalwart - 1
Arbiter - Slayer - 1
Rocklobber - RM (elf) - 1
Troll - Flare - 1
WM (elf) - Swordsman - 1
Troll - Drake Warrior - 1
Trapper - WM (elf) - 1
Captain - Swordsman - 1
Druid - Trapper - 1
Troll - Thrasher - 1
Lieutenant - Crossbowman - 1
Trapper - Grunt - 1
Steelclad - Druid - 1
Flare - Ranger - 1
Lieutenant - Bone Shooter - 1
Slayer - Rogue - 1

[Cackfiend]

looks good to me


what do you think about adding it into RBY, TBS?

[The Black Sword]

Ok, seems to check out to me, so I released a new version with the patch.

I'll have a look at those maps.

[iceiceice]

Ok, I don't know what will happen now if I delete the patch from the add-on server, will that automatically delete it for all the users? I decided to update the patch so that now it puts a note saying that it is Obsolete and to use v 1.10.21 of the RBY add-on instead.

[Quetzalcoatl]

Hello, great work!

Unit with empty type error - likely its related to shuffle sides option. I never saw this error when I was playing with shuffling disabled. As for the questions I would not worry much, to me everything seems to be fine.

Q: As far as what leader matchups occurred: No leader matchup was exactly repeated, but some leader matchups sort of repeated -- for instance we had (p1) Stalwart vs (p2) Deathblade and (p1) Deathblade vs (p2) Stalwart, and the same for Stalwart vs Crossbowman. Should we be concerned?

This issue may again be related to shuffle sides. There is patch submited (unfortunatelly uncommited) at: http://gna.org/bugs/?19055. To put is as simple as possible: if shuffle is enabled both players get leader with same id (where id is index in array of leaders). So now if p1 has leder no 7 and p2 gets leader no 7 as well. Now if p2 has only 5 possible leaders engine crashes with empty type error.

RBY has git repo so if you would like to have direct access to it let me know (https://github.com/WesnothRBY - last commit was tagged 1.10.20).

Cheers
Q