Split from [is AI cheating?]

General feedback and discussion of the game.

Moderators: Forum Moderators, Developers

Tad_Carlucci
Developer
Posts: 420
Joined: April 24th, 2016, 4:18 pm

Re: Split from [is AI cheating?]

Post by Tad_Carlucci » December 13th, 2016, 10:17 pm

iceiceice wrote:If you think the numbers being displayed are not correct, you could look at the code, it is all open source.

The most significant issue in this regard in wesnoth 1.12 is that we use a much older random number generator, the C "rand" function.
[...]
Whether it makes a noticeable difference for the user, I can't say.
1) If you think there's an error in the math, point it out and it will be corrected.

Saying "I tried 20 times and it always ..." won't do it. For example, if we assume the odds are 1-in-4 and there are 5,000 players today playing a total 100,000 games, (skipping a pile of mathematics) expect that at least one player (probably several dozen ... or even a few hundred ... I did skip the math) would be able to make that statement on a daily basis.

2) The discussion on the random number generators really only matters if we're talking about computer security such as encryption.

From the point of view of game design, the issue was that the output from the old "rand()" function is predictable. While I'm sure I could use the RAW results from a few calls to the old random number generator to PREDICT all future results, I would be amazed it anyone could take a few (or even a few hundred) 'cooked' samples from combat and do the same. Yes, it's possible; no, you probably won't live long enough to do it with certainty.

3) I would be amazed if a normal user could show any meaningful difference between the combat results obtained from the two generators.

4) While the comments about real and random number epison values are true, they are generally immaterial. Most game odds are rather large numbers (1-in-2, 1-in-6, 1-in-20, things like that). While the NET odds from a series of random events (such as combat) might be very small and might even approach epsilon, that is simply a theoretical value and has no effect.
I forked real life and now I'm getting merge conflicts.

User avatar
Inky
Forum Moderator
Posts: 516
Joined: September 22nd, 2014, 1:02 am
Location: USA

Re: Split from [is AI cheating?]

Post by Inky » December 13th, 2016, 11:31 pm

On the subject of the "force chance to hit" macro, I never liked how the displayed percentages were wrong. I wish it would just display 0% for units which are impossible to hit, otherwise you only find out after you wasted several units' moves (and probably got lots of retaliation damage in the process). (I know it's usually clear that you're not supposed to attack these units, but part of the fun of wesnoth is doing things you aren't supposed to :P )

On the original topic, a good way to get a feel for your luck is to constantly check the statistics table (s hotkey). You're probably getting better luck than you think :)

User avatar
The_Gnat
Posts: 2071
Joined: October 10th, 2016, 3:06 am
Contact:

Re: Split from [is AI cheating?]

Post by The_Gnat » December 14th, 2016, 7:40 am

In response to Tad_Carlucci i ran a randomizer 300 times between 1 and 100, here is the result:
random.png
I have highlighted in red the times when the randomizer could be claimed to be cheating, overall it appears to be fairly reliable.

User avatar
GunChleoc
Translator
Posts: 473
Joined: September 28th, 2012, 7:35 am
Contact:

Re: Split from [is AI cheating?]

Post by GunChleoc » December 14th, 2016, 11:05 am

I think people have been complaining about the RNG since Wesnoth was first published...

The thing is, you are less likely to notice your own luck than the luck of the AI. Also, the AI uses the same RNG as the player, so if the AI cheats, you cheat.

MagnetoRN
Posts: 25
Joined: December 6th, 2016, 5:37 pm

Re: Split from [is AI cheating?]

Post by MagnetoRN » December 14th, 2016, 11:59 am

Well, considering that each combat would use 6-8 results in average (3 or 4 attacks of each opponent), I would say that pretty regularly the RNG delivers a quite huge almost perfect sequence of numbers above 50% (and sometimes even 60% and 70%). I really don't know how random is that, since I didn't find such long strings with values below 50%. These ones I highlighted can pretty much frustrate the player in combats where he should have a clear advantage, creating many opportunities for a character with 70% (like Delfador) to hit only once.
Attachments
RNG.png

Whiskeyjack
Posts: 424
Joined: February 7th, 2015, 1:27 am
Location: Germany

Re: Split from [is AI cheating?]

Post by Whiskeyjack » December 14th, 2016, 12:43 pm

GunChleoc wrote:I think people have been complaining about the RNG since Wesnoth was first published...

The thing is, you are less likely to notice your own luck than the luck of the AI. Also, the AI uses the same RNG as the player, so if the AI cheats, you cheat.
As someone that has played a lot of Wesnoth over the last 10 years I can say this, even without looking at the maths: This statement is true. If you pay self-critical attention you will realize that the odd one out results occur for the player on the same rate as for the AI.

However, I think there might actually be a problem with the probabilities of edge cases. If you calculate the probabilities of some events and then the probability of multiple such events happening in the matter of 10 actions, you sometimes get probabilities that are not like 1/10.000 but rather close to impossible. And this is something that occurs every so often. I´m not sure if the not so absolute randomness results in there being a string of numbers that lead to this phenomenon (as in certain conditions you will get said string of very improbable events), it might also just be me and my brother being the odd ones out among all the Wesnoth players in the world.
I guess I´d have to prove the math to you to convince anyone and since this is my subjective assessment, I also might just be wrong. It´s a mood point anyway, except for the occasional frustration every couple hundred games it´s not so much of a problem.
Under blood-red skies an old man sits in the ruins of Carthage contemplating prophecy.

User avatar
skeptical_troll
Posts: 423
Joined: August 31st, 2015, 11:06 pm

Re: Split from [is AI cheating?]

Post by skeptical_troll » December 14th, 2016, 2:32 pm

This qualitative and subjective approaches look really pointless to me. There are rigorous ways of testing randomness, looking at a string and searching for odd things certainly isn't one of them, especially if you haven't defined what is 'odd' in advance.

The probability of having exactly that string of 300 numbers was 1 in 100^300. That's ridiculously low, but it happened! I guess you can see the fault in this way of reasoning.
There are loads of 'improbable' events, each of which has a very low chance of happening. But since there are many, finding some is not so unlikely.

Tad_Carlucci
Developer
Posts: 420
Joined: April 24th, 2016, 4:18 pm

Re: Split from [is AI cheating?]

Post by Tad_Carlucci » December 14th, 2016, 4:15 pm

skeptical_troll wrote:This qualitative and subjective approaches look really pointless to me. There are rigorous ways of testing randomness, looking at a string and searching for odd things certainly isn't one of them, especially if you haven't defined what is 'odd' in advance.

The probability of having exactly that string of 300 numbers was 1 in 100^300. That's ridiculously low, but it happened! I guess you can see the fault in this way of reasoning.
There are loads of 'improbable' events, each of which has a very low chance of happening. But since there are many, finding some is not so unlikely.
"Really pointless" vastly under-states things.

In actual point of fact, one of those "rigorous checks" is to specifically check for those "odd strings" and ensure they occur. Failure to find them is proof of failure in the random number generator.

I see conversations like this many, many times. I find it amusing that the people who begin by claiming they have found an issue almost always present proof that things are working properly as proof of their claims something is wrong.
I forked real life and now I'm getting merge conflicts.

Whiskeyjack
Posts: 424
Joined: February 7th, 2015, 1:27 am
Location: Germany

Re: Split from [is AI cheating?]

Post by Whiskeyjack » December 14th, 2016, 5:40 pm

Spoiler:
I am aware of the fact that I might be exaggerating in the example above (different kind of memory bias) and that this is my subjective experience. And as I´m not interested in checking the math at the moment, I´ll trustfully take iceiceice´s and Tad_Calrucci´s word for it that I´m wrong.

The misguided (and unnecessary, as I realized some time after posting) point I wanted to make was too much of a psychological nature in a tech discussion and I humbly apologize for chiming in at all and my choice of words, which, on reread, implies things I didn´t intend to say.
Under blood-red skies an old man sits in the ruins of Carthage contemplating prophecy.

User avatar
iceiceice
Developer
Posts: 1056
Joined: August 23rd, 2013, 2:10 am

Re: Split from [is AI cheating?]

Post by iceiceice » December 14th, 2016, 5:47 pm

+1 skeptical_troll

These pictures of numbers, with various colored boxes, supposed to be "proving" that the RNG is biased... it reminds me of the "crazy walls" scene from the movie "A Beautiful Mind".

Here's a way that you could attempt to give evidence that the RNG is biased.

1. Think of a random variable that shows the supposed bias. Like, consecutive runs of large numbers.
2. Calculate what the expected value is. I.e. how many runs of a certain length there would be, for sequences of a certain length, if the dice were totally fair.
3. Calculate what the chance of a "large deviation" is. I.e. "I calculate that the chance to get twice as many bad runs as expected is less than .01%"
4. Make a hypothesis, the "null hypothesis", that the large deviation won't occur.
5. Then, take actual data, and see if the event actually happened or not. If it did happen, despite low probability, then it's considered "statistically significant", and suggests the fair dice assumption isn't correct.

The important thing is, you don't get to change the null hypothesis after you see the data.

There's a saying, "if you torture the data, it will confess".

If you just collect random data and then look for patterns, you will not fail to find patterns. The human brain is very good at that. They just won't have any meaning.
Attachments
crazy walls
crazy walls

User avatar
The_Gnat
Posts: 2071
Joined: October 10th, 2016, 3:06 am
Contact:

Re: Split from [is AI cheating?]

Post by The_Gnat » December 14th, 2016, 7:50 pm

I fully agree with iceiceice and the reason I ran the randomizer was infact not an attempt to claim unrandomness. I personally believe that out of those 300 times (run from the 1.12 generator) it was reliably random. I think this because only 3 times (as highlighted in red) were there strings of similar numbers. Even though magneto has a valid point with the yellow boxes it is not very significant because like iceiceice says if you exam the numbers close enough you can alwaya find something.

Post Reply