Add prompt at end of campaign scenario to upload replay to help improve game balance

[max_torch]

Context:
The AI and code logic of Wesnoth sometimes undergoes change that can affect balance of campaigns, and its always an ongoing objective to improve the balance of single-player campaigns. However, balancing is not always easy and the more data available the better the game can be balanced. Also, when feedback is given, it is not always from the perspective of a new or novice player, and has a tendency to be skewed towards having more feedback from expert players. This suggestion aims to assist Single Player campaign rebalancing efforts for all available difficulties of the game by hopefully resulting in an increase in available feedback data.

Proposal:
Add a prompt at the end of a single player campaign scenario which says something along the lines of "would you like to submit this replay so that we can improve single player game balance?". A checkbox will be provided with the label "Don't ask me again" so that anyone who isn't interested and finds the prompt annoying can immediately opt out of the feature forever. Pressing a button "Submit replay" closes the prompt, uploads the replay, and advances the player to the next scenario. Also in the same prompt window which appears, another button can be included which takes the player to the forum thread of the scenario, so that they can give more detailed feedback if they wish, in addition to just uploading a replay via the prompt.

Benefits:

• The hope is that the amount of useful gameplay feedback will increase, because providing it is as simple as clicking a button once prompted.

• Which will result in having more data, including a healthy amount from novice and beginner players, that can be used to inform developers if the campaigns are balanced and what needs to be tweaked.

• Technically, if it was possible to extend this to UMC, UMC authors might be also interested since they also care about balancing their single player campaigns.

Some Example Statistics that can be compiled from mass replays:

• Seeing the amounts of bonus gold players have when starting a scenario.

• Seeing at what turn most players finish a scenario.

• Seeing how many units (and value of those units) players typically lose in a particular scenario

Obstacles:

• Wesnoth mainline single player gameplay is balanced for a certain style of play that involves not reloading a turn in the middle of a scenario in order to get a result where more of your unit's attacks hit, aka savescumming. Therefore feedback involving replays where the player savescums is not desired as it will lead to the wrong conclusions when balancing the game. The problem is that there is currently no feature which detects if a player savescummed in a replay.

Questions:

• Do you like this idea?

• Do you think there is a way to overcome the obstacle and will it be difficult or simple? Would you still benefit if you got replays from savescummed games?

• Does this idea inspire you to think of some related or similar idea that can somehow improve the amount and quality of feedback and convenience of giving feedback?

• What are other considerations?

PS:
Technically, you could ask people to create their campaign game in "multiplayer" rather than in "Campaign" or "Load" and their games will be uploaded to replays.wesnoth.org; but you would have to make a communication campaign telling people "hey, if you want to contribute to helping improve the game, play campaigns on "multiplayer" instead of through "campaign".

PPS:
I decided not to create a Feature Request on Github at this point in time, so that there will be less fragmentation of the discussion involving this topic (One central point of discussion).

[egallager]

I do like the idea, but, here are some things to consider:
- Collecting user data would mean having to worry about GDPR compliance
- If all players started doing this, how much server space would it take up?
- How exactly would the uploaded replay files get analyzed? Would there be some sort of automated tools to collect the suggested statistics, or would campaign authors have to review replays manually?

[lhybrideur]

The idea is nice.
It needs to however be automatically disable is the computer is not connected to the internet.
I don't think savescumming is a problem if you have a lot of data.
If you ask each time if the player want to upload the data and if the data is anonymized, I think GDPR is OK with that.
Why would you need to be connected to multiplayer to have the data uploaded to replays.wesnoth.org ? Sure, it would need to be reorganized to have a part dedicated to those anonymized single-player campaign files (like sort by campaign instead of date), but that's it.

[gnombat]

Didn't this feature basically exist at one time in old versions of Wesnoth? (It worked a little bit differently from the proposal above, but I think the basic intent was the same.)

Wesnoth 1.6

I don't know why it was removed.

[max_torch]

Here's a visual mockup showing how the feedback prompt might look like:

As you can see, I figured out a potential solution to the savescumming obstacle: Have the player self-report if they savescummed in the replay.
Properties of this menu:

• Both radio buttons are unticked by default

• The selected radio button is saved as metadata in the replay file

• Submit button is greyed out and unclickable until a radio button is ticked

• If "Don't show this again" is ticked, after this prompt is closed it never shows up again but an option will be provided in Preferences > Advanced if people ever want to re-enable it

• "Close without submitting" is spelled out this way to make it clear that no data is shared when clicking this button

• The button with the info icon opens up a FAQ page which contains more information such as privacy concerns and the way the game is balanced

• The button with the arrow icon opens up the forum thread for the scenario

• "https://replays.wesnoth.org/SP/" is a hypothetical new directory for storing replays submitted this way

Thoughts?

Collecting user data would mean having to worry about GDPR compliance

What's great about open source is that everyone can look at the code and see the way the data is collected. Additionally, you are asked each time you want to submit.

If all players started doing this, how much server space would it take up?

For reference, 33,407 gzipped ladder replays is 928 MB. Also, all multiplayer games are all already automatically uploaded to and successfully stored in replays.wesnoth.org.

How exactly would the uploaded replay files get analyzed? Would there be some sort of automated tools to collect the suggested statistics, or would campaign authors have to review replays manually?

Take a look at this, a site which extracts data from a replay. Im working on something similar, except it extracts more data, supports old versions, and can iterate over all files in an entire directory, but im not yet ready to fully announce it with complete details.

It needs to however be automatically disable is the computer is not connected to the internet.

Because the submission is always manual, data never gets submitted unknowingly unless the player actually presses Submit.

PS
@gnombat, thanks for the screenshot. I was able to use it for the visual mock up.

[lhybrideur]

It needs to however be automatically disable is the computer is not connected to the internet.

Because the submission is always manual, data never gets submitted unknowingly unless the player actually presses Submit.

What I meant was that there is no use prompting if the computer is not connected to the internet, because you just cannot upload the replay.

[doofus-01]

Can save-loading or some other monkey business be screened out automatically by filtering on statistics? Really lucky outliers wouldn't necessarily need to skew data, and the upload could be disabled or have a flag set if debug was enabled.

I ask because getting rid of that question with the radio buttons would be nice.

[egallager]

Can save-loading or some other monkey business be screened out automatically by filtering on statistics? Really lucky outliers wouldn't necessarily need to skew data, and the upload could be disabled or have a flag set if debug was enabled.

I ask because getting rid of that question with the radio buttons would be nice.

Well, as far as the "other monkey business" goes, when viewing a replay, it shows when debug commands were used with a big message warning that the player might be cheating, so the info is probably already there...

[max_torch]

Can save-loading or some other monkey business be screened out automatically by filtering on statistics? Really lucky outliers wouldn't necessarily need to skew data, and the upload could be disabled or have a flag set if debug was enabled.

I ask because getting rid of that question with the radio buttons would be nice.

I use "AI" and "model" interchangeably below.

Well, technically, one possible hypothetical process of developing this, is to create a dataset of a lot of replays labelled as "reloaded turns" and "no turns reloaded" and to turn those replay files into feature vectors, a format which can then be used to train a machine learning model until it accurately classifies replays.
Possible things the AI could look at would be the way luck has been distributed over the turns of a scenario in each replay, and see which ones are most likely "statistical signatures" of replays that load/saved a lot. If there is an unusually high concentration of luck, or too many lucky turns, taking into account all of the thousands of labelled replays it has observed, it will say "this replay has x% chance of having been a replay with load/saving".

We already have a lot of replays but the problem is we don't have "labelled" replays, which is what we need to do supervised machine learning to train a classifier.

TLDR a little advanced but not out of the question way to do this would be to train a machine learning model to act as a replay classifier.
Dont be dismayed if the size of our dataset is not yet in the thousands, we can try to see how much accuracy we can squeeze out of a small dataset, but of course, as these things go, the larger the training data the better.

EDIT: ironically, one way to produce said dataset would be to implement this feature, radio buttons and all, since this tags the replay as "reloaded" or "didnt reload", until such time that these tagged replays are sufficient enough in quantity to train a decent replay file savescum detector, which will then allow us to finally remove the radio buttons...
Possibly, we could include this feature only in the Dev branch of Wesnoth so that the dataset can be produced and then only add it to the Stable branch once the Savescum detector has been trained.

[Pentarctagon]

I'd say that it's probably overkill to use machine learning to create an automatic replay classifier to filter out save scumming.

[Numero]

The Discussion seemed to go a bit out of hand, just wanted to say I like the Idea, it would help make wesnoth easier to get into and campaigns could be improved.

[Spannerbag]

Would mods, if used, have an effect on all this?
Also - and this has just occurred to me, so not thought through - might it be possible to extract the data you need from the replay and not upload the full replay?
No idea if this is possible but I was thinking you might find the following sufficient?

1. Scenario identifier
2. Statistics
3. Turn limit (?)
4. Turns played

An additional advantage is that perhaps this same data could be auto-uploaded when a scenario is lost (if the player gives permission beforehand) so losses could be counted as well?
Finally the data is inherently anonymous (unless there is something in the replay)?

Just my tuppence worth

Cheers!
-- Spannerbag

[Lord-Knightmare]

Would mods, if used, have an effect on all this?

Mods in most cases have had the effect of making scenario content much easier.
Also, modded replays would not load on the umc author's wesnoth environment without having them installed.