Add-ons client/server support for add-on ratings

[iceiceice]

I agree with AI, it makes way more sense to tie it to a forum account. Frankly the wesnoth forum registration is really easy, and if someone isn't willing to register, I'm completely fine with them not being able to vote.

Additionally, if we actually tie users to an identity, we could conceivably do fancy things in the future like recommend more add-ons that they would like based on add-ons they liked in the past. If they get a new identity every time they go to a coffeeshop that data will all be ruined.

[vultraz]

I do agree it would be best to tie it in with a forum account. It'd probably involve expanding the mp authentication and making it global, however. But where would be the best place to have a user "sign in" to wesnoth then? Also assuming you'd want to game to remember you're signed in.

[Dugi]

As far as I know, many people don't even sign in to multiplayer (remember when ChaosRider complained that somebody created an account with his favourite nickname GrimReaper666 and he could not use it anymore?). It can't be expected that people will bother to sign in to the add-ons server to review stuff, you will just throw away 90% of users' opinions this way.

There are many websites that accept all kinds of reviews and ratings and require no authentication, so they can work only with IP addresses and cookies (neither of these obstacles is hard to cheat). There was only one abuser on the add-on server so far, and his ways were absolutely simple.

It isn't obvious from its behaviour that voting more times from the same IP address doesn't work, I've written that part so that the player can't learn that his rating wasn't taken in consideration by the server because of multiple votes from the server (and that the server announces when more than 10 votes are made from the same account).

Also, why do you think that forcing authentication will get rid of trolls and abusers? There is enough freaks on the multiplayer server who are logged in. There are spammers causing trouble on the wiki. Creating 10 dummy e-mail addresses is easy on less known mail services that don't require phone numbers and DNA samples like gmail, the hardest part is writing down the usernames (in fact, this is sometimes easier to fool than IP checks because not everybody can change his IP address easily).

The new slider-based rating:

[Pentarctagon]

As far as I know, many people don't even sign in to multiplayer (remember when ChaosRider complained that somebody created an account with his favourite nickname GrimReaper666 and he could not use it anymore?). It can't be expected that people will bother to sign in to the add-ons server to review stuff, you will just throw away 90% of users' opinions this way.

Not really, Wesnoth can already save your password to allow automatically logging on. So the amount of feedback lost would probably be fairly small, especially considering that the amount we're getting from this now is 0 anyway.

[iceiceice]

Dugi:

I'm less concerned about trolls / cheaters than I am about actually having one identity per legitimate reviewer for purposes of our data, so that we can tie related reviews to one another. If we think that one person is actually 5 people its going to make it very hard to give them any recommendations, and to give other people recommendations based on the patterns we see.

Spoiler:

One sort of hot topic in the last few years in computer science is how to make good automatic recommendation systems, e.g. like Amazon / Netflix do. It turns out that in most cases a very simple heuristic gives very good results.

1.) Have everyone rank on a numerical score e.g. 1-5 for instance.
2.) Make a table (matrix) of the results with e.g. rows being people, columns being items. Map the numbers so that neutral corresponds to 0. So in the case of 1-5, 3 would go to 0, 5 would map to 2, 1 would map to -2, etc. If someone didn't rank an item it is considered to be 0.
3.) Take a low rank approximation of the matrix. That is, you pick some small number r, and find the matrix of rank <= r which is nearest to A in Frobenius norm. There are well-known and extremely fast algorithms to do this, even for enormous databases.
4.) Look at the approximating matrix, and for each person, make recommendations based on the corresponding row there, and the scores it gives to them for each item they haven't ranked yet.

It's a machine learning technique -- we are assuming that peoples' preferences are actually best explained by only a few "dimensions" or "factors". The low rank approximation essentially is deducing "vectors" corresponding to these factors by pointing them in the direction that explains the data with the least amount of overall noise.

There's nothing that says that you can't use a more complicated nonlinear learning approach or something, but it turns out that in practice this tends to work very well for ranking systems, which is fortunate because it is relatively speaking quite simple. We would probably have to play with the actual rank number a bit to get the best results, but this shouldn't be too hard.

If each "person" is actually scattered over a many IP addresses then this strategy is pretty much doomed, which would be unfortunate. It might be that you can look at the IP data and somehow conclude that actually most people only use one IP in our userbase -- I really have no idea. But it's potentially quite valuable for us and for our users if we can keep good data and use it to make good recommendations.

[Paulomat4]

Well then, how about a compromise? What about allowing both? Users get the choice to login or register with an account, or they can chose not to do so. If Users chose to log in with an Account, they get proppsitions. If they chose not to, they can vote too, but their IP is stored to prevent trolls, cheaters, etc...

Also for the slider, i think it would be nice to have plus and minus button on the sides, who increase or decrease by steps of 0, 5 or 1. It frequently happens to me, that I want to grade with an roumd number, but allways miss the number by 0,1 or 0,2.

[Dugi]

@Pentarctagon
But you'll still have to make up a login and enter a password for the first time. Read my reply to Paulomat4's first idea, it's related to this.

@iceiceice
I thought your main concern were tricky trolls and intelligent abusers, but this makes sense. People who chose this also chose blablabla is good automated way to give recommendations.

There is a technical difficulty related to that, the multiplayer server, the add-on server and the forum server and the wiki server are standalone programs, and most of them run on different machines on different geographical locations. Making the add-on server use forum accounts as many suggested would be quite a serious problem. Making the add-on server have its own log ins and passwords would be much easier (log in and password would be demanded in the add-on server connect window, and stored locally).

@Paulomat4
I like the idea of allowing also connecting without log in and password, maybe with their votes counting less or something. Or, alternatively, the server might generate a nickname (based on the name generator for units) and password for the player and send it to him making the client remember it when logging in without a username.

You can avoid that problem with arrow keys on the keyboard, if a slider doesn't have enough positions, you can just set its position approximately with the mouse and adjust the exact position with arrow keys.

[iceiceice]

Dugi: I see. Well I don't see a particular reason to complicate your task. I'm probably getting ahead of myself... perhaps we can revisit what I was saying if we ever get around to implementing automatic recommendations.

[Dugi]

If the add-on server had its own account system like multiplayer, forums of wiki, making the automated recommendations system would be no hard task (the hard thing for me is actually interaction with the game's functions, objects and structures I am not very familiar with, not algorithms themselves).

[GbDorn]

So after reading the whole Players' Reviews thread (lots of OT in there...) and this one, here are some proposals:

1. Instead of a single mark or yes/no flag-of-approval, have several small flags/icons: art, story, gameplay, ... because as already said users do not play campaigns with the same end goal, eg pyrophorus said: "For instance, I'm personally very interested in stories and writing, and much less in strategy and classical fighting". This allows players to quickly sort UMC without having to look at the description of each "highest rated" UMC to see if it matches their likes. The results may be sorted, but people should have the same incentive to inspect the first element than the last one. You want users to try and play as much UMC as possible, not them selecting the first one on the list.

2. Those flags would be composite marks of subcategories, eg the "Art flag" would be awarded for high enough marks in 'units', 'portraits' and 'music' (the latter is absent even in Player's Review categories by the way). Here are some ideas for those (most are already on the Player's Review page):
- gameplay mechanics
- story
- prose
- scenario design
- map design
- unit art
- portrait art
- originality
- fun
- replay value

3. About the range of the rating scale: I think Dugi was right in the other topic about numbers having different meanings to people. But how is a [0-10] scale with decimals an improvement? People can't discriminate on a 100-points scale. What's the difference between 6.2 and 6.3 really? (don't tell me 0.1 ...) There is also a time factor. You are not going to rate 100 items in a row, where you can have some consistency in your marks. Here the average user will rate 1 or 2 UMC per week max: "Ok I want to rate this addon I've just finished. Honestly the one I played last week was better. How much did I give it? Was it 6.2, ... or 5.4?" (Of course if you log the IP, you could also help the user 'remember' his past marks, but this adds another complexity layer.)

Wikipedia put it better than I: "Qualitative description of categories improve the usefulness of a rating scale. For example, if only the points 1-10 are given without description, some people may select 10 rarely, whereas others may select the category often. If, instead, "10" is described as "near flawless", the category is more likely to mean the same thing to different people. This applies to all categories, not just the extreme points."

Labels would be much better. And I don't think one need more than 7 marks:
- near flawless
- pretty good
- slightly above average
- average
- slightly below average
- pretty bad
- catastrophic
Then associate hidden numerical values to each so you get averages.

4. The last thing is the UI. I agree with the slider idea (except with labels instead of numbers) when rating. When displaying scores I would like having some information about the variance (so you need 7 counters for each label and category to compute it). We could show the 2nd to 9th deciles on the scale for example (so we get rid of outliers).

Conclusion: do not show the absolute score, just use visual clues to suggest it.

[Dugi]

The change is already coded and is currently reviewed and corrected so that it could be merged with the main code repository. So it's quite late to suggest a total overhaul, small changes or adding features is fine, but please no do this in a totally different way.

Replying to your ideas:
1. Any kind of yes/no rating implies a threshold. Anybody slightly bellow the threshold would feel offended by those who rate it. If it is a numerical rating, the difference between 45% and 55% is only 10%, if it is a yes/no rating, the difference is always 100% or none.
2. There is a reviews feature that allows submitting reviews that describe these categories by words. You can rate that the art looks good, but lacks animations (this is what I don't like in many cases), you can rate that it looks average (this is what some other people dislike), but has decent animations, you can rate that it looks lame and has no animations either. Any yes/no approval in sprite art would be inaccurate and subjective here, words are needed.
3. There is no real difference between 6.2 and 6.3. But... assuming that you can't rate in decimals, if you see that the averaged rating for add-on A is 7.7 and the averaged rating for add-on B is 7.3 and you know that add-on C is somewhat worse than A and somewhat better than B, would you give it 7 or 8 points?
If you had only 7 possible rates, you'd end up with 100 add-ons with the same rating, some of them would be much worse than others. Furthermore, this would not support averaging and making the download count and the time players spend playing the add-on affect it.
4. Showing 7 numbers instead of average would conceal the relevant information in irrelevant information (Orwell-style information hiding being changed to Huxley-style information hiding), why do you need a chart if you can simply read a single number?

In short:
The basic idea is that some kind of numerical rating appears in the list of add-ons to quickly separate the good from the bad and then you read the reviews to find what you really want. It is based on the needs of the user. Some users just pick the first one on the list (so Aaron's Maps or the first uploaded add-on is to have an advantage). So the first add-on should be something good. Others should be able to find a good add-on that suits their demands, so they should not be drowned in an ocean of irrelevance here. They have the reviews, but who would read reviews for 300 add-ons to find one they want? They should read early the reviews for the add-ons they are the most likely to choose, and that means the most popular ones. This implies that an absolute numerical rating has to be there, even if it's not the way to determine the ultimate add-on. Then they can proceed to the reviews and choose what fits their needs, and probably they won't have to read more than ten reviews.

[GbDorn]

Anybody slightly bellow the threshold would feel offended by those who rate it

Or it would incite them to improve their addons.

There is a reviews feature that allows submitting reviews that describe these categories by words. You can rate that the art looks good, but lacks animations (this is what I don't like in many cases), you can rate that it looks average (this is what some other people dislike), but has decent animations, you can rate that it looks lame and has no animations either. Any yes/no approval in sprite art would be inaccurate and subjective here, words are needed.

This is exactly the purpose of my scales with labels. The trick is to cover all relevant categories. In your examples one doesn't need a lot of words to convey the idea. I just fear review pages will get cluttered with "simple" reviews that could have been done just as well with generic labels.

if you see that the averaged rating for add-on A is 7.7 and the averaged rating for add-on B is 7.3 and you know that add-on C is somewhat worse than A and somewhat better than B, would you give it 7 or 8 points?

Depends how I have rated A and B.

If you had only 7 possible rates, you'd end up with 100 add-ons with the same rating, some of them would be much worse than others. Furthermore, this would not support averaging

Sure it would. You associate hidden numerical values to each label. And the averages would not be limited to the 7 discrete rates, they would be continuous.

Showing 7 numbers instead of average

I didn't mean showing a histogram. I meant you have to store the 7 counters to compute the average and quantiles.

Others should be able to find a good add-on that suits their demands

I still think their demands would be better served with some categorization.

[Sapient]

For whatever it's worth, I pretty much agree with GbDorn.

[Dugi]

GbDorn, you aren't even trying to keep a fair discussion. You have skipped some of my points, and I have a strong suspicion that it's because you had nothing to say about it.

Your conclusion was removing an absolute numerical score. I have told you why it should be there, and it wasn't my idea, it was a suggestion of some developers (I can't remember which ones, it was on IRC and I can't remember when). I find it reasonable, and clearly you do too because you didn't even try to tell something against it.

Now specifically about your replies:

Or it would incite them to improve their addons.

Have you ever made an add-on? Do you know how it feels when somebody tells you that some of its parts you were quite proud of sucks? Do you know how it feels when you do your best somewhere and still somebody tells you that it sucks? If the outcome is only yes or no, or it's awesome or it really sucks, you get 0% instead of 40%, that is much more painful.
Furthermore, if somebody gets 60%, he can still improve it to 70%, but if he gets 100%, he isn't very motivated to improve it (and many add-ons would end up slightly above the 50% threshold in as many categories as possible).

This is exactly the purpose of my scales with labels. The trick is to cover all relevant categories. In your examples one doesn't need a lot of words to convey the idea. I just fear review pages will get cluttered with "simple" reviews that could have been done just as well with generic labels.

How many categories are you going to create? Three hundreds, so that nobody would find the relevant information there?
Sprite art needs to be divided into animations and base units. Base units can be taken from other add-ons, they can be frankensteins, they can be some sort of weird art that looks good but doesn't fit wesnoth or it can be the usual pixel art. Some people dislike frankensteins and non-pixel art, and other people care only if it looks good and don't care about unimportant details like origin. Some people think that no animation is better than a simple animation, and would tend to rate absent animations better than simple animations. So we need seven categories for sprite art.
Story is another complicated affair. If it is original and awesome, it's clear that it deserves a good score. But other story properties can be bad for some people and irrelevant for others. Players bored of loyalists versus orcs and/or undead campaigns will call such campaigns' stories repetitive, others will enjoy the story twists. Some people like short campaigns with a fast story, other people like long campaigns with a complex plot. Some people will find campaigns with too many battle cries like 'We fight for our families, we defend our homeland, have no fear because if you fall, you fall for a good thing!' pathetic and bad, others will find it beautifully pompous. There are tons of similar differences, and each of them would need its own category, even if these categories would be mostly useless for eras and other non-campaigns (an era can have a backstory, but you don't demand as many details from it as from a campaign's story).
Gameplay is another set of categories, it can be well balanced but absolutely the same as 70% of mainline stories, it can be not quite well balanced but otherwise very interesting (only one defeat all enemy leaders objective in the entire campaign), it can be interesting because you are discovering a custom faction but only because of that et cetera.
You're failing to understand the complexity of add-ons that can't be covered by soulless numbers, even if you name them differently than one two three four five six seven. A review can pick the most important points and write about them, that will inform the player what is it about and if it is what he seeks.

The actual reasons why is the review split into categories:
1. To enforce a certain structure
2. Because there is no multiline text editor widget in wesnoth's GUI system

This is what a review is supposed to be about. Reviews like this is awesome, this is not so awesome, this is still good and this is its weak spot are unwanted, comments like 'above average' belong only where there's not much to say.

Depends how I have rated A and B.

You're just avoiding the question. Let's say that you rated A with score 7 and B with score 8. If C is something between, will you give it 7 or 8?
Furthermore, most people will try to be consistent with the rating seen before, and the rating there is more or less regulated by numbers coming from the download count and number of hours spent playing it.

I didn't mean showing a histogram. I meant you have to store the 7 counters to compute the average and quantiles.

The meaning of the quantile number will be really obscure and would mean only how certain are the people about the add-on's quality. A rating of 5 ± 2 points will mean really a lot.

I still think their demands would be better served with some categorization.

I am not against categorisation, I am against your way to implement it because of the threshold problem and other issues.

[GbDorn]

You have skipped some of my points

I think I have answered all your points in 1-4. Your last paragraph, I synthesized because it all relies on the presence/absence of generic categories.

Your conclusion was removing an absolute numerical score. [...] I find it reasonable, and clearly you do too because you didn't even try to tell something against it.

Some parts of my post (notably my conclusion) were excessively unclear. Sorry about that.

I'm against a single global score. If I'm only interested in addons with good story, I don't care about addons with poor story but with a higher score because they have better graphics. The global score would be useless but some generic categories would be helpful.

Generic labels/categories and reviews are complementary. I never suggested to replace the latter with the former.
Most people don't have the time, will, or writing skills to write lengthy reviews as you can find in Players' Reviews. For them generic labels and categories are enough.
And then you have people speaking different languages. I imagine that in your implementation the server logs the language and only displays reviews written in the language you have set in your preferences? I hope it is the case... But even this leads to some fragmentation of the community. In your proposal the only info you get from people with a foreign language is the global score.
My proposal is a middle-ground: with generic labels and categories your ratings are visible by everyone; and there are reviews for the details. You can even have a 'global quality of addon' as a category which would be the same as your single score. The users would be able to choose which category they want to use for sorting. If the generic categories don't cut it for you because you only want addons without frankensteins, you'll have to resort to reviews anyway.

Have you ever made an add-on? Do you know how it feels when somebody tells you that some of its parts you were quite proud of sucks? Do you know how it feels when you do your best somewhere and still somebody tells you that it sucks?

I don't need to have made an addon to know this. But how is this different from reading bad reviews?

If the outcome is only yes or no, or it's awesome or it really sucks, you get 0% instead of 40%, that is much more painful.
Furthermore, if somebody gets 60%, he can still improve it to 70%, but if he gets 100%, he isn't very motivated to improve it (and many add-ons would end up slightly above the 50% threshold in as many categories as possible).

A threshold is unfair with a 100-points scale. If I missed the threshold because someone rated my addon 6.6 instead of 6.7, I would be offended because this is just insignificant at the user level. On the other hand the difference between "slightly above average" and "pretty good" is quite significant at the user level so the threshold effect is less dependent on such a fine accuracy at the population level. And in any case I think the threshold (if any) should be let to the user's choice.

Depends how I have rated A and B.

You're just avoiding the question. Let's say that you rated A with score 7 and B with score 8. If C is something between, will you give it 7 or 8?

I guess I'll round it to whichever is closest. Or maybe I'll use the lowest score, hoping the author will improve his creation (but he'll have less work to do to earn an 8 than the guy with a pure 7).
Ratings have a dual purpose IMO. They give as much info to the user as to the author.

The meaning of the quantile number will be really obscure and would mean only how certain are the people about the add-on's quality. A rating of 5 ± 2 points will mean really a lot.

Use of quantiles would indicate whether people agree or disagree on the score.

I am not against categorisation, I am against your way to implement it because of the threshold problem and other issues.

I hope my clarifications have helped then.