Is number of forum posts a better measure of UMC add-on quality than downloads?

General feedback and discussion of the game.

Moderator: Forum Moderators

Post Reply
stencil
Posts: 28
Joined: July 15th, 2021, 11:33 pm
Location: USA

Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by stencil »

In the time since I posted my reviews a couple of months ago on user-made content (UMC) add-ons I had played, I have been wondering if there is a better way to prospectively evaluate add-ons than the commonly-used ‘Number of downloads.’ This was especially true when I learned that the number of downloads listed in the add-on manager is only for the current release of Wesnoth (I know, I’m slow to just now be learning this). Therefore, I did a little research to see whether other quantifiable features of an add-on might be (slightly) better predictors of an add-on’s quality. I propose that the number of forum posts for an add-on is a slightly better indicator of quality than downloads (though still not highly reliable).
Methodology
My sample of add-ons initially included a selection of those I had personally played (~50%), those mentioned as high-quality online (~30%), and those with a high number of downloads not in the other two groups (~10%). The remaining 10% were those I added after tallying votes for favorites recorded in forum posts.

For each add-on, I recorded the number of downloads from the 1.14 server, the downloads from the 1.12 server, the sum downloads for 1.12 and 1.14, the date of the first post about the add-on on the forums, the ‘age’ of the add-on as measured in days since the first forum post, the total number of forum posts (fairly tricky, since many of the older add-ons have multiple development and feedback threads for different Wesnoth versions), and the number of ‘favorite’ votes an add-on has received in forum polls. For the latter, I used the fifteen different forum posts about add-on quality starting from 2009 that I linked to in my July reviews (plus one from 2012 I missed, so 16 total). One note, for the 2016 poll I used the votes listed in individual posts rather than the sum votes on page 4, since some people separated IftU and AtS in their posts. I realize there are duplicate voters across the years, but I believe they are a minority such that pooling to make a larger sample size outweighs the duplications. Overall there were 466 votes across 78 different add-ons. If I include the 130 other add-ons available on the 1.14 server, the distribution looks like this:
Population scores.jpg
Since I didn’t want to collect stats on all 208 add-ons, I took my existing list and added any add-ons that got more than five votes for favorite across the years (the final 10% I mentioned). This gave me a sample of 49 add-ons. Legend of the Invincibles was a clear outlier in both number of forum posts and number of downloads (almost order of magnitude larger for both), so it was excluded from the correlations, leaving 48, with 370 total votes. (LotI stats are still available in the attached spreadsheet.) I was going to attach the rating, age, downloads, and forum post distributions for this sample, but the max number of attachments is six, so I will just include the score distribution.
Sample scores.jpg
The median age of an add-on was July 29, 2012, so over 9 years old. The median number of forum posts was 168, while the median number of downloads was 13,070. If people are interested in the full dataset, it is attached as an Excel sheet.
The r2 coefficient of determination, or proportion of variance explained, for combined number of downloads from 1.12 & 1.14 vs number of forum votes is 0.15.
Downloads-rating.jpg
In contrast, the r2 value for age of an add-on versus number of forum votes is 0.2.
Age-rating.jpg
This suggests that simply looking at the age of an add-on may give a better idea of its quality than number of downloads (with older ones generally more highly rated). Of course, an important caveat is that this is only for add-ons available in 1.12 or 1.14, which makes sense; if a developer is willing to keep maintaining and improving their add-on for years or even decades, it will probably result in high(er) quality than the ‘average’ add-on.

Finally, the r2 value for number of forum posts versus number of forum votes is 0.35. I interpret this as saying that if people comment more about the add-on in forum posts, it means they like it more (or more people are playing it and therefore there are a proportionally higher number of comments).
Posts-rating.jpg
For reference, the correlations of number of 1.14-exclusive downloads vs forum ratings and 1.12-exclusive downloads vs ratings are very poor, with r2 values of 0.07 and 0.05, respectively.

Finally, if people aren’t interested in these graphs and just want to know, “What are the most highly rated add-ons?”, here is the list of the dozen which received more than 10 votes (in order):

Invasion from the Unknown (47)
After the Storm (34)
To Lands Unknown (31)
Bad Moon Rising (23)
Legend of the Invincibles (22)
Swamplings (21)
Ooze mini-campaign (19)
Sojournings of Grog (15)
A New Order (13)
Fate of a Princess (13)
Elvish Dynasty RPG (13)
Soldier of Wesnoth (11)

This only differs slightly from the 2016 poll, which included Secret of the Ancients (mainlined), Trinity, and Grnk the Mighty, but didn’t include Sojournings of Grog or A New Order. In summary, I didn't see clear very clustering of highly-rated add-ons across downloads, age, or number of forum posts, but I think number of forum posts may generally be slightly more useful than downloads. In a linear model using age, combined number of 1.12&1.14 downloads, and number of forum posts as predictors of rating, only forum posts had a significant p-value (0.002). The adjusted slope for forum posts vs rating (factoring in the effect of age and downloads) was 0.013, meaning that for every 77 extra forum posts, you would expect one additional vote as favorite in forum posts.

Thoughts?
Attachments
BfW UMC statistics.xlsx
(13.74 KiB) Downloaded 51 times
Thank you to everyone who has worked on this game!
User avatar
Aldarisvet
Translator
Posts: 836
Joined: February 23rd, 2015, 2:39 pm
Location: Moscow, Russia

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by Aldarisvet »

Actually we have no idea what people think about UMC stuff. Just one thing about the number of posts.

No bugs - no posts. I have some very positive feedbacks only because people came to point on bugs. Surely they would not came if they didn't meet these bugs, even if they was happy with the stuff.

Another thing - the number of downloads largely depends on how fast you put your campaign on the new version of the game. The same campaign could be a leader in 1,12 and a laggard in 1,14 just because it was published in 1,14 a year after 1,14 was presented.

In general - feedback is a huge problem for Wesnoth because there is no convenient instruments for users to make feedbacks. Most of users will never bother themselves registering in the forum.
facebook.com/wesnothian/ - everyday something new about Wesnoth
My campaign:A Whim of Fate, also see it's prequel Zombies:Introduction
Art thread:Mostly frankenstains
User avatar
Atreides
Posts: 1039
Joined: March 30th, 2019, 10:38 pm
Location: On the 2nd story of the centre village of Merwuerdigliebe turning the lights on and off

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by Atreides »

As for downloads... caveat emptor. I've downloaded a number of add-ons but after finding them not to my liking I deleted them. The download counter does not reflect this.
stencil
Posts: 28
Joined: July 15th, 2021, 11:33 pm
Location: USA

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by stencil »

Thank you for your comments, @Aldarisvet and @Atreides. I appreciate you being active on the forum and providing your perspectives. A couple of my thoughts:

@Aldarisvet, I agree that we don't have a good idea of what most people think of UMC add-ons, since they don't provide any feedback, but I'm not sure I would agree that we "have no idea" at all. Though 466 forum votes is certainly tiny compared to the hundreds of thousands who have played Wesnoth, it's not nothing. It's almost guaranteed to be a biased sample, since the people active enough on forums to vote on favorites probably skew differently than the average player, but I think it can still provide at least some insight into preference. I'm especially indebted to your 2016 poll, which provided so much of the data!

As for downloads, I agree with you both that they are not reliable. That's the whole point I'm trying to make in this post, that if I were looking for 'quality' add-ons, I believe looking at the number of forum posts it has would give me a better indication than just the number of downloads.

It also makes sense to me that forum posts directly correlate with bugs, but there could also be some logic there relating to quality: perhaps add-ons that get more bug reports are trying out more inventive ways to push Wesnoth, which could in turn increase player enjoyment of the add-on. (It could also just reflect an inexperienced programmer [like I would be if I ever tried making something!]. I'm not trying to argue that number of forum posts is a great way to judge add-ons, just potentially a better way than number of downloads. For instance, I recently played the Altaz Mariners and loved it, but it has only received 5 votes in online polls. In fact, my personal ratings were almost inversely correlated with the overall online ratings.)

In general, I believe that the detectable relationship between number of forum posts and number of votes as favorite in online polls is probably a reflection of overall developer engagement. If a developer is committed enough to correct bug reports and continue uploading new versions for each new iteration of Wesnoth, I reason they're going to fix unliked features of their add-on faster than someone who puts it up and neglects it for a few months. I thought about trying to use number of add-on uploads as another proxy for developer engagement, but alas, it has the same drawback as downloads, where it's reset for each new release version. If other people have thoughts on whether number of forum posts is a useful metric, I'm always delighted to read other perspectives!
Thank you to everyone who has worked on this game!
vghetto
Posts: 755
Joined: November 2nd, 2019, 5:12 pm

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by vghetto »

Ah, The Altaz Mariners, hands down my favorite UMC. I personally rate it higher than the rest of the famous ones.

Did you try the Altaz Mariners in its vanilla form or did you apply the patch? The patch is supposed to give a better single player experience by having the player control just one side instead of two. It also fixes a couple of tiny bugs in TAM.
shevegen
Posts: 497
Joined: June 3rd, 2004, 4:35 pm

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by shevegen »

Hmmm. I think that methodology is weird.

I know both Invasion from the Unknown and to Lands Unknown. Both are good; Iris did a great job with the former. But to Lands Unknown is simpler on another higher league compared to every other campaign IMO. Ok I may be biased but ... that's my opinion. I just can not want to assume that number of forum posts may be a really good indicator.

Aldarisvet pointed out the issue with bugs. And that is true - I think perhaps 30% of my comments in the add-ons is due to bugs or suggestions for improvement.

What may be interesting would be a "trending" factor like ... "these 5 campaigns are trending more, within the last 6 months" or something like that.

I make semi-regular breaks (right now I am on a break spree again), then I come back and play with a vengeance again. :D

> If a developer is committed enough to correct bug reports and continue uploading new versions for each new iteration of Wesnoth, I reason they
> re going to fix unliked features of their add-on faster than someone who puts it up and neglects it for a few months.

Not sure about that either. The most epic is probably Legend of the Invincibles. How many posts are there now? +800 or something like that? But, even though it may be the most epic one, Invasion from the Unknown is still IMO the best by far. So I don't think you can say dev activity alone outperforms other factors, even though it may help.

It would be great if new epic campaigns could be done every now and then. Like new ones, new attempts of artwork, new ideas too (although I'd be fine if everything just continues from To Lands Unknown :P - but new ideas keep a game fresh and some of them may be good enough to find integration elsewhere. I love unique hero progression in general for instance.) Like ... every 2 year a new long epic campaign is created. And then another one. And another one.

Evidently that requires effort and coordination and contributors may drop off - we all know that this can be difficult in regards to the effort. But, excluding these details, imagine if there would be regularlyl new epic campaigns ...

I think the blender project has something similar, like they showcase new things every now and well - used to do feature film animations via blender I think. I did not check on them for some time but not that long ago they kept on making animation movies I think? Something that showcases things.

Perhaps that could also be used as means to advertise new features. Some campaign authors do that anyway, but I think "featured campaign" may be nice to have for wesnoth. I guess I am not the only one to regularly check back so that may also be useful to get some idling people back engaged to the game again.
stencil
Posts: 28
Joined: July 15th, 2021, 11:33 pm
Location: USA

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by stencil »

Thanks for your replies, @vghetto and @shevegen. I apologize for my slow response. First, @vghetto, no, I just played it vanilla, but thanks for telling me about the patch! I will give it a try at some point when I return back to TAM.

@shevegen, thanks for your thoughts. I agree that number of forum posts is not perfect, I just thought it might be better than looking at number of downloads. For instance, right now there are three campaigns I haven't played that have 6800-7000 downloads on the 1.14 forum: Return from the Abyss, After Eastern Invasion, and Dawn of Thunder. If I'm wondering what order to try them out in (or which single one to try, if I don't have time to try them all out), I might quickly search for each of them on the forums and see they have 192, 50, and 147 forum posts, respectively. Then, I might decide to try out Return from the Abyss first. I definitely agree it probably will only capture a small amount of variability, but I think it might be useful as a measure for which ones to try out.

And I second your idea for regular 'featured campaigns' of the week or similar. I think the more advertising the better!
Thank you to everyone who has worked on this game!
User avatar
Aldarisvet
Translator
Posts: 836
Joined: February 23rd, 2015, 2:39 pm
Location: Moscow, Russia

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by Aldarisvet »

I just got an Idea. Is there any way to organize add-ons feedback inside Steam? To create a related topic there may be?
It is useless to do this in this forum - most of players would be too lazy to register in it.
facebook.com/wesnothian/ - everyday something new about Wesnoth
My campaign:A Whim of Fate, also see it's prequel Zombies:Introduction
Art thread:Mostly frankenstains
User avatar
egallager
Posts: 568
Joined: November 19th, 2020, 7:27 pm
Location: Concord, New Hampshire
Contact:

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by egallager »

Aldarisvet wrote: October 7th, 2021, 3:17 pm I just got an Idea. Is there any way to organize add-ons feedback inside Steam? To create a related topic there may be?
It is useless to do this in this forum - most of players would be too lazy to register in it.
Since we have bridging between Discord and IRC, I'm wondering if we could also set up some sort of bridging between the Steam forums and here?
User avatar
Aldarisvet
Translator
Posts: 836
Joined: February 23rd, 2015, 2:39 pm
Location: Moscow, Russia

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by Aldarisvet »

I made a topic in Steam.
I hope it will be filled with people's opinions and would be something like a storage of it.
Would be good to make it pinned, well, at least after the topic proves itself to be useful.
facebook.com/wesnothian/ - everyday something new about Wesnoth
My campaign:A Whim of Fate, also see it's prequel Zombies:Introduction
Art thread:Mostly frankenstains
Ptitboul
Posts: 73
Joined: May 8th, 2021, 3:45 pm

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by Ptitboul »

Indeed, for the moment the main improvement I would like in Wesnoth would be to make it easier for the casual player to provide feedback on UMC.

Especially for my campaign, because it can be played in many different ways, and to be able to make a Part II that is balanced I need that palyers that went to the end of Part I can send me their replay of the last scenario (or their final savegame)...
My campaign, Bloodlust, is is need of feedback!
User avatar
egallager
Posts: 568
Joined: November 19th, 2020, 7:27 pm
Location: Concord, New Hampshire
Contact:

Re: Is number of forum posts a better measure of UMC add-on quality than downloads?

Post by egallager »

Ptitboul wrote: December 4th, 2021, 8:18 am Indeed, for the moment the main improvement I would like in Wesnoth would be to make it easier for the casual player to provide feedback on UMC.

Especially for my campaign, because it can be played in many different ways, and to be able to make a Part II that is balanced I need that palyers that went to the end of Part I can send me their replay of the last scenario (or their final savegame)...
Something like Super Mario Maker's in-level commenting system would be cool, IMO...
Post Reply