[STATUS] Unscheduled February 7/8th downtime

Discussion of all aspects of the website, wiki, and forums, including support requests and new ideas.

Moderators: Forum Moderators, Developers

[STATUS] Unscheduled February 7/8th downtime

Postby shadowm » February 9th, 2017, 4:16 pm

Hi all,

I decided to write this to inform the community of the specifics of the situation that took the entirety of Wesnoth.org down for the past two days. I’m doing this of my own volition and intentionally without consulting our Community Manager on the matter.

The hosting company that runs Wesnoth.org’s server took it offline because of a payment issue resulting from an expired credit card issued to our financing entity, Wesnoth Inc. Dave received an email warning him that the credit card info on file was no longer valid and that the site would be taken offline on the third week of February if he did not address this soon. For some reason the hosting company decided to do it on the second week instead.

So there’s an issue of negligence both on the hosting company’s part, and Wesnoth Inc’s. For reasons beyond my comprehension, it appears that despite the board elections held last year there still isn’t a definite appointed treasurer and that there is still a single inactive person in charge of running our finances, and by extension, the legal aspect of hosting Wesnoth.org. It also appears that after one of the originally-elected board members quit, doofus-01 was appointed to the board in a heretofore unspecified role. I’d like to give him my congratulations.

Right now the development team, Wesnoth.org staff, and Wesnoth Inc board appear to have no idea what each other are up to, and as a result more incidents of this nature can’t be ruled out as a possibility in the near future. I’d like to ask you to bear with them while they sort out their organizational structure and decide which communication platform to use for their day-to-day discussions. After 8 years of similar issues taking place on a relatively frequent basis, I’m sure things will go back to ‘normal’ soon — after all, Wesnoth is slated to be published on Steam’s distribution platform as soon as version 1.14.0 is released at some point this year.

But for the time being and for the sake of my own health, I’m going to wait it all out, and see if our Community Manager will eventually remember that there’s more to Wesnoth than a select few members of the mainline development team and the people running its finances.

Regards
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.
I also made Wesnoth RCX, a team-color preview tool for artists and content creators.
Elsewhere: shadowmBlogFollow me on Twitter
User avatar
shadowm
Wesnoth.org Administrator
 
Posts: 6177
Joined: November 14th, 2006, 5:54 pm
Location: Chile

Re: [STATUS] Unscheduled February 7/8th downtime

Postby nuorc » February 9th, 2017, 9:04 pm

Thanks for your post.

I was wondering why I couldn't reach this domain for days, and, when it was back, why I didn't see any explanation.

Spoiler:
I have a cunning plan.
User avatar
nuorc
Forum Regular
 
Posts: 569
Joined: September 3rd, 2009, 2:25 pm
Location: Barag Gor

Re: [STATUS] Unscheduled February 7/8th downtime

Postby Pentarctagon » February 10th, 2017, 12:46 am

Huh. Well that's interesting, I guess. Thanks for the update.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Pentarctagon
Forum Administrator
 
Posts: 2919
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: [STATUS] Unscheduled February 7/8th downtime

Postby doofus-01 » February 10th, 2017, 3:54 am

shadowm wrote:So there’s an issue of negligence both on the hosting company’s part, and Wesnoth Inc’s. For reasons beyond my comprehension, it appears that despite the board elections held last year there still isn’t a definite appointed treasurer and that there is still a single inactive person in charge of running our finances, and by extension, the legal aspect of hosting Wesnoth.org. It also appears that after one of the originally-elected board members quit, doofus-01 was appointed to the board in a heretofore unspecified role. I’d like to give him my congratulations.
Awe shucks... Thank you, shadowm!

I am a little unsure of my role too, but there is some effort by the others in trying to find someone for the phone version.
shadowm wrote:Right now the development team, Wesnoth.org staff, and Wesnoth Inc board appear to have no idea what each other are up to, and as a result more incidents of this nature can’t be ruled out as a possibility in the near future. I’d like to ask you to bear with them while they sort out their organizational structure and decide which communication platform to use for their day-to-day discussions
nuorc wrote:Spoiler:
There's long been an IRC/forum divide, and now we have discord. Twitter and Facebook might fit in somehow, but I don't think it was as much of an issue. We'll see how it goes, I guess. But I think the communications platform issue is easier to survive than the organizational issue. That looks really difficult to solve, as it requires some trust and dedication that are hard to demand of volunteers (to a game). I hope greater minds than mine continue to ponder.
BfW 1.12 supported, but active development only for BfW 1.13/1.14: Bad Moon Rising | Trinity | Archaic Era |
| Abandoned: Tales of the Setting Sun
GitHub link for these projects
User avatar
doofus-01
Art Contributor
 
Posts: 3540
Joined: January 6th, 2008, 9:27 pm
Location: USA ...

Re: [STATUS] Unscheduled February 7/8th downtime

Postby Pentarctagon » February 10th, 2017, 7:03 am

One thing that I'd like to have clarified: What is meant by "Wesnoth.org staff"? I assume it includes the Admin group, but what about Moderators/MP Moderators? Would any/all of the Developer group count towards that? There's been a rather frustrating lack of any communication, with the only real information seemingly coming out as a result of iceiceice resigning. It just seems like very little has been actually happening - even when I wrote up a summary of the Board's IRC logs (which I am more than willing to keep doing, if taking the time to reread the logs and write up a summary is a blocker for the current Board members), there was nearly a month of no discussion at all, and this treasurer/credit card issue doesn't exactly inspire confidence either.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
Pentarctagon
Forum Administrator
 
Posts: 2919
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: [STATUS] Unscheduled February 7/8th downtime

Postby Chewan » February 11th, 2017, 3:18 am

Pentarctagon wrote:There's been a rather frustrating lack of any communication …
It just seems like very little has been actually happening

We're not going to be Hamlet, anymore. — The rest is silence. :D Sticking with the lore is more important at BfW!

Indeed, one would think that nothing special has been going on, though community management is said to involve coordinating announcements for and communication with the community.
Has anyone noticed an improvement since the end of 2015 as to the project’s engagement with the community ?

Pentarctagon wrote:summary of the Board's IRC logs – which I am more than willing to keep doing

That would be great!
Assuming that the obscure »Wesnoth.org staff« gets an idea what each other are up to, and you are permitted to set eyes on those logs…

shadowm wrote:if our Community Manager will eventually remember that there’s more to Wesnoth than a select few members of the mainline development team and the people running its finances.

My call to action: Pentarctagon for community manager! 8)

shadowm wrote:doofus-01 was appointed to the board in a heretofore unspecified role

Oh, that's good news! Our CM somehow forgot to mention it on the forums…
As replacement of the member who resigned, doofus-01 should be the official treasurer of the company:hmm:

doofus-01 wrote:I hope greater minds than mine continue to ponder.

Who would that be? With »the great shadowmaster«, regrettably, deeming his opinion not relevant anymore
Chewan
 
Posts: 129
Joined: December 19th, 2013, 1:40 am

Re: [STATUS] Unscheduled February 7/8th downtime

Postby shadowm » February 11th, 2017, 5:17 am

Pentarctagon wrote:One thing that I'd like to have clarified: What is meant by "Wesnoth.org staff"?

The people who run Wesnoth.org itself. All great people who usually don’t receive the recognition they deserve because from Q3 2009 until last year I’d been serving as their mouthpiece, and before that, it was primarily Ivanovic (our Release Manager from circa 2006 until January 2015) and his predecessors who were in charge of announcing everything involving server downtimes and such.

The current line-up:

  • Rhonda, who runs the main Wesnoth.org host infrastructure. She also usually manages to fix really complicated stuff while the rest of us run in circles like headless chickens (like that time we ended up with an unbootable system after a botched RAID1 rebuild).
  • Soliton, same as above. Also does some wesnothd stuff from time to time, mostly when I’m not around (see below) and also because he was the wesnothd expert before I even joined.
  • shadowm (who is still not shadowmaster no matter how much you insist on it), who did pretty much all of the server migration work back in Q4 2013 and inadvertently (because I didn’t read the fine print) ended up responsible for everything that isn’t listed above or below; that is to say, the web server, the database server, and most of the stuff that runs on the web server (including forums and wiki). Note that I’m by no means irreplaceable, and both Rhonda and Soliton have the skills to take over should something happen to me, that is, if given enough spare time; and have done so as needed on several occasions. I guess you could say I’m the first line of defense during an incident, then Soliton, then Rhonda, in descending order of spare time and ascending order of expertise. That’s right, I’m just your tech support grunt crossing my fingers that I can guide you through a cookie-cutter troubleshooting recipe without having to transfer your call to my seniors, who are more than capable of stir-frying me alive if I make the slightest mistake. Reportedly, I taste like chicken.
  • loonycyborg, who maintains the jenkins CI suite, and as of the past year has also partially replaced me as campaignd (add-ons server) and wesnothd (MP server) VM maintainer since the Git master/1.13 versions of both software now run with his Asio-based network code instead of the legacy network code, and they’ve required a bit of extra testing and patching after his code was first merged into master last year. I still find myself doing most of the user-facing campaignd maintenance work, though, such as tending to support requests regarding lost passphrases and unmaintained add-ons.
  • AI/AI0867, who used to run the mailer setup but is currently completely inactive. Nowadays Rhonda or Soliton or me are in charge of that depending on which one of us gets to it first (i.e. time of the day, day of the week and so on).
  • Elias/allefant, who developed and runs the units.wesnoth.org build infrastructure. He would also really like somebody to replace him and make units.wesnoth.org great and awesome.
  • Kinda sorta zookeeper, who doesn’t have access to anything at all because he’s a Windows scrub.[/sarcasm]
The fact that we are all members of the Developers group is mostly coincidental and only two of us (and zookeeper) are forum admins, and only one of us has actually done forum admin work on a regular basis for the past 8 years (guess who), other than purging spammers and other mundane stuff. The other forum admins, moderators, community, etc. all rely on us keeping Wesnoth.org running. And in turn, we rely on Wesnoth Inc paying for the hosting so that we can have a trusted server to work with for the community’s sake.

Also, I’d like to clarify that the reason that the list of Wesnoth.org staff has remained mostly secret over the years is to keep people from pestering some of us with off-topic stuff. For example, I can deal with someone requesting a username change on the forums, but Rhonda won’t tend to that because it’s not her task and she has far more pressing things to focus on and less time; Soliton and I might run the MP servers, but we don’t deal with moderation incidents because that’s the job of the MP moderators we have appointed. It’s for these reasons that we routinely encourage people to use our official communication channels instead — that is to say, group PMs on the forums, IRC, public posts in the Website section, the forum support email address that will appear any time the forum literally can’t even, and so on.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.
I also made Wesnoth RCX, a team-color preview tool for artists and content creators.
Elsewhere: shadowmBlogFollow me on Twitter
User avatar
shadowm
Wesnoth.org Administrator
 
Posts: 6177
Joined: November 14th, 2006, 5:54 pm
Location: Chile

Re: [STATUS] Unscheduled February 7/8th downtime

Postby vultraz » February 11th, 2017, 10:32 am

I'd like to apologize for not writing a post like this immediately after the restoration of services. I did tweet, but should have thought to make an immediate announcement on the forums.

But after I learned, for example, that new mainline campaigns are announced elsewhere but not here, I already lowered my expectations...

It would have been announced with the release of 1.13.7. However, you may have a point that it should be done here too.

And to address Chewan, no, I have not been as good a community manager as I could be. I apologize. I have tried to keep the community informed of any major change or event that happens, but I will certainly try to interact with people here on the forums more.
Creator of Shadows of Deception (for 1.12) and co-creator of the Era of Chaos (for 1.12/1.13).
SurvivalXtreme rocks!!!
What happens when you get scared half to death...twice?
User avatar
vultraz
Community Manager
 
Posts: 817
Joined: February 7th, 2011, 12:51 pm
Location: Dodging Daleks


Return to Website

Who is online

Users browsing this forum: No registered users and 3 guests