[STATUS] Unscheduled Wesnoth.org maintenance downtime

Discussion of all aspects of the website, wiki, and forums, including support requests and new ideas.

Moderators: Forum Moderators, Developers

Locked
User avatar
shadowm
Site Administrator
Posts: 6543
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

[STATUS] Unscheduled Wesnoth.org maintenance downtime

Post by shadowm » February 19th, 2014, 1:41 am

Hello,

As you may be aware, February 17th (two days ago), at some point after 10:00 UTC, all primary wesnoth.org services became unresponsive and the forums and wiki started presenting erratic behavior, including SQL errors. I first became aware of the issue yesterday February 18th, roughly at 00:37 UTC, and proceeded to take measures to reduce system load in order to be able to diagnose the issue, including taking down HTTP and SQL services — this meaning that from that point onwards, the Wesnoth.org website, wiki, forums, and the user authentication services on the primary MP server instances on server.wesnoth.org (master, 1.10, 1.11, 1.8, 1.6) became immediately unavailable and remained so for most of the downtime. Because of timezone differences, contacting other administrators who could confirm the initial diagnosis and take measures to resume server operation and contact our service provider took even longer.

Today February 19th, roughly at 00:00 UTC, the website, forums, and wiki became available again, with only minor consistency issues with the forums database. As of this writing (01:20 UTC), all primary MP server and add-ons server instances are operational as well. However, overall performance will be considerably reduced (but not unusable like before) while we wait for a hard disk drive to be replaced.

It is also for this reason that during the upcoming days, we will have one or two more unscheduled downtimes. If you need to host an event of any sort that requires guaranteed multiplayer server availability, we advise you use either server2.wesnoth.org or server3.wesnoth.org (reachable from the title screen → Multiplayer → Connect to Server) until this issue is sorted out.

I would like to take this opportunity to remind you that you can come check the #wesnoth channel on the freenode IRC network if you notice any issues with the website or related services. @Wesnoth on Twitter may also provide status updates as staff availability permits. Finally, an external site monitoring service implemented by me and hosted by AI is available at http://status.wesnoth.org/ — you may check the status report there if you need to make sure you are not the only one experiencing issues with the site at any given time.

Apologies for the inconvenience, and thanks for your patience.

User avatar
shadowm
Site Administrator
Posts: 6543
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: [STATUS] Unscheduled Wesnoth.org maintenance downtime

Post by shadowm » February 22nd, 2014, 9:25 am

We had another short unscheduled downtime for all services this February 19th, around 10:50 UTC, lasting approximately 12 minutes.

The original issue is now solved and things should be back to normal.

Locked