Maintenance and Service Interruption Log

[snoopy369]

The purpose of this locked thread will be to notify users of any planned maintenance, including planned outages/downtime, in a proactive manner, as well as communicate information about unplanned outages after the fact once our technical staff are aware of the nature of any outage.

We will attempt to give as much notice in advance as possible of planned outages, but as with any circumstance this may prove to not always be possible.

Please 'subscribe' to it if you want to be notified of any planned outage via email in advance (if you have set your thread notification settings to receive emails of subscribed threads).

This thread will be topped for a period in order to allow everyone to see it and subscribe to it, and then likely will be allowed to drop until it is needed for the next (hopefully distant) outage.

[Gramphos]

Earlier today a subsystem failure caused a lockup which required a reboot and some additional recovery work to bring the database back online. All should be working now.

[Gramphos]

We are currently experiencing some trouble related to the work done yesterday, and are preparing to revert parts of the work for now. We will schedule additional downtime to redo parts of the changes once we have isolated the error.

[snoopy369]

The site seems to be currently stable, and no maintenance is currently in progress. Some maintenance will be required, including at least one reboot, tomorrow morning swedish time. Assuming the reboot ends up as a planned reboot (as opposed to a server crash) we will make a notification here ahead of time.

However, the backup solution still must be placed as planned, so there will need to be a period of sporaic downtime later on (in a few weeks in all likelihood).

[snoopy369]

The site decided it preferred the unscheduled reboot, fortunately gramphos was able to bring it back up relatively quickly. The software that was causing the lock (we believe) was unloaded, so there should hopefully not be any more issues with reboots.

We will be undergoing some backup-related disk/cpu activity, however, that should last a few hours from now. Gramphos is adjusting the cpu priorities to give mySQL priority so this should have a small impact on the forums, but not the significant impact of yesterday. This should continue into the early morning US time.

We apologize for the delays and thank you for your patience!

[Solver]

About an hour and a half ago, the site's server went down completely. It was rebooted and Apache was restored quickly enough, however, problems persisted with getting the MySQL server up. Operation has currently been restored and the site is currently stable. Thanks for your continued patience!

[Locutus]

Update: We've now opened an off-site service log as well in the form of a Wordpress blog: http://apolyton.wordpress.com.

The purpose of this blog is to keep you guys informed of what's going on during (planned or unplanned) server downtime. By maintaining this blog off-site we can keep the lines of communication open when everything else fails. It does not replace this on-site log but rather supplements it when the forums are unreachable.

Whenever the site is down we will let you know what's happening at the blog as soon as we find out ourselves (which will be in advance of the outage if it's planned, as soon as possible afterwards if not) and keep you up-to-date as we work to bring the site back online.

If you cannot get to the site but see no update on the blog, feel free post a comment in the first post to ask what's going on. If it's an issue on our end we will post a new blog entry to provide details and allow for further discussion as soon as possible, if not we will try to help you figure out the problem if we can.

You can subscribe to the blog's RSS feeds to be automatically notified of updates through your news reader, or use it in combination with a service like RSS FWD to receive email notifications.

[Gramphos]

Earlier today the forum went unresponsive as I were doing some work with the database, underestimating the time inserting 7302 posts from backup would take.

On a related note the C4-SPDG now have 7302 posts restored, and post counts accounted for.

[Locutus]

As of about an hour ago, Apolyton’s hosting provider started having network connectivity issues. The server itself is up, but there are extended periods where no traffic can get to it. This has been an on-and-off affair (though mostly off) that’s outside of our direct control. Our hosting provider is handling it so we can only hope the issue will be resolved quickly.

Also note that in initially diagnosing the problem, I rebooted the server. Because of this, once connectivity to the site is fully restored you may initially experience some issues (like IRC chat not working). These will be resolved as soon as possible, but while we can’t get to the server we obviously can’t do anything about them.

Our apologies for any inconvenience, we hope it won’t last long.

[Locutus]

Just to follow-up on what you probably already knew: the problem seems to have gone away, the site has been continuously reachable pretty much ever since my previous post. All services have been fully restored. If you're still having problems, please let us know.

[Gramphos]

I'm about to make some debugging of the Apache server running Apolyton. This will cause some minor interruption as I restart the server at the start and end of the debug session at a minimum.

[Gramphos]

All done.

[snoopy369]

Locutus' note from another thread on yesterday's interruption:
[q="Locutus"]
Site is (obviously) accessible now, after a reboot this morning, but the problem isn't really solved yet, so it may come back to bite us again. I'm completely occupied with RL stuff now, because of that it may take me a while to track down the cause (and I don't think Gramphos is around ATM).[/q]

[snoopy369]

The server was down for a few hours around 2PM-5PM EDT.

[q="DanQ"]SettlerIV stopped responding to HTTP requests, so I requested ThePlanet reboot
it manually. [/q]
MYSQL did not come up smoothly after that reboot, and also was required to be rebooted, hence the further delay.

[Gramphos]

I'm going to do a few tests to see if I can figure out what happened with the server the other day. This will lead to connectivity and/or database issues for the next 10 minutes.