Maintenance and Service Interruption Log

snoopy369 · June 5, 2008, 21:38

The purpose of this locked thread will be to notify users of any planned maintenance, including planned outages/downtime, in a proactive manner, as well as communicate information about unplanned outages after the fact once our technical staff are aware of the nature of any outage.

We will attempt to give as much notice in advance as possible of planned outages, but as with any circumstance this may prove to not always be possible.

Please 'subscribe' to it if you want to be notified of any planned outage via email in advance (if you have set your thread notification settings to receive emails of subscribed threads).

This thread will be topped for a period in order to allow everyone to see it and subscribe to it, and then likely will be allowed to drop until it is needed for the next (hopefully distant) outage.

Gramphos · June 6, 2008, 07:41

Earlier today a subsystem failure caused a lockup which required a reboot and some additional recovery work to bring the database back online. All should be working now.

Gramphos · June 6, 2008, 16:56

We are currently experiencing some trouble related to the work done yesterday, and are preparing to revert parts of the work for now. We will schedule additional downtime to redo parts of the changes once we have isolated the error.

snoopy369 · June 6, 2008, 21:49

The site seems to be currently stable, and no maintenance is currently in progress. Some maintenance will be required, including at least one reboot, tomorrow morning swedish time. Assuming the reboot ends up as a planned reboot (as opposed to a server crash) we will make a notification here ahead of time.

However, the backup solution still must be placed as planned, so there will need to be a period of sporaic downtime later on (in a few weeks in all likelihood).

snoopy369 · June 7, 2008, 01:20

The site decided it preferred the unscheduled reboot, fortunately gramphos was able to bring it back up relatively quickly. The software that was causing the lock (we believe) was unloaded, so there should hopefully not be any more issues with reboots.

We will be undergoing some backup-related disk/cpu activity, however, that should last a few hours from now. Gramphos is adjusting the cpu priorities to give mySQL priority so this should have a small impact on the forums, but not the significant impact of yesterday. This should continue into the early morning US time.

We apologize for the delays and thank you for your patience!

Solver · June 15, 2008, 18:48

About an hour and a half ago, the site's server went down completely. It was rebooted and Apache was restored quickly enough, however, problems persisted with getting the MySQL server up. Operation has currently been restored and the site is currently stable. Thanks for your continued patience!

Locutus · June 19, 2008, 23:55

Update: We've now opened an off-site service log as well in the form of a Wordpress blog: http://apolyton.wordpress.com.

The purpose of this blog is to keep you guys informed of what's going on during (planned or unplanned) server downtime. By maintaining this blog off-site we can keep the lines of communication open when everything else fails. It does not replace this on-site log but rather supplements it when the forums are unreachable.

Whenever the site is down we will let you know what's happening at the blog as soon as we find out ourselves (which will be in advance of the outage if it's planned, as soon as possible afterwards if not) and keep you up-to-date as we work to bring the site back online.

If you cannot get to the site but see no update on the blog, feel free post a comment in the first post to ask what's going on. If it's an issue on our end we will post a new blog entry to provide details and allow for further discussion as soon as possible, if not we will try to help you figure out the problem if we can.

You can subscribe to the blog's RSS feeds to be automatically notified of updates through your news reader, or use it in combination with a service like RSS FWD to receive email notifications.

Gramphos · June 24, 2008, 11:27

Earlier today the forum went unresponsive as I were doing some work with the database, underestimating the time inserting 7302 posts from backup would take.

On a related note the C4-SPDG now have 7302 posts restored, and post counts accounted for.

Locutus · July 27, 2008, 18:12

As of about an hour ago, Apolyton’s hosting provider started having network connectivity issues. The server itself is up, but there are extended periods where no traffic can get to it. This has been an on-and-off affair (though mostly off) that’s outside of our direct control. Our hosting provider is handling it so we can only hope the issue will be resolved quickly.

Also note that in initially diagnosing the problem, I rebooted the server. Because of this, once connectivity to the site is fully restored you may initially experience some issues (like IRC chat not working). These will be resolved as soon as possible, but while we can’t get to the server we obviously can’t do anything about them.

Our apologies for any inconvenience, we hope it won’t last long.

Locutus · July 27, 2008, 21:09

Just to follow-up on what you probably already knew: the problem seems to have gone away, the site has been continuously reachable pretty much ever since my previous post. All services have been fully restored. If you're still having problems, please let us know.

Gramphos · August 4, 2008, 17:55

I'm about to make some debugging of the Apache server running Apolyton. This will cause some minor interruption as I restart the server at the start and end of the debug session at a minimum.

Gramphos · August 4, 2008, 18:08

All done.

snoopy369 · September 4, 2008, 11:57

Locutus' note from another thread on yesterday's interruption:

Originally Posted by Locutus

Site is (obviously) accessible now, after a reboot this morning, but the problem isn't really solved yet, so it may come back to bite us again. I'm completely occupied with RL stuff now, because of that it may take me a while to track down the cause (and I don't think Gramphos is around ATM).

snoopy369 · September 19, 2008, 17:57

The server was down for a few hours around 2PM-5PM EDT.

Originally Posted by DanQ

SettlerIV stopped responding to HTTP requests, so I requested ThePlanet reboot
it manually.

MYSQL did not come up smoothly after that reboot, and also was required to be rebooted, hence the further delay.

Gramphos · September 20, 2008, 12:00

I'm going to do a few tests to see if I can figure out what happened with the server the other day. This will lead to connectivity and/or database issues for the next 10 minutes.

Gramphos · September 20, 2008, 12:03

The tests are now completed.

Gramphos · November 14, 2008, 16:12

We are currently having issues with parts of our httpd setup. This mainly affects a subset of the apolyton subdomains, but might cause issues with the main domain or other subdomains as we are working on a fix.

Gramphos · November 14, 2008, 16:13

All systems should now be operational again.

snoopy369 · January 3, 2009, 01:32

The site went down for several hours today due to a combination of issues with apache/httpd and mysql. Gramphos was able to restore the database, and we are looking into the cause of the apache instability, which caused the server disruptions earlier today. We apologize for the interruptions and are working to restore service.

Gramphos · January 6, 2009, 02:19

I need to reboot the apolyton server, and will do so at 01:30 EST today (in about 10 minutes). The reboot is expected to take about 10 minutes until all systems are up again.

Gramphos · January 9, 2009, 18:24

We will upgrade our forums to vBulletin 3.8.0 at 2:00 AM January 10. The process is expected to take 30 minutes during which the forums will be closed.

snoopy369 · January 25, 2009, 12:41

There was an unexpected loss of service that lasted from about 4AM ET to 11:25 AM ET. The cause of that loss of service has been determined as a fault in HTTPD, and has been rectified.

We apologize for the inconvenience.

DanQ · February 25, 2009, 22:42

Apolyton's server was down from somewhere between 12AM ET and 9:30AM ET on Tuesday, February 24th (exact start time not known). This was an unscheduled and unexpected server interruption; our apologies for the inconvenience.

No know reason has been found for this downtime, and whatever reason it was, the server corrected itself.

Gramphos · April 10, 2009, 06:58

There was a slight disturbance in the Apolyton mail service this morning. This might have cause some email notifications to not e delivered.

Gramphos · April 10, 2009, 12:08

Tomorrow, Saturday April 11 2009, at approximately 1 a.m. ET the server running Settler IV will be taken down for maintenance. The process is expected to take a couple of hours. During most of the maintenance all services hosted on Settler IV will be unavailable. This includes all hosted sites and all site sections.

During the downtime status updates will be posted at our off site service interruption log: http://apolyton.wordpress.com/

Gramphos · April 11, 2009, 08:01

We are back online and the forums have been upgraded to 3.8.2.