Announcement

Collapse

Downtime Report: Wednesday Night

We had an abrupt and unannounced downtime for all games and services starting sometime after 9pm PT on Wednesday evening. This was due to a massive power outage at our cloud computing facility of the sort that redundant systems are supposed to ensure never happen. After about an hour they were able to diagnose that the power was out. It took a few more hours for them to start restoring service.

At this moment, everything is back up except for Ironclaw and TEC. At the rate they're restoring things, I hope those will be back up in the next 15-30 minutes, but it's entirely in their hands.

As is typically the case with a sudden reboot, TEC and GR have to be rolled back. This is already done with GR. The TEC rollback should be just a few hours due to the more recent backups recently instituted, but I need to get the machine back before I can verify that.

Update: Ironclaw and TEC are now back too. The TEC rollback was 2.5 hours, from 9.30pm PT (when everything crashed) to 7.00 pm PT.
See more
See less

Server Maintenance

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Server Maintenance

    Thursday, December the 3rd at around 1 p.m. the server will be down for approximately 20 to 30 minutes for routine maintenance.
    Game Master Sceadu

  • #2
    We are tentatively scheduling some maintenance following the 3pm backup on Wednesday (June 8th) that should last 30 to 60 minutes.
    Game Master Sceadu

    Comment


    • #3
      On Monday, December 19th, we will be shutting the server down for approximately one hour at 3pm EST for routine maintenance.
      Game Master Sceadu

      Comment


      • #4
        On Friday, March 17th at 4pm EST the server will be taken down for approximately an hour for routine maintenance.
        Game Master Sceadu

        Comment


        • #5
          It's that time again. On Wednesday, July 5th at 4pm EST the server will be taken down for approximately an hour for routine maintenance.
          Game Master Sceadu

          Comment


          • #6
            We've just been alerted that a migration is required for the TEC machine due to an issue with the core OS. Therefor TEC will come down immediately following the 4pm EST backup on Tuesday the 26th and should be down for approximately 2 hours.
            Game Master Sceadu

            Comment


            • #7
              Skotosí ISP will be doing some security updates. In terms of TEC, this means a shut down for a short time Monday evening beginning at approximately 6:50pm EST for the TEC game server and the Skotos website at approximately 7:50pm. Downtimes are expected to last 30 to 40 minutes if all goes smoothly.
              Game Master Sceadu

              Comment


              • #8
                We've been alerted that there will be router maintenance on the two sides of our network on April 4th, then 6th, from 1pm-7pm PT each day. Our colo says that they "do not expect any downtime during this maintenance as we have full router redundancy. However, brief periods of packet loss or increased latency may be observed at times."
                Game Master Sceadu

                Comment


                • #9
                  We've been alerted that there will be router maintenance on the two sides of our network on April 4th, then 6th, from 1pm-7pm PT each day. Our colo says that they "do not expect any downtime during this maintenance as we have full router redundancy. However, brief periods of packet loss or increased latency may be observed at times."
                  They just rescheduled these to be midnight to 6am PT on these two days, so much less likely to have any any impact on us.
                  Game Master Sceadu

                  Comment


                  • #10
                    Instead of trying to summarize, I'll give it straight from the horse's (Shannon's) mouth:

                    There is a problem with the TEC hardware that is going to cause a forced reboot in a bit under a week.

                    Fortunately, I can get ahead of that by rebooting the machine on my own, and this actually dovetails with my desire to do some work on the TEC machine.

                    I'd like to schedule this for Thursday, from 9am-1pm PT. This is a multistage upgrade, which means that the machine will go up and down several times during the period. I will do my best to make sure that each shutdown is clean, but we should warn players that the game will be unreliable during that time. There is also the chance for a rollback if any of the upgrades has an unexpected result. If there are any problems with this timing let me know, else please let players know about the scheduled downtime.

                    Here's my list for the downtime:

                    1. Upgrade machine disk space.

                    2. Resize disk space.

                    3. Resize machine type.

                    4. Upgrade machine to Squeeze.
                    Game Master Sceadu

                    Comment

                    Working...
                    X