Announcement

Collapse

Grendel's Revenge Free Accounts

On December 15th of 2012, Grendel's Revenge enabled free accounts for play. This announcement is following in the suit of TEC's own free account announcement, serving as a reminder to any new or returning players that are browsing the forums.

For all the details about what free accounts are, look no further than here: http://forum.skotos.net/showthread.php?t=95655

In order to log into GR for free, simply visit this page to sign into your Skotos account (ignoring any subscription notices it may give): https://www.skotos.net/user/login.php

Then click the following link for Zealotry/Firefox: zealotry:@GrendelsRevenge.skotos.net...login/Zealotry

Or this link for Alice/Internet Explorer: http://grendelsrevenge.skotos.net/mo...in/alice.shtml

Problems? Questions? You can email the GR staff group at grendel.staff@gmail.com
See more
See less

GR down until further notice [GR Up]

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • GR down until further notice [GR Up]

    From the announcement on the forum homepage:
    Co-Lo Stability Problems

    Unfortunately, our co-lo had some stability problems Friday morning.

    First up, they lost all of their connectivity to the facility that holds all of our games. This took them about 40 minutes to diagnose and a bit longer to restore.

    Much more astonishingly, they rebooted all of their machines (which means all of our machines) while they were fixing the problem, with no notice. This probably had minimal repercussions on the SkotOS games (Marrach, Ironclaw, Lazarus, and Lovecraft). There may have been a rollback of an hour or two.

    TEC and Grendel's Revenge are more troublesome. TEC will suffer a rollback of 10 hours, to midnight (PT) last night, since the unannounced reboot happened one hour before the next backups. GR suffered a more complete failure. We're being forced to restore the entire machine from its backups, then we'll need to restore the game from the backups stored within that backup. The lineup of dates look promising (which is how we laid them out, 'lo these many years ago), so the hope is that we'll only suffer a 7 hour rollback, to the 3am (PT) game backup, but it's a bit dicey at the moment.

    Apologies for the inconvenience; we're now making sure everything is stable, and we're looking into whether our most recent upgrades to TEC will support increasing the backups to every 6 hours.

    Edit: The GR problem is proving even more intractable than hoped. As best I can tell, our colo is still having some issues because I can't boot the first restore I made of GR and I can't build a new node either. I expect this will just clear when they get their act together, but in the meantime I'm not able to get GR back online until they fix these problems.
    So, apparently GR is down until further notice. A bit of bad timing for me, as I'd just renewed my alt account, but hopefully it will be up before long.

    What are people's thoughts on the outage? It seems like there's at least one active clan, so any outage time at all stinks... but maybe if more people start actively playing right after the outage it'll be a blessing in disguise? I seem to remember similar things happening once or twice in the past... What are people's thoughts?
    Serenity is passion.
    Knowledge is power.

  • #2
    Just to offer an update: yes, still down. Our backup service for the machine is still not accessible, and we need it to rebuild the machine. This morning, our co-lo FINALLY started looking into the problem. Last update, which said still looking, was about an hour ago.

    Comment


    • #3
      Originally posted by Cyrria the Lich View Post
      Today, 04:17 PM
      Originally posted by ShannonA View Post
      Today, 04.19 PM
      Well, we certainly know that you're paying close attention. Thanks for the update!

      Serenity is passion.
      Knowledge is power.

      Comment


      • #4
        Still Waiting
        SMARMY the GIANT

        Comment


        • #5
          "What are people's thoughts on the outage? It seems like there's at least one active clan, so any outage time at all stinks... but maybe if more people start actively playing right after the outage it'll be a blessing in disguise? I seem to remember similar things happening once or twice in the past... What are people's thoughts?"

          To OP. Tbh I am a bit miffed. The last time I was around and something like this happened years ago was just before GR became a "lightly supported" game. There was a huge tarfu event and the game was down for several days. When the game was brought back, the roll back wasn't hours, but weeks. As a result I lost three characters, including a tier 4 that was two levels away from t5. I was so upset I rage quit until about 2 weeks ago when I came back to try and work on getting a t6 wm and join Scoundrels lair for some rp. Its been fun being back and getting to know some old and new monsters and their players. It would be nice to see an influx of people once GR is back up, I certainly hope that is the case. I would however be sad to see more people jump ship like I did a few years ago. Guess either way I'm gonna stay playing. It would be super nice if some kind of compensation were offered to current players for the loss of time and favor. Doesn't have to be huge compensation, something as simple a magic ring, armor, weapon would make me happy.

          Anknut the hopefully still Dwerger

          P.S.
          It would be nice to get some kind of update on the progress or lack there of re the game.

          Comment


          • #6
            Originally posted by cjmccoy View Post
            It would be nice to see an influx of people once GR is back up, I certainly hope that is the case. I would however be sad to see more people jump ship like I did a few years ago. Guess either way I'm gonna stay playing. It would be super nice if some kind of compensation were offered to current players for the loss of time and favor. Doesn't have to be huge compensation, something as simple a magic ring, armor, weapon would make me happy.

            Anknut the hopefully still Dwerger

            P.S.
            It would be nice to get some kind of update on the progress or lack there of re the game.
            Glad you won't jump ship. I was thinking about how this happened before when you were playing (and possibly teasing you of being jinxed). A compensation is definitely called for here. BUT, no magic rings! There are SO many magic rings in the game that we could build a mountain with them. Other supreme magic armor or something else special I am all for.

            I also want more frequent updates on progress. After all, I am a GR junkie.... I needs my GR!!
            SMARMY the GIANT

            Comment


            • #7
              Agreed, some kind of compensation would be welcome, but no rings...way to many rings. Other kinds of supreme magic armor or weapons would be nice.

              Comment


              • #8
                There's no particular date when the maintenance will be over?

                Comment


                • #9
                  This problem has nothing to do with Grendel's Revenge being a lightly supported game (though, indeed, it is). This has to do with problems of a jaw-dropping gravity and of a very frustrating longevity at our collocation facility.

                  Here's the full synopsis:

                  Friday morning, around 10am, our colo started having networking problems at their Fremont, California facility where all of our games are located.

                  Around 11am, while fixing the problem, they somehow managed to reboot every machine in their facility, which is a catastrophe at a hosting facility. We had this happen once at our old facility and now once here. So, twice in about 15 years, which is frankly twice more than an unannounced, accidental reboot should happen, since colo facilities are supposed to have multiple redundancies to ensure this never occurs.

                  The Genesis games do not respond well to reboots. The current data is always corrupted. Fortunately, we maintain backup data from 0-12 hours old on the same machine, and we can usually restore from that. That's what we did for TEC. Unfortunately, there was some deeper problem with Grendel's Revenge. Unannounced reboots can cause widespread data corruption on a hard drive, so I just went out to our backup. This has always been 100% reliable at our colo facility: we have never failed to cleanly restore from a backup, and we've had to perhaps half-a-dozen times in the last several years.

                  I rebuilt from the ~5am Friday backup, which should contain the same game backup files as had been on the hard drive, and thus should have just been the same <12 hour rollback which is standard for a sudden reboot. Unfortunately, that backup would not reboot.

                  I contact our colo and they took a full day to get back to me, which is also unheard of, but was apparently due to their creating such a widespread disaster with their sudden reboot, leaving them totally inundated with calls. They said they were working on it, and I requeried late Saturday night and got the same result.

                  This morning, just as I was composing a new letter, I got the latest response from them: their backup team wanted me to give things a try. For the first time in almost three days I had access to our backups again. I discovered:
                  • The Friday morning backup still results in an unbootable file system, and this time I was able to go deeper into it and discovered that only a fraction of our files are accessible.
                  • Two other backups I tried (one from nine days ago and one from two years ago) just won't restore.

                  I reported the continuing problems to our colo and am awaiting more responses. We are keeping on top of this as much as possible, but unfortunately at the moment we're dependent on our colo actually giving us a backup that works.

                  The biggest problem (and biggest danger) is that I currently don't have the Grendel's Revenge game. You know, the actual code. Any backup could give that me, but I need one of them.

                  If I get that in our most recent backup from Friday morning, no problem, things are exactly as I expected. If they can't deliver that backup, but can deliver any other backup that we have (there are three more), then I can merge that with a copy of a GR data file that I keep backed up over on another machine that keeps backups of local data files. These are slightly out of date; they're our last ditch emergencies. But I have one from the 14th.

                  So that's where I am at currently. The lesson learned is probably that I should have a backup of the actual GR game files too, but honestly we have a number of games where if we lose their core infrastructure, the data files might not be enough. (As it happens, TEC is our one really safe game, because we have a full copy that we use for testing purposes, in addition to all the ways we backup data files.)

                  Comment


                  • #10
                    Thank you, ShannonA for this detailed explanation.
                    SMARMY the GIANT

                    Comment


                    • #11
                      Thank you very much for the explanation. Sounds like you are working as hard as you can to resolve the problem so thank you for that, we know your time is valuable. With luck, things will get back to normal soon.

                      Comment


                      • #12
                        It is looking like this is a global problem with our Genesis backups. That's because the data files use something called "sparse data", which means that you have a data file that's mostly blank. I don't know why WAP opted to use them. I'd guess efficiency. But they've always been problematic, and we have to be careful when moving them around, to make sure they remain sparse. Otherwise, a file that is 14-15G can suddenly bloat to 109G (or at least that's what TEC would do right now).

                        It looks like our colo's backup and restore functionality is *not* careful and that when trying to restore the data file it bloats it, which fills our disk. And no one knew this before because we've never done a full restore of TEC or GR, because we've got so many data file backups.

                        Our colo just bumped our disk size from 50G to 150G, and the restore went for about 10 minutes instead of 3 before failing. I have a suspicion that if they double it again, we'll get our restore correctly, and then I can resparsify the files.

                        So, fingers are crossed.

                        Comment


                        • #13
                          At least some progress! All my fingers, toes, arms, and legs crossed. TY for the update again. BTW going upstairs with legs crossed is more difficult than I thought.

                          Comment


                          • #14
                            Originally posted by cjmccoy View Post
                            At least some progress! All my fingers, toes, arms, and legs crossed. TY for the update again. BTW going upstairs with legs crossed is more difficult than I thought.
                            You forget to cross your eyes! I am sure that should you do that, GR will be up and running pronto!
                            SMARMY the GIANT

                            Comment


                            • #15
                              Originally posted by ShannonA View Post

                              Our colo just bumped our disk size from 50G to 150G, and the restore went for about 10 minutes instead of 3 before failing. I have a suspicion that if they double it again, we'll get our restore correctly, and then I can resparsify the files.

                              So, fingers are crossed.
                              Can you request that they do so?

                              SMARMY the GIANT

                              Comment

                              Working...
                              X