No announcement yet.

Ironclaw Down [and Back; though you May Have to Wait for DNS to Catch Up]

  • Filter
  • Time
  • Show
Clear All
new posts

  • Ironclaw Down [and Back; though you May Have to Wait for DNS to Catch Up]

    We've had a hardware failure on the Ironclaw machine. Our ISP is working to repair it ASAP. I'll the game back up when the hardware is back.

  • #2
    To update:

    1. They brought the machine back up.
    2. They moved us into a migration queue because of the instability.
    3. The machine crashed again.
    4. They moved us to their emergency migration queue.


    • #3
      The hardware crash unfortunately corrupted the main database. I went ahead and recovered from my last backup, which was 9.30am PT. That unfortunately means there was some rollback.

      For some reason the game's art files are not working. I'm looking into, but it's very bizarre since they're on another machine.

      If there seem to be other problems, let me know ASAP. It was a hard crash, and if there is any lingering corruption on the disk we can revert to an earlier backup of the whole disk, from Sunday. That would be not preferred, however, if things look good.


      • #4
        Images fixed. I'm hopeful everything is working now.


        • #5
          Unfortunately, the restore experienced a crash just before 1am which suggests it's not stable. Rather than building up more history on an unstable build, I'm going to clean up the actual backup, which turns out to be 5.30am on Monday, so just a few hours back.

          This is going to require creating a new machine and moving the DNS over, so there may be some lingering slowdown for people to get over to the new machine.

          Apologies for the rollback & inconvenience, but hardware problems sometimes cause this type of nasty issue.


          • #6
            Because of the second crash of ironclaw this evening, I decided that more data had been corrupted by the crash this afternoon and that it was best to go to a totally clean backup. I've done so, refreshing to the complete-machine backup from Monday 5.30am. This should be a clean, trouble-free copy of the game, since it's all from before this afternoon's damaging crash.

            Unfortunately, doing this rebuild required updating Ironclaw's IP address. (Or, at least doing it at 1.30am in the morning and not wanting to throw out the old version until we know we're happy with this backup did.) I've updated the DNS records, and people should be able to get back in as soon as they have the new DNS ... but that can take time.

            Anyone who is still stuck with the old IP address (which will show Ironclaw not available) can just put the following in their /etc/hosts file or equivalent, and it should immediately give them access to the rebuilt game:


            If you do this I suggest marking yourself a TODO item to take this entry back out in two days time, by which all of the DNS should be totally updated. Otherwise, you may end up with a confusing situation if I have to change the IP address sometime again in the future ... and this does happen from time to time!

            The other unfortunately is that this upped the total rollback to about 11 hours. But I decided that was superior than having a game that might be continually unstable going forward.


            • #7
              PS: Our standard DNS TTL is 24 hours, which means that the maximum amount of time it *should* take you to see the new IP address is one day. The average is obviously more like 12 hours, which is one of the reasons that I made sure to do this update in the middle of the night: that way it'll be propagated to maybe a third of y'all by morning.

              Some folks may see longer because some ISPs illegally cache IP addresses past the TTL.

              You may also see caching in your browser, so before you go play with your hosts file, try restarting your browser, just in case.


              • #8
                At this point, the DNS records should be updated for everyone, unless you've got an ISP doing bad things.

                I suggest:

                1.) If you put an entry in your hosts file, take it out.
                2.) If you're still having problems, restart your browser.
                3.) And if that doesn't work, restart your machine.

                The latter too should shake free any caches on your side of the world.