Co-Lo Stability Problems

Unfortunately, our co-lo had some stability problems Friday morning.

First up, they lost all of their connectivity to the facility that holds all of our games. This took them about 40 minutes to diagnose and a bit longer to restore.

Much more astonishingly, they rebooted all of their machines (which means all of our machines) while they were fixing the problem, with no notice. This probably had minimal repercussions on the SkotOS games (Marrach, Ironclaw, Lazarus, and Lovecraft). There may have been a rollback of an hour or two.

TEC and Grendel's Revenge are more troublesome. TEC will suffer a rollback of 10 hours, to midnight (PT) last night, since the unannounced reboot happened one hour before the next backups. GR suffered a more complete failure. We're being forced to restore the entire machine from its backups, then we'll need to restore the game from the backups stored within that backup. The lineup of dates look promising (which is how we laid them out, 'lo these many years ago), so the hope is that we'll only suffer a 7 hour rollback, to the 3am (PT) game backup, but it's a bit dicey at the moment.

Apologies for the inconvenience; we're now making sure everything is stable, and we're looking into whether our most recent upgrades to TEC will support increasing the backups to every 6 hours.

Edit: The GR problem is proving even more intractable than hoped. As best I can tell, our colo is still having some issues because I can't boot the first restore I made of GR and I can't build a new node either. I expect this will just clear when they get their act together, but in the meantime I'm not able to get GR back online until they fix these problems.
See more
See less

The Eternal City

Sub-Forums Topics Posts Last Post
News from the Eternal City gamemasters.
Topics: 147 Posts: 1,529
147 1,529
General Announcements
by Scolve
Events in the Eternal City
Topics: 1,849 Posts: 6,840
1,849 6,840
[Event] The Great Hunt
by Kered
Announcements from Eternal City players.
Topics: 4,385 Posts: 58,999
Last Post: I miss you guys
4,385 58,999
I miss you guys
by Syden
Administrative Q&A about The Eternal City.
Topics: 1,865 Posts: 38,301
1,865 38,301
Talk about Iridine and the world of Midlight.
Topics: 5,822 Posts: 128,253
5,822 128,253
The Tile of the Day!
Talk about the mechanics of TEC.
Topics: 5,179 Posts: 101,718
5,179 101,718
Make concealment great again!
by lykatos
Off-topic Chatter
Topics: 7,930 Posts: 170,580
Last Post: Blast from The Past
7,930 170,580
Blast from The Past
by Tyril
  • Filter
  • Time
  • Show
  • New Topics
Clear All
new posts

  • ArchMagi
    replied to List of Website Updates Needed?
    what pages need a log out link? Forums have it at the top, play now has it at the top, what other pages need one? serious question.
    See more | Go to post

    Leave a comment:

  • lykatos
    replied to Make concealment great again!
    They removed the who list once... there was a great revolt. Two days later it returned in it's current form.
    See more | Go to post

    Leave a comment:

  • ArchMagi
    replied to Welcoming to new players?
    Orchil is in beta, but it's also new, meaning your concerns are being addressed, and once Orchil is finished being in beta, it will have been address...
    See more | Go to post

    Leave a comment:

  • Tyril
    replied to Blast from The Past
    Fun, I remember spending most of my 9th grade study hall reading the blocks page of runes website....
    See more | Go to post

    Leave a comment:

  • Kered
    replied to [Event] The Great Hunt
    Hopefully both, except Kered can't get saplings curses!
    See more | Go to post

    Leave a comment:

  • isgrimnur
    replied to Thoughts on New Veteran Packages
    The Senate decreed on the 13th of Lufical, 285, that a special set of games are to held. The vote was unanimous. The pressing topic of the influx of refugees...
    See more | Go to post

    Leave a comment:

  • Events
    started a topic [Event] Celebrating Glory and Honor

    [Event] Celebrating Glory and Honor

    9:30pm - Celebrating Glory and Honor

    To honor the upcoming quell and its participants, the Swordcat is...
    See more | Go to post

  • Syden
    replied to I miss you guys
    I wouldn’t say that. Up to 50 on weekends. Forums are dead though.

    Theres this great event happening where the senate is allowing everyone...
    See more | Go to post

    Leave a comment:

  • GinaDana
    replied to The Tile of the Day!
    The view from the top can only be described as breathtaking. To the north, south, and east the cliff drops off for hundreds of feet to the ravine below....
    See more | Go to post

    Leave a comment:

  • Events
    started a topic [Event] Auction

    [Event] Auction

    8:00pm - Auction

    Everyone Welcome!
    Weapons, Armor, Furniture and Clothes!
    Some items not sold...
    See more | Go to post
There are no conversations in this channel.
  • Filter
  • Time
  • Show
Clear All
new posts
Please log in to your account to view your subscribed posts.