Downtime Report: Wednesday Night

We had an abrupt and unannounced downtime for all games and services starting sometime after 9pm PT on Wednesday evening. This was due to a massive power outage at our cloud computing facility of the sort that redundant systems are supposed to ensure never happen. After about an hour they were able to diagnose that the power was out. It took a few more hours for them to start restoring service.

At this moment, everything is back up except for Ironclaw and TEC. At the rate they're restoring things, I hope those will be back up in the next 15-30 minutes, but it's entirely in their hands.

As is typically the case with a sudden reboot, TEC and GR have to be rolled back. This is already done with GR. The TEC rollback should be just a few hours due to the more recent backups recently instituted, but I need to get the machine back before I can verify that.

Update: Ironclaw and TEC are now back too. The TEC rollback was 2.5 hours, from 9.30pm PT (when everything crashed) to 7.00 pm PT.
See more
See less

Mailing List trouble

  • Filter
  • Time
  • Show
Clear All
new posts

  • Mailing List trouble

    I am having trouble with some of the Marrach mailing lists I administer for organizations in game, primarily in that any administrator approval action is not working. I have tested this on multiple internet connections, multiple browsers, and even asked a trusted friend in another country completely to repeat the effort, to no avail.

    When someone sends a letter to the mailing list and aren't on it, or tries to join, we're sent an email with a link to approve, deny, reject or defer the email/application. It can also be accessed from within the system. However, when we click the radio button we want (accept or reject, primarily), and click send, NOTHING HAPPENS. The screen refreshes, the email or application is still there. What is new from two years ago, when I actively was moderating a mailing list for the Chroniclers and Collegium, is that when I click send now I get a popup notice about information being sent plain text. That didn't happen before.

    Other features still work. I can manually add people. I can remove them. But I cannot delete emails held in queue, I cannot approve them. I cannot approve requests to join the list.

    What really sucks about this is that I am getting an email every day to remind me about each person waiting to join a mailing list, even if I've manually added them, plus an email for every held email in queue.

    Can we please figure out what's going on here? It'd be great. If only to stop the gd spam.