Server corruption/outage and backup restored

We’re back!

Our host Linode had a major incident which affected most IFTF sites. While the others came back online after not too long, the forum’s server seems to have suffered actual damage or corruption. We’ve been able to restore from a backup, though I think about 1 day and 7 hours of posts have been lost. After waiting for updates from Linode (which were very thin on details), and then time zone issues, and making time after work, it unfortunately took 3 whole days to get the forum back online. But we made it eventually!

I’ll post a fuller postmortem in a day or two.

69 Likes

Congratulations on the restore. I’m sure everyone was working overtime on this one!

10 Likes

I feel fairly confident that I’m speaking for everyone here when I say thank you getting the forum back. We realize it’s an unpaid gig and are very grateful for your all your time. :heart:

17 Likes

Phew, I was worried for a few days there. Many thanks to everyone who worked on resolving this incident.

7 Likes

I missed this place so badly.

10 Likes

Thank you to everyone involved in getting this mess sorted out. It must have been traumatic. I think they should all be named so that we can put them in the intfiction Hall of Fame.

8 Likes

Thank you for your (& others involved) efforts to restore the service. Much appreciated.

5 Likes

Welcome back! I was beginning to worry.

5 Likes

Yay, thanks for all your work getting things back on line! Three days is too long to be away from y’all :slight_smile:

4 Likes

My thanks for Dannii’s work on restoring the forum !

Work more or less as early, save this very post editor. the M slider and the + menu button don’t work more (so, for example, I can’t blur spoilers…)

Hope that I wasn’t a cause of the major issues here (I was doing a complex editing of a prior post when the the crash happened)

Best regards from Italy,
dott. Piergiorgio.

5 Likes

Hooray for working backups!

3 Likes

Well done Dannii and team for getting things up and running again!

6 Likes

Thanks everyone for your great work!

5 Likes

Glad to see everything with the forum and services is working again! Well done on restoring. Was missing it, as I consider it as a very friendly community, always nice to read conversations and take part in them.

P.S. I see that recently the post creator has been updated.

4 Likes

I forsee a new IF game… “Attack of the Linodes”, about how the intfiction heros assembled and fought the linode aliens away from intfiction castle.

4 Likes

Thank you for all the hard work to get things back up and running!

3 Likes

Thanks @dannii and the IFTF for amazing work. I have never felt so useless not being a web-dev.

Forum is having a few glitches - likely some settings may have got twiddled from the backup, it might take some time for everyone’s individual read-message pointers to get updated, and likely there’s a lot of traffic with people catching up. We’ll try to get everything put back together over time.

Right now I’m noticing the message compose window is not the width of the window, though that may be a personal setting. I’ve also noticed if you hide the compose window it doesn’t like coming back up. When that happens for now I’m refreshing the browser window to get the compose pane back.

I’m also seeing weirdness clicking on notifications and having it not navigate to the correct message. That might be due to notifications for messages that no longer exist since the restore rolled back a week.

5 Likes

I confirm the various issues noted by Hanon.

Also, the links to the likes in the notification bubble don’t point to the liked post, but to the start of the topic. Which can be annoying, esp. in very long topics.

Best regards from Italy,
dott. Piergiorgio.

2 Likes

It looks like Discourse ended up updated in the process of reinstalling and restoring from backup, so now there’s a new “compose message” interface. If you click the toggle with a blue “A” on it:

It’ll switch back to the old Markdown editor that’s the full width of the screen, with the text box on the left and the preview on the right. (Far superior imo.)

It uses a new font now, but beyond that should work the same as before.

4 Likes