Back to blog

The Weekend the Server Took a Nap

incidentpost-mortemtransparency

Let me start with the part that matters: the site was down for a stretch this weekend, and I'm sorry. If you came by to check a streak and got nothing but the error message "Failed to load games. Please try again." staring back at you, that one's on me.

Now let me tell you how it happened, because the whole thing is almost funny. Almost.

What went wrong

While fetching the latest FIFA results, the data pull hit something it didn't like and threw an error. Fair enough — errors happen. The problem is that this particular error didn't just get logged and shrugged off. It marched straight upstairs and took the whole server down with it.

In other words: one bad row of football data walked into the building and flipped the main breaker on its way out.

The part where the safety net also fell

"That's fine," you might think, "surely there are scripts to restart the server automatically." There are! I wrote them myself! They were ready, willing, and completely useless.

The restart scripts — the ones whose entire job is to spring into action when things go wrong — picked this exact moment to also fall over. It's like calling the fire department and the fire truck catches fire. Beautiful stuff, really.

And of course, it was the weekend

Naturally, none of this happened on a quiet Tuesday afternoon while I was sitting at my desk with some tea in hand. It happened over the weekend, while I was off, gloriously unaware, doing literally anything other than watching server logs.

So the site sat there, patiently broken, waiting for me to come back and notice. Which I eventually did. On Moday.

The honest bit

This is the risk that comes with a hobby project that's grown enough to be presented to a real community. There's one person behind it — me — and that person occasionally has a life, a weekend, and an unreasonable amount of faith in scripts he didn't double-check.

I'd always rather tell you exactly what happened than pretend the uptime was perfect. So here it is, warts and all.

What I'm doing about it

  • Making the FIFA results fetch fail gracefully — one bad batch should degrade a feature, not detonate the server.
  • Actually testing the restart scripts instead of trusting them like an old friend who's never let me down (because, as it turns out, he just did).
  • Adding alerting so I find out the server is down before you do.

Thanks for sticking around, and thanks for your patience while I was off being a human being. The server's back up, the streaks are streaking, and the restart scripts are on notice.

See you in the data.