Praying to the servers

If you work in tech, you've probably seen the picture. A group of people bowing in front of a rack of servers, as if the machines were gods. It gets shared every time something big goes down, with a caption like "Pray to the servers."

It's funny because it's true. There are moments when you've tried everything and all you can do is wait and hope.

Those moments love bad timing. A Saturday night. Your first real vacation in a year. Or Tet, the Lunar New Year, when the whole country is home with family and your phone starts buzzing during dinner. Sometimes the cause is a mystery. Sometimes it's a silly mistake. Sometimes you know exactly how to fix it, but the recovery script needs two more hours, and there's nothing left to do but stare at a progress bar.

My own prayer moment

One of mine came early, in my first year as an engineer.

Our order lookup was slow. I had just finished school with a high grade in databases, and I was proud of my SQL. I knew exactly what to do: add an index.

An index is like the index at the back of a book. Without one, the database reads every page to find what you asked for. With one, it jumps straight to the right page. For a slow lookup on a big table, it's usually the right answer.

So I added it. In production. On the orders table.

What my grade hadn't taught me: on a big table, building a normal index can lock the table while it works. Reads may still go through, but new writes have to wait until the index is finished. And on a busy orders table, that can take a long time.

New orders couldn't be saved. Requests piled up behind the lock. Timeouts spread. In an online business, orders are the heart of everything. For a while, that heart stopped beating.

The idea wasn't wrong. The index was exactly what the lookup needed. It just wasn't 100% right. The fix was a small change: build the index without locking the table, which many databases support. In Rails, it's a couple of extra lines in the migration. I didn't know they existed.

Keep learning, especially now

That day taught me that knowing the theory is not the same as knowing how it behaves under real traffic.

If you use Ruby on Rails, I can't recommend strong_migrations by Andrew Kane enough. It catches dangerous database changes like mine before they run, and tells you the safe way to do them.

It matters even more now. Ask AI to speed up a slow query and it will happily write the index for you. It may not mention the lock. That's why I keep saying vibe coding is simple, but not easy. The code you can see is the tip of the iceberg.

A place to pray

Years later, a colleague joked that we should have a real place to pray to the servers. For those moments when you've done your part and you're just waiting: for someone else's fix, for a script to finish, for a vendor to reply.

So I built one. It's called Wot Da Shrine. You open a shrine for the outage, share the link with your team, and everyone gathers at a little virtual altar. You can light incense, bow together, leave offerings, and type your prayers. ("Rollback, I beg you" is a popular one.) There's even a censer where you can burn a blame, so the confession goes up in smoke instead of into the group chat. And when the servers come back, someone gets to press "They're back!"

It doesn't fix anything. But it turns a stressful hour into something a team can laugh about together, and that matters more than it sounds.

Pray less

The prayer page is fun. I still hope we never need it.

Outages will happen. But most of them come from things we could learn: how databases really behave, how systems fail, how to change them safely. Study the fundamentals. Design for failure. Keep learning, and never stop.

The best server prayer is the one nobody has to say.