Adventures of a Script Kiddie

Over the past few years, my engineering team and I have deployed some highly unconventional solutions to production to save time and cut costs. Every time we rolled one out, we told ourselves we’d come back to clean it up the next quarter. But with each new quarter came a fresh set of engineering challenges, and we found ourselves patching things up with makeshift solutions—just enough to keep us afloat for another night.

In the process, we ended up designing systems that could handle tens of thousands of writes per second and roughly six times that in reads. Some of these services now boast uptimes that surpass those of Fortune 500 companies and have operational costs that are orders of magnitude lower (🥜). After more than five years of keeping these platforms running and serving hundreds of thousands of users they’ve evolved from bootstrapped fixes into what actually now feel like legitimate engineering solutions.

This blog is a memoir of all the problems we encountered on production and how we "solved" them.