Expensify Blog

Years ago Witold and I joked that performance is a problem we’d “love to have”.  It means that you have so many people using and liking the thing you’ve made that it’s starting to slow down.  After all, an iPhone has enough horsepower to power most websites: it really takes a lot of work to overload an actual business-class server anymore.  If you’re experiencing genuine slowdowns, it’s a sign that things are on the right path.  It’s a problem you should love to have.

However, it’s still a problem.  In our case, things aren’t quite as snappy as we’d like.  We’ve increased from 3 to 5 datacenters, vastly increased hardware, and the users keep coming.  So we’re embarking on a much more deliberate optimization campaign than we’ve ever done (or had need to do) in the past.  And that’s great because optimization it’s fun!  It’s full of all sorts of “classic” engineering problems — phrases like “Big O” get tossed around with nary a chuckle.

But like most fun things, at its heart lurks a deep evil.  Here’s what I advised our team, and I thought I’d share it with you as a public awareness campaign highlighting a universal problem.

Premature optimization is the root of all evil. I mean that in the most literal and expansive way possible. Resist this devil with all your might. Here are some tips:

1) DON’T OPTIMIZE IF YOU DON’T KNOW IF IT’S SLOW.

I repeat: don’t make something faster if you don’t actually know how fast it is. It’s possible — in fact, highly probable — that it’s actually adequately fast as is. I’d estimate that 80% of optimizations have absolutely no beneficial effect, and actually make code slower and less maintainable. Don’t be “that guy” who spends all day optimizing something that doesn’t need it, while ignoring the enormously huge low-hanging fruit all around.

2) BENCHMARK BEFORE OPTIMIZING.

Optimization is a field where you can spend an unlimited amount of energy accomplishing precisely nothing of value (or even less). The best protection against this is measure its real-world performance before starting, as this is the only effective way to prioritize areas to optimize. The best measure is that which is experienced by the end user.

3) BENCHMARK AGAIN AFTER OPTIMIZING.

If you don’t know that your optimization worked, it’s safest to assume it didn’t — just like the vast majority of optimizations. If it didn’t, then revert the change. It doesn’t matter how long you spent building some fancy caching layer or tweaking settings in ways that “I’m sure” are better. Don’t trust your gut, trust the numbers. If the numbers say you’re wrong, you’re wrong.

Note: Optimizing is not the same as refactoring, though often they go hand in hand. If you’re refactoring code to make it cleaner and simpler, then it doesn’t matter if it goes faster, so no need to revert a cleanup if it has no speed improvement.

4) FOCUS ON ABSOLUTE TIMES.

It means nothing to say that something is 50% faster if it was already fast enough to start. Don’t even think about problems measured in “miliseconds per X” until all of the “seconds per X” problems are already handled.

5) DO THE MOST IMPORTANT OPTIMIZATION FIRST.

Not the one you just found, or the one that’s been nagging you, or the one that lets you use that really awesome algorithm. This is the absolute hardest of all: when you find something new, add it to the list, but always start with the optimization at the top of the list. If the thing at the top of the list hasn’t already been benchmarked, benchmark it. If after benchmarking it you find that it’s actually not as important as something else on the list, log your benchmark results, then put it back on the list at a lower priority.

6) SHOW LOYALTY TO THE USER, NOT TO A LAYER.

Don’t optimize something merely because it’s the slowest in a particular layer; always do whatever increases end-user performance first. Trace the issue and benchmark through all layers before deciding how to speed it up. Leave no layer unturned.

7) CHANGING USER EXPECTATIONS IS THE BEST OPTIMIZATION OF ALL.

Remember, the goal is to make the user satisfied. One way is to actually live up to the user’s expectations. Another is to change their expectations by adjusting the flow in such a fashion that they just don’t expect as much. Some blocking operation really slow? Make it asynchronous such that the user doesn’t care how long it takes.

8 ) BROADCAST BIG PLANS BEFORE YOU DO THEM.

Before building out some really crazy ambitious awesome thing, email to the team explaining what you’re going to do. *Sell* your change, based on the benchmarks. Be proud of your plans; if you feel hesitant it’s probably a sign that you don’t really know what you’re doing or why — and thus probably shouldn’t.

I’m serious, premature optimization is what sinks ships. It’s the tarpit of any engineering organization; it’s where clean and simple architectures go to die. Resist the devil. Be strong.