Come on Danger, Backups Aren’t Hard

David Barrett —  October 10, 2009 — 1 Comment

You might have read how Danger, the company that manages the Sidekick phone, managed to, uh, lose all its data.  Like *all* of it.  For all customers.  Irrecoverably.

Granted, I’m probably following this story closer than most because I have a Sidekick in my pocket.  (Actually, it’s not in my pocket this instant because it’s plugged in lest it run low on batteries and lose all its data forever.)  I hosted the first developer dinner, when it was still called the Hiptop.  I wrote its first calculator.  (That’s right, it originally shipped without a calculator or an alarm clock.)  I’ve upgraded through four different versions of the handset.  I’ve been a loyal customer.  Even when Sidekick retargeted its marketing to blinged-out rappers, I stuck by its side(kick).

So the fact that they completely blew this totally basic operation is simply infuriating.  I mean, come on.  This isn’t hard stuff.  There are so many spectacular ways for problems to arise, but no backups?  That’s just… demoralizing.

Anyway, why am I taking this depressing walk down memory lane?  So I can say this: I commit to you, loyal Expensify users, that this will never, ever happen to you.  Not on my watch.  (And given that I’m the CEO, my watch is 24/7.)

Here’s why:

Expensify runs three geo-redundant datacenters, located in three different cities using different ISPs, replicated in realtime using distributed two-phase commit transactions.  This means even if two of those cities suddenly fell off the face of the earth, we still wouldn’t lose a single transaction.  Furthermore, we do 2 different nightly backups of the database to a storage volume that itself is backed up many times over — in entirely different datacenters.  All this is encrypted in more ways you can count (including ways that even I don’t have the authority to decrypt), with procedures in place for how to recover from backups or even rebuild the entire site from scratch at a moment’s notice.

Anyway, I don’t know where I’m going with this.  It’s just so absurd for a company like Microsoft to lose *everything* on their customers.  I mean, we back up even our *log files* twice nightly.  To not back up your customer data is downright offensive.  Security and reliability aren’t just good ideas.  They’re obvious ideas.  Obvious ideas that we take incredibly seriously, even other trusted names don’t.

David Barrett

David Barrett


Founder of Expensify, destroyer of expense reports, and savior to frustrated employees worldwide.

Trackbacks and Pingbacks:

  1. Unplanned Maintenance, Game On « Expensify Blog - October 1, 2010

    […] larger than we anticipated.  This made synchronization take longer than expected between our three realtime replicated datacenters, causing a timeout to occur between the parent database and its children.  The children gave up on […]

Have something to say? Share your thoughts with us!

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s