Expensify Blog

There’s nothing more frustrating than wasting time trying to save time, and the easiest way to do that is to use some complex off-the-shelf solution for a really easy problem.  Take something as straightforward as user help.  If someone needs to ask for help, they already have a problem — it’s the absolute worst time to make a user jump through hoops, and certainly not hoops that don’t work.  Accordingly, we’ve tried to make it as easy as possible to get support:

• just email help@expensify.com,
• click Help in the upper-right corner,
• or if all else fails, click Feedback in the upper-right corner, type your issue into a box, then click Send.

Easy peasy.  Everything goes to a help person, they answer within 24 hours (if not immediately), and everybody’s happy.  It’s why Expensify has legendarily good support.

Sometimes we get asked, why “waste time” writing our own help system, and not “save time” by using something off the shelf, like GetSatisfaction?  And one major reason is because writing it ourself took so little time; integrating an outside system (with CNAMES, SSO, etc) would have taken longer.

But the real reason we don’t is because the one thing it needs to do best — help users get help — it does poorly.  Indeed, GetSatisfaction gets satisfaction for the company, by making the user do a bunch of extra work just to ask the frickin’ question in the first place.  Take my recent attempt to figure out why I can’t upgrade Thunderbird:

1. I find the relevant help page on Thunderbird’s site.
2. The answer isn’t there, sadly, but I see this cool button titled “I have this problem, too!”
3. Given that I do the have problem, I click it thinking that it’ll — I dunno — add me to a list of people with this problem.
4. Instead, it asks me to log in to GetSatisfaction, and provides a zillion options to do so.
5. I’m not really interested in doing this, but I figure I’ll give it a shot: I choose the Facebook option.
6. A big pop-up flashes before my eyes, and now I see my profile image — cool, it connected to my account!  Except.. it’s still prompting me for my email address.
7. So I type in my email address, even though the whole point of Facebook Connect is it shouldn’t need it (or should automatically get it).
8. I click Continue, and it tells me “This email is in use with an existing account. Please login with the password for this account.”  What the hell?  Why did I bother Facebook Connecting?
9. I try my password.  Naturally, I can’t remember it.
10. I try resetting it.
11. While waiting for the password reset email, I instead decide that a better use of my time is to write this blog post explaining why Expensify doesn’t use GetSatisfaction: because I don’t want to subject our users to the same bad experience I’m currently enduring.

Don’t get me wrong — in the best case scenario, GetSatisfaction is really good: if you already have and are signed in to an account, or if you remember your password or if FaceBook connect doesn’t flake out; if everything loads quick; if the question already exists and somebody bothers to respond to it; etc.  The best case is great.

But when dealing with support, you should be far more interested in the worst case, as that’s what the user who’s having trouble will probably endure.  Probably they don’t already have a GetSatisfaction account, or if they do, they’re not signed in to it, don’t remember the password, or have some FaceBook connect problem.  Even if the their question has already been asked a million times, it doesn’t change the fact that they want to ask it again — if they wanted to search for an answer, they probably would have already found it.   And even if they survive the process and successfully ask the question, you’ve got to remember to actually respond.

I don’t mean to bag on GetSatisfaction — it’s one of the best of its class of system.  But it’s “class of systems” shares a lot of the problems of classic enterprise products: a huge product solving a small problem, sold to someone at the “top” even though it’s used by people “at the bottom”.  Indeed, this dynamic is what makes enterprise products suck: the people using the product (employees, end users, etc) have no real say in the selection, and thus make up a very small part of the pitch.

For example, consider the text on the GetSatisfaction homepage:

• Measuring value from social media shouldn’t be this hard
• Give a voice to brand champions
• 50,000+ companies trust Get Satisfaction
• Get Satisfaction for Facebook: turn Likes into Loves
• 2011 Community Manager Insights
• Enterprise: Sometimes size does matter
• Get Satisfaction powers the world’s best brands to solve problems, give a voice to champions, bring out the best ideas and drive better business.
• Who is JarGon?  JarGon is the customer service robot. He has no heart and isn’t capable of love. He was created in a secret lab to frustrate customers, and Get Satisfaction is locked in an epic battle to protect the populace from this bumbling, metallic menace.

Is ANY of that interesting to the end user?  The only thing a user wants is “Get answers to your questions fast and reliably”, but that message just isn’t there.  Again, it’s for an obvious reason: GetSatisfaction isn’t sold to users.  They don’t participate in the buying process.  Their concerns are, frankly, secondary.  Not necessarily an afterthought, but it’s clear from their language that their focus is on making the company, the customer service department, the community manager — the “top” happy.  The “bottom” just has to deal with it.

This top-down dynamic gradually — inexorably — focuses the product on the buyer’s needs, not the user’s.  It forces the product to adopt a bunch of features the user doesn’t care about, or that the user actively dislikes, in order to justify a thicker bullet list and bigger price tag to the buyer.  It makes the product suck.

Stay focused on the end user or your product will also suck.

PS: Wouldn’t it be slick if there were some sort of help system taking the Expensify “bottom up” approach that serves as sort of help agent? For example, it could:

• Have a 24/7 concierge staff that is generally versed with how to navigate help systems and specifically familiar with supporting the top products
• Automate integration with the top support systems (GetSatisfaction, UserVoice) as well major brands help systems.
• Have a case-management system that works for youso when you log a support ticket with this service, the service goes out and tracks down the answer for you — finding and emailing the right address, nagging whoever needs to be nagged, calling and waiting on hold for you, posting to the right forum and waiting for the response, etc.
• This service wouldn’t actually answer your support questions, they would just serve as your agent to ensure your question gets asked (and re-asked) in the right place, give you ongoing status of each outstanding question, and then route the response back to you.

Maybe it’s a terrible idea that wouldn’t work. But if you could make it work, you’d have a userbase that loves you, and would be able to gradually disrupt all the other help CRMs out there. Users are sick of being cut out of the process, and if you give them a voice, they’ll speak up loudly.

All datacenters fail.  Amazon EC2 is down (edit: still down 40 hours later!!), for the 2nd time in 2 months.  Rackspace, the most reliable (and most expensive) has a long history of going down.  365 Main — perhaps the most prestigious host in the Bay Area — went down in a fascinating story that I sometimes use as a brain teaser during interviews.  Basically, it doesn’t matter how good your datacenter is: it’s going to go down.

And if it doesn’t, your hard drive is going to crash.  Or your transformer will fry.  Or some crazy process will peg all the CPUs.  Or whatever.  It doesn’t matter how good your hardware or software is, how fast your ops team can respond to pages, or how totally invulnerable your cloud hosting provider claims to be: it will all fail you one day, and your only response will be to frantically do nothing while waiting for someone else to fix it.

Unless, of course, you have so many datacenters that you simply don’t care.

A couple weeks back I mentioned that we put an unbelievable amount of effort into creating a secure hosting environment.  A part of that that’s often overlooked is the importance of realtime redundancy.  Not having spare parts on hand, and not having backups readily available, but having multiple live systems replicating in realtime at all times, such that if any of them fail, there are still plenty left over to pick up the slack.  And a subtle point that is is even more often overlooked is that the minimum number of datacenters certainly isn’t one, and surprisingly isn’t two, but is actually three.  Here’s why:

• One datacenter obviously isn’t enough, because if it goes down, your offline.  Furthermore, if it goes down forever (or sufficiently forever to cripple your business), then that’s especially bad.  Backups are ok, but you’ll still lose several hours of data — and if you’re a financial service like us, that can mean a lot of money that you don’t know where it went.  So obviously, one isn’t enough.

• But two isn’t enough either.  Sure, having two datacenters operating in parallel gives you confidence that if one goes down, the other will still be there: it can provide uninterrupted service without any data loss, and that’s great.  But if one goes down, then you’re left with only one datacenter remaining — and we already agreed that was completely unacceptable.  Remember: your servers absolutely will go down, plan on it.  And there’s no rule saying that just because one of your datacenters died, the other won’t.   So if you can’t ever reasonably operate with one datacenter, then you just as reasonably can’t operate with only two.

• That’s why three is the magic number.  With three datacenters, you can safely lose any one, and still have two left over.  Which means even if you lose one entire datacenter — and you will, with depressing regularity — you’ll still have two more, which is the minimum required to maintain a live service.  (And in case you’re wondering, Expensify is programmed to go into a “safe mode” and voluntarily take itself offline in the event that it loses two datacenters — we figure if there’s some sort of global catastrophic event going on, odds are people aren’t filing expense reports.)

Granted, I can understand why people don’t do this.  Most startups follow some variation of the following thought process (assuming they get that far):

1. Hey, I’ve got this great idea, let me whip it up on my laptop with a local LAMP stack!
2. That’s awesome, let me put it on some co-located / dedicated / virtual server.
3. Whoa, it’s starting to take off, let’s get a bigger box.
4. Ok, it won’t fit on the biggest box, let’s get a few webservers talking to a giant MySQL server.
5. One MySQL server isn’t enough, let’s get a ton of memcache’d webservers talking to a few MySQL slaves, coming off of one giant MySQL master.
6. Oh shit, my datacenter just went down.  Who’s job was it to get more datacenters?
7. What do you mean my dozen servers all need to be in the same datacenter on a GigE network?
8. Ok, that sucked, let’s go to Rackspace.
9. Wait, now Rackspace is down too??  Shit shit shit
10. sleep( rand()%12 months )
11. goto 9

And frankly, I think it’s a reasonable plan.  Most startups don’t honestly require serious availability or durability because most startups don’t deal with hyper-sensitive financial data like us.  However, as much as I’d like to claim we did all this out of some commitment to engineering excellence, I’ve got to admit: we did it because we had no choice.  Our thought process was:

1. Hey, I’ve got this great idea, let me go ask a bank if we can do it!
2. They said no.  Hm…
3. Repeat steps 1-2 about 20 times
4. Ok, I’ve found someone who will let us do it, but we’ve got totally insane security and uptime requirements
5. Research… research…
6. Ok, MySQL doesn’t really do what we need
7. Research… research…
8. Hell, nothing does what we need, at least at startup prices.
9. Well, luckily we’re P2P experts so let’s just write a custom PCI-compliant WAN-optimized distributed transaction layer.
10. Wow, that was hard, but I’m glad we’re done.  Let’s launch!
11. Things are taking off, let’s upgrade all our datacenters one by one… cool!  No downtime!
12. Oh crap, that hard drive died in the middle of the night.  But cool, no downtime!
13. Our datacenters are too far apart and the latency is affecting replication times; let’s replace them one by one with lower-latency alternatives.  Cool, no downtime!
14. One of our datacenters seems to be flaking out every once in a while… screw it, let’s just replace it.  Cool, no downtime!

And so on.  Not to say that we don’t have problem, as we obviously do.  But I can’t overstate how comforting such a highly-redundant system, and I hope it provides a degree of comfort to you.

On a recent flight from ATL to SFO I sat next to a woman coming out to San Francisco for an advertising conference.  We got to talking, and when I mentioned that I had started Expensify, she shared her own entrepreneurial story with me.  What struck me most about her story wasn’t the uncanny similarity to the movie Sunshine Cleaning (she’s cute, starts a cleaning company, struggles to make it work, and eventually finds success), but rather how she began:

When I started the company, I had no idea what I was doing.

I thought that was so refreshing to hear, because this town seems full of people who will tell you with utter confidence that they not only know what they’re doing, but what you should do as well.  The blogs celebrate visionary founders who ruthlessly pursue crystal-clear plans, as if operating out of some proven playbook that is secretly shared amongst the rich and powerful.

But I think that’s all a sham.  There is no playbook.  The visionaries are called that after their visions have come true.  For every one Mark Zuckerberg or Steve Jobs, there are a thousand others who try the same techniques but toil endlessly in obscurity.

At the end of the day, I just don’t think entrepreneurship is a science — at least, not one that anybody understands.  Life is simply too short for anybody to personally accumulate enough first-hand knowledge to speak with authority on the “right” or “best” way to start a company; nobody has enough reliable data to draw any sort of statistically-significant conclusion.  And the people best positioned to learn from others — investors — are too often fed a very carefully-crafted, narrow vision of what’s happening in their portfolio companies, and thus never really know the real story.  Even (or especially?) the authors and academics who make a business out of studying business — they seem to disagree even more than most, producing endless shelves of controversial and contradictory books, each touting itself as a different yet still final word.

So if you’re starting a company and feel you’re out of your league and usually don’t know what to do: don’t feel bad.  Nobody does.  And the people who think they know, they’re probably just fooling themselves.

As much as we might prefer otherwise, starting a company — even a series of companies — isn’t a scientific endeavor.  It’s not a series of repeatable experiments.  Rather, it’s like drinking one unique, once-in-a-lifetime bottle of wine at a time: each of which is stored, opened, and tasted under different conditions, paired with different foods, and enjoyed with different company.  What tastes good and works well for me might have the totally opposite effect for you, and that’s ok.  Because entrepreneurs aren’t professional scientists, they’re amateur sommeliers, and that’s what makes it so fun.

Security is a topic near and dear to my heart. Most of the year leading up to launch was spent making a PCI-compliant, super-redundant, ultra-secure financial transaction layer.  And can you even imagine how hard it is to train lions to fight with lightsabers at anything more than the most rudimentary level?  When the blogosphere got all atwitter because Twitter *finally* made a “default HTTPS” option, I was baffled.  And now to see TechCrunch lauding FourSquare for making a:

solid move. And not the easiest one to make. One of the main reasons that every site/service doesn’t turn it on is a simple one: it means a performance hit.

What?!  This isn’t exactly genius stuff.  Quite the opposite: it’s a long-delayed fix to a glaring security hole.  It’s an embarrassment.  We’ve been doing this from the very first day, even before it was cool.  Scratch that — it was never cool, it was just obvious.  It makes me wonder: what are these guys still missing?